To: steve@xemacs.org
Subject: [Tonny Madsen <tma@nettest.dk>] Re: A working version of shared-lib.
Errors-to: wmperry@aventail.com
Reply-to: wmperry@aventail.com
From: wmperry@aventail.com (William M. Perry)
Date: 19 Jan 1998 07:20:51 -0800
Message-ID: <863eikv6b0.fsf@kramer.in.aventail.com>
Lines: 630
Xref: altair.xemacs.org inbox:16807

------- Start of forwarded message -------
To: wmperry@aventail.com
Subject: Re: A working version of shared-lib.
References: <rr91dvlu0p.fsf@crow.netman.dk> <199703070555.VAA00480@wmperry.in.aventail.com> <ix7mes5vh4.fsf@balder.i-did-not-set--mail-host-address--so-shoot-me> <86pvrg2v3v.fsf@wmperry.oz.net>
From: Tonny Madsen <tma@nettest.dk>
Date: 01 Sep 1997 14:28:07 +0200
Message-Id: <ixk9h11anc.fsf@nettest.dk>

Hi' Bill,

Of cause I had to be sick when the wheather has at its best. (30-32 C,
which isn't al that usual here in Denmark) But now I back (and the
wheather too is back to normal)...

wmperry@aventail.com (William M. Perry) writes:

> Tonny Madsen <tma@nettest.dk> writes:
> 
> > Sorry about the late answer, but I have changed jobs in the meanwhile
> > and I have had some servere problems getting private my communication
> > from my old company.
> 
>   That's ok - just glad you were still around. :)
> 
> > At present, I don't have any time for serious work on XEmacs or any of
> > the other interest areas of mine. That means that the version you
> > probably already have (for 20.1b26 ??) is the current version.
> 
>   My hard drive has crashed since then and I can't seem to find a copy on
> any of my backups...

I have appended a copy (uuencoded gzipped tar) to this message.

> 
> > Another reason for the current state of affairs, was the talk that was at
> > some time about using a completely different LISP kernel than the native
> > Emacs one. Doing that (which I personally think is the right thing to
> > do), would probably solve all the problems that I have tried to solve in
> > my patch... and in a much cleaner way!
> 
>   Unfortunately, I think that will take much longer to do than just firming
> up your stuff and merging it in.  Would there be any copyright issues with
> your stuff?

No copyright issues... I'm happy to say that the laws on work done
outside working hours are quite strict here in Denmark. Unless your
contract says differently, you always get the copyright yourself. And
unless you company has a commercial interrest in the line of work you
do in your spare time, it is next to impossible for them to get a
copyright on your work (even if your contract says differently).

> > If you do choose to use shlib, you should know that the patch is rather
> > large, as it involves a complete rewrite of the object system of
> > XEmacs. From my benchmarks, my new object system seems to be 10-15
> > faster(!) than the existing object system!
> 
>   Do you mean 10-15%, or 10-15 _times_ faster? :)

% not * !!!

The main reason for the speedup seemed to be in the handling of small
objects (previously used FROB lists), of which CONS cells play a
rather large part.

Note that there seems to be a couple of new objects in XEmacs since my
patch, so you will have to do some work on intergrating the new object
types.

And I don't know how the dynamic sizing of the pure-store-array is
handled. There might be a little work there too...

> 
> > Send me a line and I will skip the patches to you (remember relative
> > to 20.something)
> 
>   If you can drop me that, it'd be great - I can see about merging it into
> the 20.3 betas and also my abstraction layer for dynamic loading as well.
> 
> > As for the low-level interface, I was inspired by the perl stuff, as they
> > have already solved the problems. Unfortunately, perl does not support
> > unload of an library, so there will still be some work left...
> 
>   If it is a major hassle, I'd say we could skip the unloading portion.

I think it is quite easy, once you know the different platforms...

> 
> -Bill P

3 attachments (sorry not MIME):

- a tiny example on a shared library (makefile and C file)

- the old description for the patch

- and the path itself in uuencoded gzipped tar format.

Good luck...

/tonny

p.s. It is amazing how fast W3 has become!!!!

------------------------------------------------------------
#include <shlib.h>

Lisp_Object		Qetest_var;

DEFUN ("etest-val", Fetest_val, Setest_val, 0, 0, 0 /*
Return the value of the etest variable.
*/ )
     ()
{
  return Fsymbol_value (Qetest_var);
}

Lisp_Object
shlib_of_etest(Lisp_Object shlib, Lisp_Object function, Lisp_Object arg)
{
  struct Lisp_Shlib	*shlibc = XSHLIB (shlib);
  int			func = XREALINT(function);

  switch (func) {
  case SHLIB_ENTRY_LOAD:
    defsymbol (&Qetest_var, "etest-var");
    defsubr (&Setest_val);
    break;
  case SHLIB_ENTRY_TEST_UNLOAD:
    return Qt;
  case SHLIB_ENTRY_UNLOAD:
    return Qt;
  case SHLIB_ENTRY_GET_UNLOADABLE:
    return Qt;
  case SHLIB_ENTRY_GET_VERSION:
    return make_int(SHLIB_VERSION);
  }
}
------------------------------------------------------------
Shared Library Support for XEmacs 20.0-b26

** Table of Contents **

- Introduction
- The Major Changes
- Future plans
- User's Guide (sort of) for the shared library module
- User's Guide (sort of) for the CLOS module
- How to apply the patch
- Other Changes
- SHLIB Low-Level Interface

----------------------------------------

** Introduction **

This is the '960814' ALPHA patch for XEamcs-20.0-b26 that adds support
for shared libraries and (VERY) limitied support for CLOS.

* Shared library module *

With this new module you can load shared libraries into Xemacs at
run-time and add new functionality to the running XEmacs this way. You
can add just about all the functionality as you can in an ordinary C
file linked into Emacs, i.e. new classes (see description below),
subr, symbols, keywords and special variables (defmumble). You can of
cause not modify the existing functions, so you might need some new
hooks instead.

A "hacked" version of the module already existed for XEmacs 19.13, but
at that time it didn't always work as wanted due to it "hacky" (sp?)
nature and consequently I just kept it for myself.

* CLOS support *

There are some very, very, very limited support of CLOS. A lot more
will be added in the next version.

BUT, for the next 4 weeks (until September 16.) I'll be on vacation in
the States. So don't expect any answers from in that period!

/tonny

----------------------------------------

** The Major Changes **

The are some major changes between this version of the pathes and the
last version (called alpha 1).

The header field of all LOBJECT structure has disappeared! All the
common information, like marks, classes and slots, is now handled
transparently by lobject.c

The SHLIB does not have any built-in properties any more. They are now
slots that can be accessed with slot-value and (setf slot-value). See
below.

The name used for make-shlib does not need to be the full name of the
shared library file. If a name on the form "lib..." is specified, the
library is searched in a set of standard locations (see the shlib-path
variable).

----------------------------------------

** Future Plans **

I have a few ideas for the SHLIB module. After that I want to continue
with the CLOS part.

Ultimately, SHLIB could be a way to make a smaller XEmacs core, where
some of the advanced functionality is left out until needed. I have
made a small test with md5 as noted below. Which brinds up an
important question I hope somebody can answer: Which functionality to
include in base XEmacs and which to put in separate shared libraries?
The choise should be controled like databases are controlled
now. Obvious candidates are

- md5 (small)
- cde (small, but require external library)
- sound
- tooltalk
- glyph sub-types

Maybe even X11 and TTY support, tough these parts might be to mixed
with the rest of the base functionality.

I want to add support for solaris and linux in the next version
(sometime in the beginning of October). I have most of the code for
this from Tcl, but I need to make the Makefile's more flexible as the
.o files that will be part of the shared libraries must be compiled
differently on some systems (most notably solaris with the -PIC
switch).

SHLIB does not support defmumble yet. This will be supported for the
next version, as I have a rework of symeval.h and symbols.c in the
works. The macro interface (DEFVAR...&co) is the same as before :-/

Some operating systems does not support unloading of shared libraries
at all. These should define a special flag (in the src/s/...h file)
that disable all the unload functionality. Next version??

The naming of shlib functions is not consequent. Currently the name
shlib is used for most functions and variables, but Ben and Bill both
wished for a longer and more descriptive name like dynamic-lib or
shared-lib -- next version.

SHLIB does not handle documentation strings yet. I haven't thought too
much about this yet... I will need for do some major work on doc.c for
the CLOS implementation, so I have kind of postponed this... [But how
would you do it?]

Shared libraries should be autoloaded. I have started this, but I
haven't finished yet. It seems to be a simple extention to do_autoload
and Fautoload. I will add an autoload object type in the next version
as I need this for autoloable variables too -- should be part of the
next verison.

All these patches should be split up into smaller and more managable
pieces! For now they will stay together, as I don't have the time to
do anything about it.

----------------------------------------

** User's Guide (sort of) for the shared library module **

The new shared library module adds a new Emacs object type, shlib in
elisp and Lisp_Shlib i C, which is used to handle shared
libraries. A shlib object go through the following states:


       	       	       |
		       v
		 +> unloaded --> deleted -->
		 |     |
		 |     v
		 |  loaded
		 |     |
		 |     v
		 |  unloading
		 |     |
		 +-----+


The states are:

- unloaded: a shlib object exists for the shared library, but the
  library has not been loaded.

- loaded: the shared library has been successfully loaded into the
  XEmacs process

- unloading: XEmacs is trying to unload the shared library again. When
  there are no more references to data of the shared library the library
  is unloaded automatically

- deleted: the shlib object has been deleted and will be completely
  removed when there are no more references to the object.

It should be quite easy to use the shlib module (in my opinion at
least ;->), and I have actually already used it for some internal work
in the company I work for. I have also made a simple shared library
with the md5 functionality of XEmacs. Granted, this doesn't really do
anything special, but as it was very easy, it illustrates an idea I
have (see the notes below) and works just like the old C linked-in
versions with the addition of the following C function.

    #ifdef MAKE_SHLIB_MD5
    #include <shlib.h>

    Lisp_Object
    shlib_of_md5(Lisp_Object shlib, Lisp_Object function, Lisp_Object arg)
    {
      Lisp_Shlib	*shlibc = XSHLIB (shlib);

      if (EQ (function, Qload)) {
	syms_of_md5 ();
	vars_of_md5 ();
      } else if (EQ (function, Qtest_unload)) {
	return Qt;
      } else if (EQ (function, Qunload)) {
	return Qt;
      } else if (EQ (function, Qget_unloadable)) {
	return Qt;
      } else if (EQ (function, Qget_version)) {
	return make_int(SHLIB_VERSION);
      } else {
	signal_simple_error ("Illegal function", function);
      }
    }
    #endif

To use a shared library in XEamcs, you must first make a new shared
library object. This is done with

	(setq sl (make-shlib "libemacsmd5"))

The name specified as the argument of make-shlib can have one of two
forms: either it is the full pathname of the shared library file to
load, or it is the library part of a shared library file. In the later
case the shared library is searched for via the shlib-path variable
(just like load-path is searched). The suffix should be either ".so"
or ".so.1" [In the future the possible suffixes should be specified
via the src/s/ file like the rest of the configuration of the shared
library module.]

Most, if not all, information of a shlib object is available as slots
on the object. Some of the more important are shown below. Those that
are read-only is marked with (ro). [When the defclass stuff is ready,
these slots will have corresponding accessor functions, like (name sl)
and (setf (entry-name sl) "no_name").]

The name of the shared library file. (ro)

	(slot-value sl 'name)

The name of an entry function in the shared library that is called
when the library is loaded and unloaded. The default for the md5
library is "shlib_of_md5". The name can be changed as long as the
library isn't loaded. The usage of this function is described below.

	(slot-value sl 'entry-name)

The version of shlib interface utilized by the shared library. This
allow for older versions of shared libraries to co-exist with newer
versions of XEmacs. Note that the version number does *not* just
specify the shared library interface (in this case the interface to
the function "shlib_of_md5"), but also the interface from the shared
library to Emacs. (ro)

	(slot-value sl 'version)

A flag that shows if the shlib object is currently loaded. (ro)

	(slot-value sl 'loaded)

A flag that shows if the library can be unloaded once it has been
loaded. Not all libraries can be unloaded again. (ro)

	(slot-value sl 'unloadable)

A flag that shows if the shared library is currently being
unloaded. In most cases a shared library will be completely unloaded
at the next garbage-collection, but in some cases where the user has
made other alternative reference to the internal structures of the
shared library, that is impossible. Consider the case a shared library
exports a subr, e.g. "md5". If the user makes other reference to a
subr object, e.g. (fset 'my-md5 (symbol-function 'md5)), then shlib
must wait until that reference disappears agian - which can easily
take forever. (ro)

	(slot-value sl 'unloading)

A list of objects exported by shared library. While the library is
loaded this is a list of all objects exported via defsymbol,
defkeyword, defclass, defsubr and defmumble. While the shared library
is being unloaded, this is a list with the objects that was still
referenced after the last garbage-collection. (In case you wonder,
internaly this is implemented as a list while the shared library is
loaded and a weak list under unloading). The unload of a shared
library can not be completed until the list is empty. (ro)

	(slot-value sl 'objects)

A hook that is run after the library has been loaded. This allow a
developer to load other stuff at the load time or simply change
internal values. Use (add-hook (slot-value sl 'after-load-hook) 'some-hook)
to add a new hook function. (ro)

Similar hook exists for before-load, before-unload and
after-unload. The before-... hooks can decline the function by
returning nil.

	(symbol-value (slot-value sl 'after-load-hook))

[The function vars_of_shlib gives a complete list of all the defined
slots.]

Now we can load the shared library into the running emacs and add
the md5 function.

	(load-shlib sl)

And run the md5 function on something valuable ;->

	(md5 "Tonny Madsen")

When you don't want the shared library any more, it can be unloaded.

	(load-shlib sl)

The library will be completely unloaded as soon as all references to
internal data disappear as described above.

	(garbage-collect)

Now the library can be deleted

	(delete-shlib sl)

----------------------------------------

** User's Guide (sort of) for the CLOS module **

The CLOS support is still very, very limited. Only the following
functions are supported, but more will be present in the next
version. [This is where my current interest lies!]

    slot-value
    (setf slot-value)
    slot-boundp
    slot-makunbound
    slot-exists-p
    find-class
    (setf find-class)
    class-of
    object-list	(not really CLOS - what *is* the CLOS variant of this?)

If you want to know more about CLOS and CL, have a look at 

  http://www.harlequin.com/books/HyperSpec/FrontMatter/index.html

You will find many references to this hyper-book in the sources. I'm
current checking out whether Harlequin will accept this use of their
server or if they want us to have a separate copy of the standard, to
which we can reference.

Try to use

  (object-list (find-class 'standard-class))

The functions slot-value and (setf slot-value) can now be used to
inspect the classes. The current slots are (check clos.c for the
documentation, if any):

	class-name
	objects-per-frob
	slot-map
	slot-vector
	class-id
	class-type
	flags
	keep-free-list
	use-frob-blocks
	invisible-class
	invisible-objects
	protected-objects
	have-slots
	objects-in-use
	bytes-in-use
	objects-freed
	bytes-freed
	objects-on-free-list
	bytes-on-free-list

Hopefully the documentation will change alot for the next
version... Or the next++ version

----------------------------------------

** How to apply the patch **

To apply the patches get a clean 20.0-b26 distribution and use the
following commands

cd .../xemacs-20.0-b26
zcat xemacs-diff.gz | patch -f -N -p1

You can removed the src/lrecord.h file as it isn't used anymore...

SHLIB support is added with the configure switch '--with-shlib'.

No patches for 19.14 this time, sorry.

----------------------------------------

** Other Changes **

In order to make shlib I have made some other changes to XEmacs. The
most important, by far, is the addition of classes as first-class
XEmacs objects. I expect these to change into a scaled-down version of
CLOS as soon as I get the shlib module finished.

I have intergrated the two object types LHEADER and LCHEADER into one
new object type LOBJECT. We now have three object types. Check
lobject.h for documentation (though it might be a little out-of-date).

----------------------------------------

** SHLIB Low-Level Interface **

In order to use the shlib module on an architecture, a small set of
low-level functions must be created. The following is from shlib.h

shlib depends on a 5 low-level functions for the actual operations
on shared libraries. These functions must be defined in a separate
C file with the name "shlib-<type>.c". <type> describes the type of
the low-level interface and is defined in the relavant m/*.h header
files as the value of the preprocessor macro
SHLIB_TYPE. E.g. m/alpha.h contains the following lines.

#define SHLIB_TYPE dlopen

This line means that the interface for Digital Alpha machines is
found in the file shlib-dlopen.c

The low-level functions are

- void shlib_lowlevel_init()

This function is called when XEmacs is started. It is *not* called
until Emacs is initialized (i.e. it is not called when emacs is
dumped).

- void *shlib_lowlevel_load(char *name)

This function loads the library with the specified name into the
running XEmacs and returns a handle for the library. If name == 0,
a handles should be returned for the running XEmacs itself. This
handle should be usable with the unload and get functions below. If
an error occur (library does not exist, library not loadable, etc)
a null-pointer should be returned and the next call of the error
function should return a string that describes the error condition.

- void shlib_lowlevel_unload(void *handle)

This function unloads a library previously loaded with the load
function. After this call, the handle will never be used again and
any memory allocated for the handle should be released.

- void *shlib_lowlevel_get(void *handle, char *name)

This function returns the address of the function or variable with
the specified name in the specified library previously loaded with
the load function. Currently this function will only be used once
for each library, but don't depend on this. If an error occur
(function or variable does not exist, etc) a null-pointer should be
returned and the next call of the error function should return a
string that describes the error condition.

- char *shlib_lowlevel_error()

This function should return a string that describes the last error
encountered with the load or get functions as described above. The
function will only be called if an error has occured.

The following is the code corresponding to the dlopen library used
on Digital Alpha machines.

void
shlib_lowlevel_init()
{
}

void *
shlib_lowlevel_load(char *name)
{
  return dlopen (name, RTLD_NOW);
}

void
shlib_lowlevel_unload(void *handle)
{
  dlclose (handle);
}

void *
shlib_lowlevel_get(void *handle, char *name)
{
  return dlsym (handle, name);
}

char *
shlib_lowlevel_error()
{
  return dlerror ();
}

As it can be seen, this is real easy - at least for this architecture ;->.

The following lowlevel interfaces are been defined so far:

Interface	Machines
==================================================
dlopen	OSF/1

------------------------------------------------------------


------- End of forwarded message -------