Modification to the way libraries load

[breath]

I would like to change the way that Sys.java loads the native libraries.  I want to be able to load the libraries in my own code, perhaps by extracting them from a JAR first.  This would give more control over packaging options.

Right now Sys.java calls Sys.loadLibrary regardless of whether the libraries are already loaded or not.  So we want to change it so that it only loads the library if it's not already loaded.  Here's my current strategy (this is the static initializer block in Sys.java)

static {
		implementation = createImplementation();
		try {
			// try a method to see if native lib already loaded
			implementation.getNativeLibraryVersion();
		} catch(UnsatisfiedLinkError er) {
			// not loaded, should go ahead and load it here
			AccessController.doPrivileged(new PrivilegedAction() {
				public Object run() {
					System.loadLibrary(LIBRARY_NAME);
					return null;
				}
			});
		}
		// at this point, the library should definitely be loaded
		String native_version = implementation.getNativeLibraryVersion();
		if (!native_version.equals(VERSION))
			throw new LinkageError("Version mismatch: jar version is '" + VERSION +
                             "', native libary version is '" + native_version + "'");
		implementation.setDebug(LWJGLUtil.DEBUG);
	}

The try/catch block tries a native method to see if the library is already loaded.  If it's not loaded, an UnsatisfiedLinkError is thrown, and the library gets loaded in the catch block.  I built LWJGL using this code, and it works correctly.  It still loads the libraries on its own (you still have to set java.library.path correctly), and if you load lwjgl.dll/.so in your own code prior to executing this block, it doesn't try to load it a second time.

What do you think?  Could we get this into .99?

[elias]

Not an unreasonable request, but how about this alternative: The System.loadLibrary in turn calls the ClassLoader.findLibrary to get a path to the requested library. The solution would be to define your own classloader that on-demand unpacks the requested library and return the path to it on an overridden findLibrary().

The advantages of this approach are that:

1. We won't have to change LWJGL
2. The same scheme will work for other libraries (openal, devil, fmod)

- elias

[breath]

That's a clever idea.  I salute your knowledge of Java's internals.  It's going to be pretty tricky to get it to work right.  I'll give it a try and let you know how it went.

[breath]

It turns out that in order for calls to loadLibrary to get routed through your custom findLibrary on your custom ClassLoader, the custom ClassLoader has to actually do the work of finding and loading the class that calls findLibrary.  It's very weird.  If you subclass ClassLoader and just have loadClass call super.loadClass(), the class that you have thus loaded will route its calls to findLibrary through some other classloader.  The magic where your specific classloader is associated with the class that it creates is, I think, in the method defineClass, which takes in an array of bytes.

So I've spent some time writing a ClassLoader that basically duplicates the behavior of the system classloader.  However, it's choking on one file: ContextCapabilities in lwjgl.jar.  Here's the error I'm getting:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at cyclescore.init.CustomLoader.main(CustomLoader.java:163)

Caused by: java.lang.ClassFormatError: Illegal UTF8 string in constant pool in class file org/lwjgl/opengl/ContextCapabilities
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.lang.ClassLoader.defineClass(Unknown Source)
        at CustomLoader.loadClass(CustomLoader.java:74)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClassInternal(Unknown Source)
	at org.lwjgl.opengl.GLContext.useContext(GLContext.java:294)
	at org.lwjgl.opengl.Context.makeCurrent(Context.java:182)
	at org.lwjgl.opengl.Display.makeCurrent(Display.java:599)
	at org.lwjgl.opengl.Display.createWindow(Display.java:266)
	at org.lwjgl.opengl.Display.create(Display.java:657)
	at org.lwjgl.opengl.Display.create(Display.java:630)
	at org.lwjgl.opengl.Display.create(Display.java:614)
        ... 5 more

I wish it would tell me which UTF8 string is illegal, but defineClass is pretty much a black box to me.

Here's the source for my classloader:

import java.io.InputStream;
import java.lang.reflect.Method;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.net.URLConnection;
import java.util.Hashtable;

public class CustomLoader extends URLClassLoader {

	protected char sepChar;

	protected Hashtable cache;

	public CustomLoader(URL[] t) {
		super(t);
		cache = new Hashtable();
		sepChar = System.getProperty("file.separator").charAt(0);
		loadClasspath(System.getProperty("java.class.path"));
		// add current class directory
		addURL(getClass().getProtectionDomain().getCodeSource().getLocation());
	}

	protected void loadClasspath(String classpath) {
		String separator = System.getProperty("path.separator");
		String[] paths = classpath.split(separator);
		for (int i = 0; i < paths.length; i++) {
			URL url = urlFromPath(paths[i]);
			addURL(url);
		}
	}

	protected URL urlFromPath(String path) {
		URL retval;
		try {
			// get ready to peek into jars
			if (path.endsWith(".jar")) {
				path = slash(path);
				retval = new URL("jar:file:/" + path + "!/");
			} else {
				// treat it as a file
				path = slash(path);
				retval = new URL("file:/" + path);
			}
		} catch (MalformedURLException e) {
			e.printStackTrace();
			return null;
		}
		return retval;
	}

	protected String slash(String input) {
		return input.replace(sepChar, '/');
	}

	protected synchronized Class loadClass(String name, boolean resolve)
			throws ClassNotFoundException {
		Class retval = (Class) cache.get(name);
		if (retval != null) {
			// in the cache
			System.out.println("Found in cache " + name);
			return retval;
		}

		byte[] data = loadClassBytes(name);
		if (data != null) {
			retval = defineClass(name, data, 0, data.length);

			cache.put(name, retval);
			System.out.println("Loaded " + name);
		} else {
			// can't find the class, delegate to system classloader
			System.out.println("Delegating " + name);
			retval = findSystemClass(name);
		}
		// ----- Resolve if necessary
		if (resolve) {
			resolveClass(retval);
		}
		return retval;
	}
	
	protected byte[] loadClassBytes(String className) {
		className = formatClassName(className);
		try {
			URLConnection connection = findConnection(className);
			if (connection == null) {
				return null;
			}
			InputStream inputStream = connection.getInputStream();
			int length = connection.getContentLength();
			byte[] data = new byte[length];
			while (length > 0) {
				int bytes_read = inputStream.read(data, data.length - length,
						length);
				if (bytes_read == -1) {
					throw new IOException("Unexpected EOF in loading class "
							+ className);
				}
				length -= bytes_read;
			}
			inputStream.close();
			return data;
		} catch (Exception ex) {
			ex.printStackTrace();
			return null;
		}
	}

	protected String formatClassName(String className) {
		// convert binary notation to URL path
		return className.replace('.', '/') + ".class";
	}

	protected URLConnection findConnection(String pathName) {
		URL[] urls = getURLs();
		// look through all the URLs we have for the class
		for (int i = 0; i < urls.length; i++) {
			try {
				URL url = new URL(urls[i], pathName);
				// detect if the class exists by whether the connection works
				URLConnection connection = url.openConnection();
				if (connection.getContentLength() > 0) {
					return connection;
				}
			} catch (Exception e) {
				e.printStackTrace();
				continue;
			}
		}
		return null;
	}
	
	protected String findLibrary(String arg0) {
		System.out.println("Find library " + arg0);
		String retval = super.findLibrary(arg0);
		System.out.println("Found library " + retval);
		return retval;
	}

	public static void main(String[] args) {
		CustomLoader cl = new CustomLoader(new URL[] {});
		String classname = "project.Main";
		// Load the main class through our CCL
		try {
			Class clas = cl.loadClass(classname);
			// Use reflection to call its main() method, and to
			// pass the arguments in.
			// Get a class representing the type of the main method's argument
			Class mainArgType[] = { (new String[0]).getClass() };
			// Find the standard main method in the class
			Method main = clas.getMethod("main", mainArgType);
			// Create a list containing the arguments -- in this case,
			// an array of strings
			Object argsArray[] = { args };
			// Call the method
			main.invoke(null, argsArray);
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}

Any ideas on what's wrong?

Update: Changed the code in loadClassBytes per elias's suggestion.  It does work now, and I give permission to anyone to use this code for whatever reason.

[elias]

I'm guessing that you don't read the entire file with input.read(). Here's my section of the Tribal Trouble class loader:

           URLConnection conn = class_url.openConnection();
            InputStream in = conn.getInputStream();
            int length = conn.getContentLength();
            byte[] buffer = new byte[length];
            while (length > 0) {
                int bytes_read = in.read(buffer, buffer.length - length, length);
                if (bytes_read == -1)
                    throw new IOException("EOF reached");
                length -= bytes_read;
            }

Notice the the while loop to make sure every byte is read. Hope it helps.

- elias

[breath]

That did it.  I can't believe I missed that.  Thanks for your help!  You have a sharp eye.  I updated my post above to have working code, so anyone who's interested in writing their own library loader has a good starting point to work off of.

Do you do interesting things with class loading for Tribal Trouble, or are you just aping the standard loader like I am?  I imagine that a souped-up classloader would be useful for managing plugins.