LWJGL Forum

Programming => Lightweight Java Gaming Library => Topic started by: Orangy Tang on December 03, 2010, 00:27:55

Title: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 03, 2010, 00:27:55
I have a user on 64bit linux reporting the following exception on startup:

Exception in thread "main" java.lang.UnsatisfiedLinkError: /home/minecraft/.tectonicus/native/liblwjgl.so: /home/minecraft/.tectonicus/native/liblwjgl.so: wrong ELF class: ELFCLASS32 (Possible cause: architecture word width mismatch)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
        at java.lang.Runtime.load0(Runtime.java:770)
        at java.lang.System.load(System.java:1003)
        at org.lwjgl.Sys$1.run(Sys.java:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.lwjgl.Sys.doLoadLibrary(Sys.java:66)
        at org.lwjgl.Sys.loadLibrary(Sys.java:82)
        at org.lwjgl.Sys.<clinit>(Sys.java:99)
        at org.lwjgl.opengl.Display.<clinit>(Display.java:130)


What I do is ship the native files dll/so/etc. inside my jar, and then extract them to a temporary directory at startup, then set the system property. All this happens before any LWJGL calls.

private void unpackLwjgl()
{
String[] files =
{
"jinput-dx8_64.dll",
"jinput-dx8.dll",
"jinput-raw_64.dll",
"jinput-raw.dll",
"libjinput-linux.so",
"libjinput-linux64.so",
"libjinput-osx.jnilib",
"liblwjgl.jnilib",
"liblwjgl.so",
"liblwjgl64.so",
"libopenal.so",
"libopenal64.so",
"lwjgl.dll",
"lwjgl64.dll",
"openal.dylib",
"OpenAL32.dll",
"OpenAL64.dll"
};

File cacheDir = new File(System.getProperty("user.home"), ".tectonicus/native");
cacheDir.mkdirs();
for (String f : files)
{
File outFile = new File(cacheDir, f);
FileUtils.extractResource("Native/"+f, outFile);
}

String nativePath = cacheDir.getAbsolutePath();
System.setProperty("org.lwjgl.librarypath", nativePath);
System.setProperty("net.java.games.input.librarypath", nativePath);
}


This works on OSX, WinXp, Win7 (personally tested) and I believe on 32bit linux,  but not on 64bit linux. It looks like it's trying to use the non-64bit native library.

I'm not sure what would cause this behaviour - is there another step I'm missing? Alternatively, does anyone know how I can get more information on exactly what might be going wrong?

Thanks.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: kappa on December 03, 2010, 00:35:55
which version of LWJGL are you using? I remember this was a bug on the early 2.x releases but was fixed in a subsequent LWJGL release.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Matthias on December 03, 2010, 00:47:55
This is the error for which I wrote that RFE (http://lwjgl.org/forum/index.php/topic,3585.0.html) :) While the original reporter still has not tested it.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 03, 2010, 01:34:07
That does sound like my issue (running headless on ubuntu 64bit).

I don't have a linux box handy - if you could provide me with a binary build I could get my user to test it and report back? That would be greatly appreciated.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Matthias on December 03, 2010, 01:44:19
Quote from: Orangy Tang on December 03, 2010, 01:34:07
I don't have a linux box handy - if you could provide me with a binary build I could get my user to test it and report back? That would be greatly appreciated.

I send you the link to the lwjgl.jar via PM. It is a patch LWJGL 2.6
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 03, 2010, 11:11:57
Hmm, no dice. It now fails with this exception:

org.lwjgl.LWJGLException: X Error - disp: 0x7f65b01dce10 serial: 58 error: BadRequest (invalid request code or no such operation) request_code: 155 minor_code: 179
        at org.lwjgl.opengl.LinuxDisplay.globalErrorHandler(LinuxDisplay.java:277)
        at org.lwjgl.opengl.GL11.nglGetError(Native Method)
        at org.lwjgl.opengl.GL11.glGetError(GL11.java:1280)
        at org.lwjgl.opengl.Util.checkGLError(Util.java:57)


However I don't know if that's a related error or a new one. Is there anything else I can try? (note this happens after pbuffer creation, so it's getting past that point at least).
Title: Re: Runtime native extraction fails on 64bit linux
Post by: jediTofu on December 03, 2010, 20:44:59
Just for testing purposes, you could try having your "String[] files" array only contain the 64-bit linux versions, and try using both your original lwjgl.jar and the one by Matthias.

What is the code at the "BadRequest" line?
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 04, 2010, 00:39:43
Quote from: jediTofu on December 03, 2010, 20:44:59
Just for testing purposes, you could try having your "String[] files" array only contain the 64-bit linux versions, and try using both your original lwjgl.jar and the one by Matthias.

What is the code at the "BadRequest" line?
I think it's finding the right version now (since it's not complaining about the architecture version mismatch), but I'm not sure what this new error is.

The calling code is actually a direct call to checkGLError, just after I've done some vertex array setup. Since it happens on the first frame, i suspect it's the first time checkGlError is being called after Pbuffer.makeCurrent().

Some googling suggests that 'request code 155' might be a X11 code for 'bad gl context', but I don't know what the second code is, nor why I'm getting 'bad gl context' after seemingly creating a pbuffer without a problem.

Hopefully someone more familiar with X and linux can give me some pointers.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: jediTofu on December 12, 2010, 09:00:50
Is there a way I can try the actual program?

I finally got around to testing LWJGL on Linux Mint 64-bit (Ubuntu Based).  I did all of the LWJGL demos, and they worked fine.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 12, 2010, 10:39:56
Quote from: jediTofu on December 12, 2010, 09:00:50
Is there a way I can try the actual program?

I finally got around to testing LWJGL on Linux Mint 64-bit (Ubuntu Based).  I did all of the LWJGL demos, and they worked fine.
Sure, you can download it from the first post in this thread: http://www.minecraftforum.net/viewtopic.php?f=25&t=95739

Also contains an ever growing number of people on linux having display exceptions too. :(
Title: Re: Runtime native extraction fails on 64bit linux
Post by: jediTofu on December 12, 2010, 22:18:08
It worked for me; there's probably more that I need to do though or something.

Info:

java -version output:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

uname -a output:
Linux mint 2.6.35-22-generic #33-Ubuntu SMP Sun Sep 19 20:32:27 UTC 2010 x86_64 GNU/Linux


Here's what I did:

I used MCmapper (http://www.neurogrill.net/mc/mcmapper.htm) to create a level for me (i.e., level.dat).

Then I played Single Player (http://www.minecraft.net/) in order to get minecraft.jar.
Then I searched my file system for it and ran this command after I found it:

java -jar Tectonicus.jar minecraftJar=/tmp/www.minecraft.net/Minecraft/minecraft.jar  worldDir=./world outputDir=./map numSamples=0
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 12, 2010, 22:45:51
Well I'm glad it's not totally buggered on linux - I've got the usual problem that I'm only really hearing a lot of noise from people who it doesn't work for, so it's nice to see a report of it working cleanly.

I'll admit to feeling a little lost, especially on reports like this one: http://www.minecraftforum.net/viewtopic.php?f=25&t=95739&start=210#p1532749
Title: Re: Runtime native extraction fails on 64bit linux
Post by: jediTofu on December 13, 2010, 03:06:15
Quote from: Orangy Tang on December 12, 2010, 22:45:51
Well I'm glad it's not totally buggered on linux - I've got the usual problem that I'm only really hearing a lot of noise from people who it doesn't work for, so it's nice to see a report of it working cleanly.

I'll admit to feeling a little lost, especially on reports like this one: http://www.minecraftforum.net/viewtopic.php?f=25&t=95739&start=210#p1532749

Hmmm, ok, have you had them try using sudo?

sudo java -jar Tectonicus.jar ...

If that doesn't work, then you'll have to modify Sys.java to have 'System.loadLibrary("awt");' like the bug link you posted in your other thread, which would make this an actual bug.

Out of curiosity, the patched lwjgl Matthias did work for the original problem?  I guess that means that patch needs to be added in the main version as well.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 13, 2010, 10:03:00
1. Matthias's jar patch does solve the original problem for some people, yes. So that definitely needs to go in.

2. I'll ask them about sudo.

3. I've already manually added a Toolkit.getDefaultToolkit() (supposed to be more effective than System.loadLibrary("awt") since it copes better with headless environments), but that didn't help. :(
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 13, 2010, 13:56:41
Further info:

Quotesudo didn't help.

It is a headless 32-bit Linux Amazon EC2 instance

http://aws.amazon.com/free/

Which is interesting, since I assumed it was a 64bit issue.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: jediTofu on December 13, 2010, 19:51:24
So he's using some type of virtual system to put linux on?  Sorry, I'm not familiar with EC2.  I wonder if the LWJGL demos even work on that.  Worst-case scenario, if it's just on EC2, tell people not to use EC2 lol.

I think that I already saw this answered, but it happens using both Open Java and Sun Java, correct?

That's odd that the toolkit solution didn't work, so that might actually not be the bug.  You should be able to try the "awt" solution as well without building LWJGL:

class ... {
  static {
    System.loadLibrary("awt");
    System.setProperty("org.lwjgl.librarypath","...");
    ...
  }
}
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 13, 2010, 20:15:43
Quote from: jediTofu on December 13, 2010, 19:51:24
So he's using some type of virtual system to put linux on?  Sorry, I'm not familiar with EC2.  I wonder if the LWJGL demos even work on that.  Worst-case scenario, if it's just on EC2, tell people not to use EC2 lol.
EC2 isn't that complicated really - you just ask amazon for a virtual box of a particular size (micro in this case) and pick one of their preconfigured machine images. But I don't have any experience of it myself. They're almost certainly headless, so probably don't have X installed by default.

I suspect it's more than just an EC2 bug though, because I've had quite a few reports from different people, not all of which are on EC2.

QuoteI think that I already saw this answered, but it happens using both Open Java and Sun Java, correct?
I believe so, but I'm horribly confused at the moment as I've got lots of different people all reporting variants on the same bug, so I've kinda lost track. ???

I'll try the loadLibrary("awt") thing but I'm not hopeful...
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 14, 2010, 10:06:30
Ok, two user reports, two posts. User one, previously:


System:
   OS Name: Linux
   OS Architecture: i386
   OS Version: 2.6.32-26.1.BHsmp
   Java vendor: Sun Microsystems Inc.
   Java version: 1.6.0_22
   Awt toolkit: null
   Headless?: null
java.lang.UnsatisfiedLinkError: /home2/exiledhe/.tectonicus/native/liblwjgl.so: /home2/exiledhe/mc/jre/jre1.6.0_22/lib/i386/libjawt.so: symbol awt_FreeDrawingSurface, version SUNWprivate_1.1 not defined in file libmawt.so with link time reference
   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
   at java.lang.ClassLoader.loadLibrary0(Unknown Source)
   at java.lang.ClassLoader.loadLibrary(Unknown Source)
   at java.lang.Runtime.load0(Unknown Source)
   at java.lang.System.load(Unknown Source)
   at org.lwjgl.Sys$1.run(Sys.java:70)
   at java.security.AccessController.doPrivileged(Native Method)
   at org.lwjgl.Sys.doLoadLibrary(Sys.java:66)
   at org.lwjgl.Sys.loadLibrary(Sys.java:93)
   at org.lwjgl.Sys.<clinit>(Sys.java:110)
   at org.lwjgl.opengl.Display.<clinit>(Display.java:130)
   at tectonicus.TileRenderer.<init>(TileRenderer.java:82)
   at tectonicus.TectonicusApp.run(TectonicusApp.java:244)
   at tectonicus.TectonicusApp.main(TectonicusApp.java:398)


And with System.loadLibrary("awt"):

System:
        OS Name: Linux
        OS Architecture: i386
        OS Version: 2.6.32-26.1.BHsmp
        Java vendor: Sun Microsystems Inc.
        Java version: 1.6.0_22
        Awt toolkit: null
        Headless?: null
Exception in thread "main" java.lang.UnsatisfiedLinkError: Native Library /home2/exiledhe/mc/jre/jre1.6.0_22/lib/i386/libawt.so already loaded in another classloader
        at java.lang.ClassLoader.loadLibrary0(Unknown Source)
        at java.lang.ClassLoader.loadLibrary(Unknown Source)
        at java.lang.Runtime.loadLibrary0(Unknown Source)
        at java.lang.System.loadLibrary(Unknown Source)
        at tectonicus.TectonicusApp.main(TectonicusApp.java:392)


So I think we can rule out awt not being loaded. Odd that 'awt toolkit' is null - that's the result of System.getProperty("java.awt.toolkit").
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 14, 2010, 10:30:51

User 2 has the same result as well. First on OpenJDK:

System:
        OS Name: Linux
        OS Architecture: i386
        OS Version: 2.6.34.7-56.40.amzn1.i686
        Java vendor: Sun Microsystems Inc.
        Java version: 1.6.0_20
        Awt toolkit: null
        Headless?: null
Exception in thread "main" java.lang.UnsatisfiedLinkError: /home/ec2-user/.tectonicus/native/liblwjgl.so: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386/libjawt.so: symbol awt_Unlock, version SUNWprivate_1.1 not defined in file libmawt.so with link time reference
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1750)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1646)
        at java.lang.Runtime.load0(Runtime.java:787)
        at java.lang.System.load(System.java:1022)
        at org.lwjgl.Sys$1.run(Sys.java:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.lwjgl.Sys.doLoadLibrary(Sys.java:66)
        at org.lwjgl.Sys.loadLibrary(Sys.java:93)
        at org.lwjgl.Sys.<clinit>(Sys.java:110)
        at org.lwjgl.opengl.Display.<clinit>(Display.java:130)
        at tectonicus.TileRenderer.<init>(TileRenderer.java:82)
        at tectonicus.TectonicusApp.run(TectonicusApp.java:244)
        at tectonicus.TectonicusApp.main(TectonicusApp.java:398)


And then on the sun VM:

System:
        OS Name: Linux
        OS Architecture: i386
        OS Version: 2.6.34.7-56.40.amzn1.i686
        Java vendor: Sun Microsystems Inc.
        Java version: 1.6.0_23
        Awt toolkit: null
        Headless?: null
Settings:
        interactive:false
        worldDir:/home/ec2-user/minecraft/world
        outputDir:/home/ec2-user/minecraft/carto/.
        useCache:true
        cacheDir:/home/ec2-user/minecraft/carto/./cache
        minecraftJar:/home/ec2-user/minecraft/minecraft_server.jar
        texturePack:none
        colourDepth:24
        alphaBits:8
        numSamples:4
        cameraAngle:45
        closestZoomSize:12
        numZoomLevels:8
        imageFormat:png
        eraseOutputDir:false
        extractLwjglNatives:true
        isVerbose:false
        forceLoadAwt:false
        force32BitNatives:true
        force64BitNatives:false

Forcing 32-bit native libraries
Cache directory found and is valid. Using cache at /home/ec2-user/minecraft/carto/./cache
Exception in thread "main" java.lang.UnsatisfiedLinkError: /home/ec2-user/.tectonicus/native/liblwjgl.so: /home/ec2-user/java/jdk1.6.0_23/jre/lib/i386/libjawt.so: symbol awt_FreeDrawingSurface, version SUNWprivate_1.1 not defined in file libmawt.so with link time reference
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
        at java.lang.Runtime.load0(Runtime.java:770)
        at java.lang.System.load(System.java:1003)
        at org.lwjgl.Sys$1.run(Sys.java:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.lwjgl.Sys.doLoadLibrary(Sys.java:66)
        at org.lwjgl.Sys.loadLibrary(Sys.java:93)
        at org.lwjgl.Sys.<clinit>(Sys.java:110)
        at org.lwjgl.opengl.Display.<clinit>(Display.java:130)
        at tectonicus.TileRenderer.<init>(TileRenderer.java:82)
        at tectonicus.TectonicusApp.run(TectonicusApp.java:244)
        at tectonicus.TectonicusApp.main(TectonicusApp.java:398)


Again, forcing Awt to be loaded just gives the "awt already loaded" exception.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Matzon on December 14, 2010, 10:59:06
I assume all of these are on headless environments? if not, either use a terminal launched from X or export the display
Try seeing if xvfb works?
Alternatively, the workaround mentioned that they "might" want to fix this for Java7 - so the latest OpenJDK might have a fix for it.

Anyhow, if someone can create a small executable that dynamically loads libjawt.so in a headless environment preferably by also dlsym JAWT_GetAWT and calling it, then I will try and see it it works for lwjgl too. I did not have any success with it.
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Orangy Tang on December 14, 2010, 11:11:49
Yes, these are all headless environments. I'll see if they can try the latest OpenJDK.

The sun bug seems to have some code for dynamically loading the libraries at runtime, does that do what you're looking for?

http://bugs.sun.com/view_bug.do?bug_id=6539705

Thanks for looking into this. :)
Title: Re: Runtime native extraction fails on 64bit linux
Post by: Matzon on December 14, 2010, 11:21:19
No, I have not been able to make it work with the workaround. Not in LWJGL nor in a seperate test system.
From my POV, this is an Oracle bug, with no clear fix.

See more here: http://lwjgl.org/forum/index.php/topic,3585.msg20180.html#msg20180

So again, try:
Title: Re: Runtime native extraction fails on 64bit linux
Post by: creatis on June 30, 2011, 07:51:09
Hi,

We've been running into the same error trying to load VTK native libraries from a java program executed on a headless box:

public class Test {
        public static void main(String [] args){
System.loadLibrary("vtkCommonJava");
         System.loadLibrary("vtkFilteringJava");
         System.loadLibrary("vtkIOJava");
         System.loadLibrary("vtkImagingJava");
         System.loadLibrary("vtkGraphicsJava");
         System.loadLibrary("vtkRenderingJava");
        }
}

produces:

java Test
Exception in thread "main" java.lang.UnsatisfiedLinkError: /usr/lib64/libvtkRenderingJava.so: /usr/java/jre1.6.0_24/lib/amd64/libjawt.so: symbol awt_FreeDrawingSurface, version SUNWprivate_1.1 not defined in file libmawt.so with link time reference
   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
   at java.lang.ClassLoader.loadLibrary0(Unknown Source)
   at java.lang.ClassLoader.loadLibrary(Unknown Source)
   at java.lang.Runtime.loadLibrary0(Unknown Source)
   at java.lang.System.loadLibrary(Unknown Source)
   at Test.main(Test.java:10)

The described workarounds (xvfb and  static{System.loadLibrary("awt");}) didn't work but we solved the issue using the following trick:

[root@kingkong amd64]# pwd
/usr/java/jre1.6.0_24/lib/amd64
[root@kingkong amd64]#
[root@kingkong amd64]# ls -l headless/ xawt/
headless/:
total 36
lrwxrwxrwx 1 root root    47 Jun 30 09:47 libmawt.so -> /usr/java/jre1.6.0_24/lib/amd64/xawt/libmawt.so
-rwxr-xr-x 1 root root 33892 Feb  3 02:25 libmawt.so.old

xawt/:
total 388
-rwxr-xr-x 1 root root 392981 Feb  3 02:25 libmawt.so

This way we're sure to get rid of the buggy headless libmawt ;-)

We have little experience with this lib though; do you think always using the xawt instead of the headless could crash anything?