[Solved] Errors when using newer version of LWJGL 3

[bobjob]

I have been using:
LWJGL 3.1.1 SNAPSHOT build 1

and my code works fine

I then switched to:
LWJGL 3.1.2 build 29

And I am now getting temperamental errors.
My program will work sometimes, but then other times it will either crash
or output something like to many errors to report:


The only changes required in my code where as follows
this:

  			ByteBuffer newImage =
    					Stdlib.malloc(scaleWidth * scaleHeight * comp.get(0));

to this:

    			ByteBuffer newImage = 
    					LibCStdlib.malloc(scaleWidth * scaleHeight * comp.get(0));

Also another error I have observed is that sometimes only every second mouse click works



I have attached the error log on the last crash

[bobjob]

I also get this output:

java(590,0x7fffcb4d73c0) malloc: *** error for object 0x7fd2f4e445a8: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

Im guessing its a memory leak, I just don't understand why It wasn't causing errors before

[spasi]

Sounds like you're allocating with one memory allocator and freeing with another.

The memAlloc*/memFree methods in MemoryUtil use a configurable memory allocator. It can be the system allocator (from the LibCStdlib class), jemalloc (from the JEmalloc class), or a user-specified allocator. This allocator is also used internally by LWJGL in various places. For example, when you call .free() on a struct, it assumes that the struct has been allocated with memAlloc and calls memFree. If MemoryUtil has been configured to use jemalloc and you had allocated the struct using direct calls to LibCStdlib, then the free is going to crash.

Suggestions that may fix the problem:

- Do not use the allocator bindings directly, always use MemoryUtil.
- If you do use an allocator directly, make sure MemoryUtil is configured to use the same allocator.

[bobjob]

Thanks for you help spasi. I am trying to track down all the calls to allocate and free memory. It seems as though all of them are using memory utils. The only issue I can think that may be causing this is that I am allocate memory in a seperate thread, and then after the object is loaded I am freeing it in the main thread.

how do I "set a breakpoint in malloc_error_break to debug"?
I would really like to know which free/alloc is causing this error. As its temperamental and doesn't happen all the time which makes it very hard to locate

[spasi]

That message comes from jemalloc, so it's not really applicable. You can set a Java breakpoint to memFree though, should achieve the same result.

You can also try -Dorg.lwjgl.util.DebugAllocator=true and -Dorg.lwjgl.util.Debug=true. It will report memory leaks on exit (for memAlloc only).

[bobjob]

THANK YOU! that helps a lot. getting a lot more useful debug info

[bobjob]

Im pretty sure the issue is related to multi-thread. which explains why it only happens sometimes:

This is the code that Im using to load in another thread. If I do this in the main thread, the program works every time.
But when I move it back into the loading thread, it crashes sometimes

ByteBuffer ttf = IOUtil.ioResourceToByteBuffer(name, 160 * 1024);
			
			
			STBTruetype.stbtt_PackBegin(pc, bitmap, BITMAP_W, BITMAP_H, 0, 1);
			
			for ( int i = 0; i < scale.length; i++ ) {
				int z = 0;
				for (int s = 0; s < samples; s++) {
					if (sf[s + (samples * i)] == 0) {
						int p = (i * samples + z) * 128 + 32;
						chardata.limit(p + 95);
						chardata.position(p);
						org.lwjgl.stb.STBTruetype.stbtt_PackSetOversampling(pc, 1, 1);
						org.lwjgl.stb.STBTruetype.stbtt_PackFontRange(pc, ttf, 0, scale[i], 32, chardata);
						z++;
					}
					if (sf[s + (samples * i)] == 1) {
						int p = (i * samples + z) * 128 + 32;
						chardata.limit(p + 95);
						chardata.position(p);
						org.lwjgl.stb.STBTruetype.stbtt_PackSetOversampling(pc, 2, 2);
						org.lwjgl.stb.STBTruetype.stbtt_PackFontRange(pc, ttf, 0, scale[i], 32, chardata);
						z++;
					}
					if (sf[s + (samples * i)] == 2) {
						int p = (i * samples + z) * 128 + 32;
						chardata.limit(p + 95);
						chardata.position(p);
						org.lwjgl.stb.STBTruetype.stbtt_PackSetOversampling(pc, 3, 1);
						org.lwjgl.stb.STBTruetype.stbtt_PackFontRange(pc, ttf, 0, scale[i], 32, chardata);
						z++;
					}
				}
				
			}
			
			
			STBTruetype.stbtt_PackEnd(pc);
			
			chardata.position(0);

if a call to memory allocate is called at the same time from 2 different threads, will this likely be the cause of the error?
Should all memory allocation be done on a single thread?

This is strange as this issue didn't exist in the previous version (LWJGL 3.1.1 SNAPSHOT build 1), any thoughts on why this may be the case?

[Cornix]

Have you tried synchronising your calls to allocate? Does that change anything?

[bobjob]

not yet. I was hoping for a simpler solution. As I assume to synchronise all memory allocation I would have to 'lock' all areas over all possible code that uses memory allocation in order to prevent any conflicts, Im guessing that would take a while. I also assume it will take some guessing without insight into their implementation as to know which methods also allocate or free memory within them such as "stbtt_PackEnd"

I attached the console error output (I didnt include it all as it keeps going for a while before terminating)

[spasi]

if a call to memory allocate is called at the same time from 2 different threads, will this likely be the cause of the error?
Should all memory allocation be done on a single thread?

No and no. Memory allocators are thread-safe. Obviously there are situations that will blow up (using allocated memory after free, or doing a double-free), but it's not the allocator's problem. Almost all allocators have a fast-path when doing malloc/free in the same thread, but it's a performance optimization only.

ByteBuffer ttf = IOUtil.ioResourceToByteBuffer(name, 160 * 1024);

Are you holding on to this buffer while rendering? This is a common mistake when using stb_truetype. The buffer must not be freed, its data will be accessed by some stb_truetype functions, even after packing.

[bobjob]

I dont hold onto the bytebuffer.

the 'ByteBuffer ttf' is created using BufferUtils not MemoryUtils. So if anything holds a copy of it, it will retain the object. Note nothing requires the buffer to be passed to it later.

I do hold onto private STBTTPackedchar.Buffer chardata; which is allocated using:
STBTTPackedchar.malloc(scale.length * samples * 128);

I am still confused as to how this code works fine when this function is called on the main thread, but crashes only sometimes when loaded over a seperate thread.

I have made sure not to access any of the data until it is fully loaded. I'm also confused as to how a version change will allow for this bug.

[spasi]

I dont hold onto the bytebuffer.

Do you create an STBTTFontinfo to read metrics etc? If yes, then the buffer must stay alive while the STBTTFontinfo is used.

the 'ByteBuffer ttf' is created using BufferUtils not MemoryUtils.

...

I am still confused as to how this code works fine when this function is called on the main thread, but crashes only sometimes when loaded over a seperate thread.

I have made sure not to access any of the data until it is fully loaded. I'm also confused as to how a version change will allow for this bug.

It sounds like the crash is related to GC. Buffers allocated via BufferUtils are subject to GC, but due to how the JVM handles direct buffers, it might take a while (minimum 2 GC cycles) before the native memory is freed. This means that unrelated changes to your program, or doing things in different thread, or even a new JVM version, might affect when GC kicks in and when the memory is reclaimed. Which then affects when and how your program will crash (if at all).

[bobjob]

These are the classes I use to load the true type font.

import org.lwjgl.stb.STBTTAlignedQuad;
import org.lwjgl.stb.STBTTPackContext;
import org.lwjgl.stb.STBTTPackedchar;
import org.lwjgl.stb.STBTruetype;

Should I consider trying this for debugging:
download the LWJGL source (java)

edit the method:

public static long nmemAlloc(long size) {
        return ALLOCATOR.malloc(size);
}

and turn it into:

Object lock;
public static long nmemAlloc(long size) {
     synchronized(lock) {
         return ALLOCATOR.malloc(size);
     }
}

[spasi]

If you suspect a bug in jemalloc, you can revert to the system allocator with -Dorg.lwjgl.system.allocator=system.

To eliminate my suspicion about this being GC related, enable GC output with -XX:+PrintGC (or -Xlog:gc if you're on Java 9) and check if the crash happens immediately after a GC cycle.

[bobjob]

when I put in -Dorg.lwjgl.system.allocator=system
the problem goes away


Do you still want me to try the GC check?
Im using Java 8. Where do I put those options?