glMapBufferRange is limited to 2GB because of LWJGL

[freduni]

Hi,

According to the OpenGL spec, glMapBufferRange can access the contents of any buffer, including buffers larger than 2GB:

https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glMapBufferRange.xhtml

(GLintptr and GLsizeiptr are 64 bits on a 64-bit system, which LWJGL appropriately takes care of through 'long' parameters)

In LWJGL, however, glMapBufferRange returns a java.nio.ByteBuffer, which is a big issue.

A ByteBuffer has attributes such as 'position', 'limit' and 'capacity', that are defined in byte units. Unfortunately, ByteBuffer use 'ints' (integers) to define these values.

Because of this, ByteBuffer access won't work with buffer larger than 2GB.

I tested the following : if you call glMapBufferRange with a very large region (much larger than 2GB), it succeeds (as per OpenGL spec). However, the returned ByteBuffer is useless, as it won't work past its 'limit'.

I cannot use .asIntBuffer() as a workaround, as the returned ByteBuffer capacity is negative, and invalid.

What can I do to use glMapBufferRange with large buffers?

Thanks,
Fred

[KaiHH]

Accessing native/off-heap memory bigger than this limit is indeed a big issue in Java and known for quite some time in the Java world, because (currently) the NIO Buffer API is the only standard way to do that. Everything else is hacked using mostly sun.misc.Unsafe API. So, it's not so much a limitation of LWJGL as it is one of the Java platform and its API. LWJGL just provides methods returning the "standard" way of working with native/off-heap memory, by returning java.nio.ByteBuffer instances.

However, as a work-around for the lacking NIO Buffer API, LWJGL provides "unsafe" methods (with a lowercase 'N' prefix): https://javadoc.lwjgl.org/org/lwjgl/opengl/GL30C.html#nglMapBufferRange(int,long,long,int)
It returns the direct virtual memory address of the mapped memory region.
Obviously, you can also not use a single NIO Buffer to read from or write to those memory regions, so you have to either use org.lwjgl.system.MemoryUtil.memPut/Get*(address + offset, value) or create a NIO Buffer from a particular region using MemoryUtil.memByteBuffer(address, size).
Another option is to "page" the mappings: Map the first 2GB, write/read it, then map the next 2GB, and so forth.

[spasi]

This will be resolved in LWJGL 4. NIO buffers in the LWJGL API will be replaced by JEP 383's types, which natively support 64-bit indexing.

Until then, KaiHH's suggestions are valid alternatives.

Also note that, with the way LWJGL constructs buffer instances, it is technically possible to create an IntBuffer that addresses 8GB of memory or a LongBuffer that addresses 16GB of memory. You cannot do that with the JDK API. However, you'd still need to slice smaller chunks to access byte-level data.

[freduni]

Thank you both. It works.
Didn't know about the unsafe methods until now!