Hey Lankyware,
Does it happen much earlier if you add System.gc() calls (say, once per frame) in your application? It would indicate that a direct ByteBuffer that is still in use is being garbage-collected, and its associated native memory freed, by mistake. Note that a major issue with GC-managed ByteBuffers (allocated with ByteBuffer.allocateDirect or BufferUtils in LWJGL) is that they are not freed unless full GC cycles happen. The JDK uses Cleaner for this, which is a PhantomReference. Weak/soft/phantom reference processing happens during full GC pauses with most GCs. Such pauses are kind of rare on modern JVMs and that can explain unpredictable crashes at a later time.
I hope this helps.