Crash during allocation of VkInstance

[cpope9141]

Hello everyone.

I have an indie game on Steam (Area Zero), and it is written using the LWJGL. It is in early release, so I am actively collecting feedback from players. One player was recently updated to the latest patch via Steam. When they attempted to launch the new patch, the game crashed almost immediately. I created a debug build which included logging prints during initialization and sent it to the player (not through Steam). The logging ends as soon as a call is made to org.lwjgl.vulkan.VkInstance.VkInstance. According to the logs, vkCreateInstance returns a status code of VK_SUCCESS and the pointer buffer populated by vkCreateInstance appears to have a valid address (not null).

It seems strange that this player was running the game on a previous patch without issue and that I cannot recall any code changes that would affect the Vulkan instance creation. I decided to send this player a build of the last known working patch, but this time with debug logging included. Now it appears that the previous working patch crashes in the same way as the latest patch (at or during org.lwjgl.vulkan.VkInstance.VkInstance). So, the previous patch went from working to not and appears to be failing in the same way as the latest patch.

I am using Launch4j to create an executable that bundles jdk-23. Has anyone seen this issue or something similar? I am open to suggestions for additional debugging and willing to share any code that can clarify the issue. Thanks for reading and I hope you can help.

[spasi]

Hey cpope9141,

Without a stacktrace to examine, I'd say you're hitting a known bug in LWJGL 3.3.6 that has been fixed with https://github.com/LWJGL/lwjgl3/commit/b5b363e2. Please try upgrading to 3.4.0-snapshot.

[cpope9141]

Thanks for the response spasi.

It appears I may be experiencing the same crash after updating my NVIDA driver. Rolling back the driver to what was previously installed stops the crash from occurring. Is this consistent with the "fix(Vulkan) VkExtensionProperties array allocation" commit? That is, would the updated driver be returning an array of extensions too large to be allocated on the stack?

[spasi]

Yes, the extensions array is now allocated normally.

[cpope9141]

I have updated lwjgl.jar (with natives) to 3.4.0-snapshot. Before updating my Nvidia driver, the game continued to work (as expected). After updating the driver, I see a crash and this stack trace:

Exception in thread "main" java.lang.OutOfMemoryError: Out of stack space.
    at org.lwjgl.system.MemoryStack.nmalloc(MemoryStack.java:321)
    at org.lwjgl.vulkan.VkExtensionProperties.malloc(VkExtensionProperties.java:208)
    at org.lwjgl.vulkan.VkInstance.getAvailableDeviceExtensions(VkInstance.java:82)
    at org.lwjgl.vulkan.VkInstance.getInstanceCapabilities(VkInstance.java:45)
    at org.lwjgl.vulkan.VkInstance.<init>(VkInstance.java:29)
    at renderer.vulkan.VulkanInstance.create(VulkanInstance.java:77)
    at renderer.RendererCore.create(RendererCore.java:238)
    at renderer.Renderer.create(Renderer.java:544)
    at game.Game.run(Game.java:47)
    at game.Game.main(Game.java:15)

If I also update lwjgl-vulkan.jar to 3.4.0-snapshot, I continue to see a crash, but it is past the vulkan instance creation during physical device selection:

Exception in thread "main" java.lang.OutOfMemoryError: Out of stack space.
    at org.lwjgl.system.MemoryStack.nmalloc(MemoryStack.java:321)
    at org.lwjgl.vulkan.VkExtensionProperties.malloc(VkExtensionProperties.java:200)
    at renderer.vulkan.physicalDevice.PhysicalDevice.checkDeviceExtensionSupport(PhysicalDevice.java:156)
    at renderer.vulkan.physicalDevice.PhysicalDevice.meetsRequirments(PhysicalDevice.java:234)
    at renderer.vulkan.physicalDevice.PhysicalDevice.selectPhysicalDevice(PhysicalDevice.java:119)
    at renderer.RendererCore.create(RendererCore.java:239)
    at renderer.Renderer.create(Renderer.java:544)
    at game.Game.run(Game.java:47)
    at game.Game.main(Game.java:15)

I am using "try(MemoryStack stack = stackPush())" to create the stack passed into VkExtensionProperties.malloc. It appears that the capacity being requested from the stack at the time of the crash is only (252 * SIZEOF).

[spasi]

That's the same bug, just in your code instead of inside LWJGL. VkExtensionProperties is a relatively big struct (260 bytes), so 252 such structs make it easy to overflow the default MemoryStack size (64kb).

[cpope9141]

So, I need to set the stack capacity with something like:

MemoryStack stack = MemoryStack.create(capacity).push(); {
//use stack
} stack.pop();

The above seems to be working for me, but I'd be happy for someone to double check that I am not misusing something.

[cpope9141]

Do you have a timeline for when the 3.4.0-snapshot will be upgraded to release? If not soon, how much of a risk would it be for me to use the snapshots of lwjgl.jar and lwjgl-vulkan.jar in their current state?

[spasi]

No timeline yet. I think you only need lwjgl-vulkan.jar and it should work fine with the 3.3.6 core. However, I don't see a reason to not switch to 3.4.0-snapshot altogether. LWJGL snapshots are very stable due to its nature: most changes happen in the bindings & native libraries and those are always stable releases.

MemoryStack stack = MemoryStack.create(capacity).push(); {
//use stack
} stack.pop();

This will work, but it's wasteful (a GC-managed ByteBuffer is allocated internally). I would recommend sticking to the default stack for everything, except the extension properties array. This is what LWJGL does now:

try (VkExtensionProperties.Buffer extensions = VkExtensionProperties.malloc(capacity)) {
  // ...
}

This does explicit malloc/free.

[cpope9141]

Thank you for your help, spasi!