LWJGL 3, VR Direct Mode & OpenVR

[phr00t]

I understand LWJGL 3 has some bindings for LibOVR. However, I'm working with Valve's OpenVR because I want to support more headsets (and operating systems, since LWJGL is cross-platform and all).

NVIDIA & AMD are announcing VR Direct modes, built into their drivers. Is this something LWJGL can hook into, so rendering can be done directly to a virtual reality headset? I'd prefer not to use the Oculus SDK, because I don't want any library to be specific to just the Rift & Windows if possible.

[kappa]

hey phr00t, noticed that you already had some OpenVR stuff working over on this thread, is this VR Direct stuff something different?

[phr00t]

Hey kappa!

Yep, that's me. Yes, I have quite a bit of OpenVR stuff working. The current VR Direct stuff "works" because it goes through the SteamVR Compositor. Developers submit their rendered, undistorted frames to the SteamVR Compositor, which then does a bunch of stuff including displaying the final result directly to a virtual reality headset. When a Rift is plugged in, the SteamVR Compositor forwards the texture to the Oculus Compositor, which does some more stuff like Timewarp & Vignette, before finally drawing it directly to the headset. It still needs some work, because it is "juddery".

Ultimately, I want more control over the final direct image sent to the headset & remove some of the "middle men" we are relying on. For example, to get VR Direct mode working on a Rift, it appears you have to use vignette & their distortion, which reduce FoV & is a bit of extra overhead. OpenVR has functions for creating a simple distortion mesh that uses the whole screen on any operating system -- quick and nice wide FoV. It'd also be nice to support more custom rendering methods in a more straight-forward pipeline:

LWJGL -> Direct to HMD

... instead of ...

LWJGL -> SteamVR Compositor (Windows only) -> Oculus Compositor (Windows only, Rift only) -> Direct to HMD

The straight-forward pipeline has always been possible using "Extended mode", but Oculus is doing away with that in their v0.7 SDK.

It seems like "Direct Ouput to HMD" would be a useful, built-in LWJGL feature similar to outputting to multiple monitors. I'm not sure if NVIDIA & AMD have (or will have) Direct drivers for Mac or Linux, but it also seems LWJGL would be a nice place to put in support as it comes.

[spasi]

Hey phr00t,

The plan is to have support for all VR APIs/SDKs. No one had asked about StreamVR until now, that's why there is no support. Hopefully, by early 2016 (when VR devices will be generally available to end-users) we'll have the necessary bindings in place.

If direct-to-HMD mode requires additional work on our side, we'll make it happen (or GLFW will).

I can't promise when StreamVR bindings will be ready, as I'm focusing on the 3.0 release right now. If you've got time for a contribution, you can start by copying and adapting the LibOVR binding. Let me know if you need help, I'll be glad to explain how the bindings generator works in detail.

It is actually best if someone other than me does this work, as I don't own a VR headset yet and don't plan to buy one until the final consumer devices are released.