[RFE] LWJGL 3.0

[spasi]

* WORK IN PROGRESS * This post will be updated over the next days with additional info and results of the discussion that will follow.

So, it's time to start talking about LWJGL 3.0. For those that haven't heard it before, this next major release will not be backward compatible with the current one. The plan is to clean up some legacy stuff and improve the library design so that we can support features that are either impossible right now or too hard to implement. Any feedback on the proposed changes or suggestions for additional ones will be very welcome.

First, lets get something clear up front. The static nature of the AL/CL/GL bindings, which sets LWJGL apart from similar libraries, will NOT change. This means that when migrating to 3.0, required changes will be quite limited. Everything else is subject to discussion (context management, input APIs, utility classes, etc).

[ High Level Goals ]

- Migrate the codebase to Git and possibly host on Github. The existing codebase will remain on SVN/Sourceforge for critical/minor fixes only.
      The main incentive is to attract more contributions to LWJGL. Git is more popular than SVN, hence more likely to be already installed on a potential contributor's environment, removing one extra step from the process. It operates more efficiently than SVN and allows for cheap branching, essential for exploring new features. Github is a great hosting environment and free for OS projects. Even though alternatives exists, it's the most mature and popular one. The brilliant pull request automation is another feature that will hopefully attract more contributions.

- Improve context management. Allow LWJGL to more easily use external contexts. Better support multiple contexts.
      The goal here is to expose much more of the functionality currently hidden behind Context and Drawable. The critical change for this to happen is to make the native layer much lighter; we need to transform it to a simple binding layer, like the GL extensions, and move all the complicated functionality to Java code. In the process, we'll expose the low-level APIs so that advanced users will be able to take over context management and do things we didn't predict or didn't have the time to implement.

- Fully expose vendor-specific and OS-specific extensions, just like we do with normal GL extensions (with automated generation with the extension generator).
      Similar to the above goal, this will allow users to easily explore advanced features (multi-GPU solutions, video I/O, etc).

- Move the native windowing system to an independent package. Remove any dependencies between the windowing system and GL contexts. Make windowing system pluggable.
      Besides the obvious cleanup that is required, this is probably the most complicated problem we have. Discussion below.

- The native Display should be OOfied to allow for multiple native windows.
      The benefit for this is arguably limited, but since we're refactoring everything, it shouldn't be too hard to accomplish.

- Remove legacy features: pbuffers, AWT integration, the debug jar, the util package, deprecated functions.
      pbuffers are obsolete, AWT/Swing are basically dead (JavaFX is everywhere we want LWJGL to run anyway), the debug jar is useless since ARB_debug_output (and well, there are other debug tools too). I guess we should talk a bit about the util package, but imho it belongs to a higher-level library than LWJGL.

[ Technical Issues ]

- Bump minimum required JRE version to 6.0. Or maybe 7.0?

- Bump minimum required GL version from 1.1 to 2.0 (or 2.1?). This will allow us to unify the GL11..GL20 classes for convenience.

- The extension generator uses annotation processing through apt, which has been deprecated since Java 6 and removed in Java 8. It needs to be updated to use javax.annotation.processing instead. Another option is to re-implement the whole thing without annotations; the current code is very powerful but also a nightmare to understand and maintain. There are easier/cleaner ways to do what it does. This of course will require a different representation for the extension templates, something other than Java interface + annotations.

- I'd be nice to remove the license message that exists in every LWJGL source file. We could replace it with something much more compact, like a link to the actual license.

- Support for OpenGL ES should be cleaned up and no dependencies should exist between ES and desktop GL. Improve the build process and provide pre-built binaries if possible.

[ Other features (not critical for 3.0) ]

- Provide "profile-based" GL classes. An example would be a GLCore class that contains all the functions/tokens from 1.1 to 3.1 (or 3.2), without the deprecated stuff.

- Allow building LWJGL with extension filters. For example, a user might want a light build without any vendor-specific extensions, only core + ARB/EXT extensions.

- Implement JavaFX integration if technically possible.
   
[ Discussion ]

We have two major issues: What to do exactly about the windowing system and how to organize the library in general.

- Windowing System

There's the current solution and there are the alternatives. Obviously, what we have now is years of battle-tested code, it works well but it's also a THE major headache to maintain and improve. The multi-platform nature of the code and the fact that we're targetting Java developers has made it very hard over the years to find reliable people to contribute changes to the windowing system. On the other hand, having our own solution is an obvious huge benefit, so far we have never depended on a third-party for new features and fixes.

Alternatives exist. The JogAmp people seem to have done a good job with NEWT, though I haven't explored their code yet and am not sure if it covers everything we need. JavaFX is getting fully open sourced soon (last I heard was January?) and its Glass Windowing Toolkit is another solution. Though, we're not sure how full the open sourcing will be and if we'll be able to use it, both for technical and licensing reasons. Then there are various C/C++ based cross-platform windowing toolkits we could bind to. Again, we'd need to explore and choose a solution that fits our needs.

To me it seems that the best way forward would be to abstract away the windowing system. It would take too much time to decide and commit to a single solution. It's better to go through the pain of hidding the current solution behind an interface and enable pluggable windowing systems.

- Library Organization

A few issues here:

1. How to handle the util package and other utility classes. I would like to see a more clean separation between what's essential and what's an utility. The extreme would be to move everything out of LWJGL and create a separate library (maintained either by us or a third-party if there's interest) that takes care of simplifying people's life. This library would probably get richer than what we currently have, a propery vector lib, shader utilities, etc. Personally I would happily contribute to such a library (I got a lot of stuff hanging around that I use in my various projects). LWJGL would only keep the absolutely basics, like the BufferUtils class.

2. There's plenty of duplicated code in LWJGL that we haven't eliminated so far in the name of proper encapsulation (e.g. package private classes) or maybe safety. I'm tired of this bullshit tbh and I plan to just make stuff public and reusable. For where this code will reside we have 2 options: a) Create an "org.lwjgl.impl" package and b) Create a package that doesn't start with "org.lwjgl", equivalent to the com.sun or com.oracle packages in the JRE. In either case the message to the user would be: "don't care about what's in here, it's only stuff we use to implement the library". I've seen both ways in other projects and I personally don't care what we choose. Let me know if anyone has a preference either way or maybe a package name suggestion.

3. Deployment and modularity. Depending on what we decide with the windowing toolkits, the util package and the cleanup of dependencies in general, we might want to revisit how we build and deploy the library. Continue with one fat jar or split it up a bit?

[CodeBunny]

Can we not get rid of AWT support? Display.setParent is nice, especially for applets, and I'd be sad to not have an equivalent in 3.0.

[Liam]

Seconded, I vote for the interface solution and modify the existing AWT stuff to live behind a more general pluggable interface. This makes life easy for those of us depending on the current implementation and makes it easy for us to migrate to better solutions as they are added with minimal effort~

[spasi]

With a clean abstraction of the windowing system and the new context management, it should be relatively easy to re-implement AWT integration. Whether this happens within the library or through a 3rd party I'd like to leave open for now. The key point is to not "infect" the Display API with anything AWT related. An AWT Canvas is not a LWJGL Display and the Display stops being what it is when you use setParent.

Besides, there'll always be a pre-3.0 LWJGL available and hopefully we'll have JavaFX integration in 3.0 as well.

[kappa]

Very nice, agree with most.

1) The biggest issue is the native windowing, it needs to be moved out of and decoupled from the opengl package, the rest of the core library is pretty stable and needs little maintenance once implemented. I'd say just put it in something like org.lwjgl.display.*; so you'd have org.lwgl.display.Display, org.lwgl.display.DisplayImplementation and all the other classes needed to implement your own Display for a platform. Put each implementation in its own package (already done for native code but also needs to be done for the java side), e.g. org.lwjgl.display.windows;, org.lwjgl.display.linux, etc, this will allow programmers to easily hack on the platform they know without having to see or deal with any other platforms implementation or code.

2) Moving to GitHub is a good idea, SVN is now pretty much a stone age system in comparison.

3) Get rid of the all the utility packages, I'd say just put them in their own repo, pretty easy to do on github and still have them linked to the main LWJGL repo page. Just maintain them as separate projects with their own release cycle, its all Java code and as its utilities they don't need to be signed and the bits that aren't interesting to anyone can just die gracefully without causing code rot in the LWJGL core. The less there is in the main LWJGL repo the less there is to maintain.

4) Bumping to Java 6 shouldn't be a problem as Java 5 is pretty much dead market share wise.

5) As for annotations, wasn't apt mostly merged into javac? might be easier to use that then "javax.annotation.processing", not sure though.

[The below are just random idea's]

6) There should still be some sort of offscreen capability in the LWJGL core, maybe something based on FBO (possibly with a pbuffer fallback)? That way we can just grab the texture and write utilities to support all other windowing systems. Just pass the offscreen texture to AWT (Volatile Image), JavaFX, SWT, etc. Will have a tiny overhead but if its an idea that is workable would make life a lot easier rather than trying to maintain large chucks of code (including native) to support all the different frameworks, and as the texture is being drawn by the frameworks themselves, input, focus, etc would just work. Be nice to have the LWJGL core free from dependencies on any other framework (should make stuff like native compilation a lot easier).

7) Investigate the possibility of using something like the Waffle Project internally, GLX, WGL, CGL, EGL all need to be maintained separately and a pain as its native code which is different for each platform, if workable it'd mean a lot less maintenance for us and much cleaner code. The project is actively maintained by Intel and under a BSD Licence. There is already a push to try get Desktop OpenGL to use EGL, if such a move does happen (or someone needs the option), we'd already have the infrastructure to support this and just need to push a switch.

8 ) Just a pet peeve but guess i'll throw it out anyway, the name 'LWJGL' is quite a mouthful and not very brandable or memorable, how about dropping the 'W', since technically the word Lightweight is a single word, so it'll become 'LJGL'

I've got a more detailed list, which I'll post later once I find it.

[abcdef]

Might be a bit controversial and it will definitely be a big effort...but maybe having a lightweight physics library like JBullet (it hasn't been updated since 2010 so maybe a fork in to this library)?

Also the library currently has graphics, sound and parallel computing CL area's. But it doesn't have video (like xuggler), physics (as above), and probably some others. Is the plan to build out the area's LWJGL covers or to keep it to its core of opengl/al/cl?

[broumbroum]

...

- Improve context management. Allow LWJGL to more easily use external contexts. Better support multiple contexts.
      The goal here is to expose much more of the functionality currently hidden behind Context and Drawable. The critical change for this to happen is to make the native layer much lighter; we need to transform it to a simple binding layer, like the GL extensions, and move all the complicated functionality to Java code. In the process, we'll expose the low-level APIs so that advanced users will be able to take over context management and do things we didn't predict or didn't have the time to implement.

- Fully expose vendor-specific and OS-specific extensions, just like we do with normal GL extensions (with automated generation with the extension generator).
      Similar to the above goal, this will allow users to easily explore advanced features (multi-GPU solutions, video I/O, etc).
...

Hi spasi, I'd suggest a better multi-threading support for LWJGL. as you mentioned context switching and about multi-GPU support, here's something I want to ask : Is the following screen of piece of code from the "optimizing texture transfers" from nVidia GTC  '12 portable to LWJGL ?

capture thread	render thread	playout thread
// Wait for signal to start upload CPUWait(startUploadValid); glWaitSync(startUpload[2]); // Bind texture object BindTexture(capTex[2]); // Upload glTexSubImage(texID…); // Signal upload complete GLSync endUpload[2]= glFenceSync(…); CPUSignal(endUploadValid);	// Wait for download to complete CPUWait(endDownloadValid); glWaitSync(endDownload[3]); // Wait for upload to complete CPUWait(endUploadValid); glWaitSync(endUpload)[0]); // Bind render target glFramebufferTexture(playTex[3]); // Bind video capture source texture BindTexture(capTex[0]); // Draw // Signal next upload startUpload[0] = glFenceSync(…); CPUSignal(startUploadValid); // Signal next download startDownload[3] = glFenceSync(…); CPUSignal(startDownloadValid);	// Playout thread CPUWait(startDownloadValid); glWaitSync(startDownload[2]); // Readback glGetTexImage(playTex[2]); // Read pixels to PBO // Signal download complete endDownload[2] = glFenceSync(…); CPUSignal(endDownloadValid);

[spasi]

Yes it is. Just use SharedDrawables for the capture and playout threads. SharedDrawable will probably go away in 3.0, you'll have direct access to the GL contexts and you'll be able to share objects with the main thread context.

[princec]

Also the library currently has graphics, sound and parallel computing CL area's. But it doesn't have video (like xuggler), physics (as above), and probably some others. Is the plan to build out the area's LWJGL covers or to keep it to its core of opengl/al/cl?

Absolutely core only. One persons JBullet is another person's "YOU'RE DOING PHYSICS ALL WRONG!!!". Just look at the ridiculous situation with JogAmp for an example Let people maintain other libraries on top of LWJGL, not bundle them.

Cas

[tomb]

I have one request for the windowing api. Make it possible to get or calculate the screen space rectangle of the opengl canvas. For example by exposing the position and size of the of the window and the insets of the frame. We use head tracking with a large back projected wall at work. The location of the canvas on screen is needed to calculate the correct view frustum.

Support for multiple windows are also welcome as it makes it possible to setup a CAVE.

[kappa]

I have one request for the windowing api. Make it possible to get or calculate the screen space rectangle of the opengl canvas. For example by exposing the position and size of the of the window and the insets of the frame. We use head tracking with a large back projected wall at work. The location of the canvas on screen is needed to calculate the correct view frustum.

With LWJGL, you can get the size of the Display (since LWJGL 2.8.0) by using Display.getWidth() and Display.getHeight(), and you can get the position of the Display (since LWJGL 2.8.4) by using Display.getX() and Display.getY(), now sure if that is what you need or are asking?

[CodeBunny]

The getX() and getY() return the position of the window's corner, not the corner of the rendering area.

[kappa]

The getX() and getY() return the position of the window's corner, not the corner of the rendering area.

hmm, true, one quick trick you could use, grab the LWJGL Mouse location on the Display (Mouse.getX() and getY() this will be relative to the rendering area with (0,0) being top left corner of the rendering area). At the same time grab the screen position of Mouse (java.awt.MouseInfo.getPointerInfo().getLocation().getX() and getY() being relative to the screen top left being 0,0) and the Display position (Display.getX() and getY()). From these 3 points you can calculate the top left inset of the Display window.

Anyway better avoid derailing thread

[spasi]

6) There should still be some sort of offscreen capability in the LWJGL core, maybe something based on FBO (possibly with a pbuffer fallback)? That way we can just grab the texture and write utilities to support all other windowing systems. Just pass the offscreen texture to AWT (Volatile Image), JavaFX, SWT, etc.

See the attachment.

[kappa]

Wow, really cool, how did you do that? (technically)