Possible to use Bytebuffer directly for WirtableImage (JavaFx)?

[Lobo]

Hi,

I have a small question. I try to render stuff offline and show it asap at the moment at an image view in javaFX.

So at the moment I have two significant bottlenecks in my pipline. I need ~20ms to read and show a fullHD image.

The first bottleneck with ~10ms is glReadPixels(). I read a lot about why is it that slow and I think I can eliminate the overhead.
I'm using already a FBO to rende rin a texture, therefore I think I can probably just take this texture. Because at the moment I render in a
texture, then I show the texture at a fullscreen quad (offscreen) and then I'm using glReadPixels(). Sounds inefficent :-)

The second bottleneck is at the moment the point I have no idea if I can accelerate it.

This code needs about 10ms:

        private WritableImage writableImage = new WritableImage(Config.SCREEN_RESOLUTION_X, Config.SCREEN_RESOLUTION_Y);
	private byte[]        imageBuffer   = new byte[Config.SCREEN_RESOLUTION_X * Config.SCREEN_RESOLUTION_Y * 3];
	
	@Override
	public void afterRendering(ByteBuffer pixelData) 
	{
		pixelData.get(imageBuffer);
		PixelFormat<ByteBuffer> pixelFormat = PixelFormat.getByteRgbInstance();
		writableImage.getPixelWriter().setPixels(0, 0, Config.SCREEN_RESOLUTION_X, Config.SCREEN_RESOLUTION_Y, pixelFormat, imageBuffer, 0, Config.SCREEN_RESOLUTION_X * 3);
		pixelData.flip();
		iv_renderImage.setImage(writableImage);		
	}

The ByteBuffer is what I get after glReadPixels()

        private ByteBuffer getPixels()
	{
		if(glPixelsBuffer == null)
		{
			glPixelsBuffer = BufferUtils.createByteBuffer(Config.SCREEN_RESOLUTION_X * Config.SCREEN_RESOLUTION_Y * 3);
		}
		GL11.glReadPixels(0, 0, Config.SCREEN_RESOLUTION_X, Config.SCREEN_RESOLUTION_Y, GL12.GL_BGR, GL11.GL_UNSIGNED_BYTE, glPixelsBuffer);
		return glPixelsBuffer;
	}

My question now is, if there is a possibilty to bring the data from the ByteBuffer significantly faster to the WritableImage or is probably the ImageView not the best component in JFX for what I try?

The goal is at least to get below 16ms, to have the 60frames available without the rendering overhead. For sure the rendering needs also time, so each ms this copy process needs I have not for rendering if I want to beat the 60fps.

Would be really really great if someone give me a hint what I can do to speed up the copy from ByteBuffer to WritableImage.

Thanks a lot and merry christmas.

Best regards
Lobo

[Cornix]

So let me get this straight:
1) You render a scene into an OpenGL texture
2) You render the texture from 1) into an offscreen OpenGL window
3) You copy the data from the offscreen window into a client-side ByteBuffer
4) You copy the data from the ByteBuffer into a byte array
5) You copy the data from the byte array into the JavaFX image which, as far as I know, uses OpenGL under the hood
6) You render the JavaFX image into a window.

So you are basically copying the same data over and over and over again.
The first question you should aks yourself is: Do you really need to use OpenGL with LWJGl for this one? Why can you not directly use the rendering build into JavaFX?

[spasi]

What you describe is exactly what I tried to do better a few years ago with LWJGL-FX. Was written for LWJGL 2 but it's trivially portable to LWJGL 3.

It's usable if you're willing to sacrifice latency, but as long as you cannot avoid the GPU->CPU->GPU transfer, it's going to be horribly inefficient. There's no indication that JavaFX will ever support a more efficient way to interoperate with OpenGL/Vulkan. The only thing I've heard is they may add support for direct ByteBuffers to PixelRead/Writer. That's going to save a copy and... that's it.

[Lobo]

...
So you are basically copying the same data over and over and over again.
The first question you should aks yourself is: Do you really need to use OpenGL with LWJGl for this one? Why can you not directly use the rendering build into JavaFX?

Yes thats true. As I described I think I can redruce the overhead a little bit.
Why I cannot use the rendering build into JavaFX is realtive simple, as far as I know does JavaFX not support OpenGL.

Sure I can render stuff directly, but the main functionality is the usage of openGL and with this for sure GLSL shaders.

What you describe is exactly what I tried to do better a few years ago with LWJGL-FX. Was written for LWJGL 2 but it's trivially portable to LWJGL 3.

It's usable if you're willing to sacrifice latency, but as long as you cannot avoid the GPU->CPU->GPU transfer, it's going to be horribly inefficient. There's no indication that JavaFX will ever support a more efficient way to interoperate with OpenGL/Vulkan. The only thing I've heard is they may add support for direct ByteBuffers to PixelRead/Writer. That's going to save a copy and... that's it.

Thanks for your answer, but sad to hear. Then maybe I can reduce the glReadPixels overhead and have to life with the javafx overhead :-/

[Cornix]

Have you looked at this?
http://stackoverflow.com/questions/16946030/how-to-use-opengl-in-javafx

as far as I can tell JavaFX does use OpenGL.

[Lobo]

Have you looked at this?
http://stackoverflow.com/questions/16946030/how-to-use-opengl-in-javafx

as far as I can tell JavaFX does use OpenGL.

Yes I also found this post, but maybe I dont get it. As far as I see is there still no "native" opengl wrapper for java fx? So I have to use existing some java/OpenGL APIs or
have to write my own which is probably a waste of time. Because it seems to be that there are actually two really good APIs LWJGL and JOGL.

So I decided for the lwjgl API, because of the ongoing development and the possibilities to use openGL and OpenCL (and vulcan).

The only problem is to integrate an openGL window efficent into javafx. I also thought about using Swing, but swing is not that nice and the java support will end in the future.

Would be cool if javafx would work a little bit like QT but its not. I will try to understand how the example from Spasi will work and hopefully I#m able to adept it to lwjgl3.0 

Thanks for your help.

[spasi]

Have you looked at this?
http://stackoverflow.com/questions/16946030/how-to-use-opengl-in-javafx

as far as I can tell JavaFX does use OpenGL.

The JavaFX part that does rendering is called Prism. There are multiple Prism backend implementations and OpenGL is just one of them. There are software implementations, OpenGL implementations (used on Linux and macOS), OpenGL ES implementations (used on embedded systems, basically identical to the OpenGL implementation) and a Direct3D implementation. The Windows runtime does not ship with an OpenGL Prism backend, only software & Direct3D are available.

So you cannot depend on a specific implementation being available.

Even if you could, there's no public API that you can use to make integration possible. One alternative is to write your own Prism backend. I did that about a year ago. I copied the OpenGL ES backend and refactored it to use desktop OpenGL using LWJGL 3. It was relatively simple, though currently out of date (there are no compatibility guarantees and JavaFX developers are free to change Prism as necessary).

So, at the Prism level you can have a hardware accelerated context that you can hook into. The problem is, it's so abstracted away from the JavaFX API that you have no information on what it is you're rendering. If that information was available, then a high-performance integration would be possible. I guess it could be made to work right now with a few hacks, but I didn't have time to work on it any further.

[Cornix]

Well, this sounds like a wasted opportunity by the JavaFX folks.

[Lobo]

Just a question about JavaFX and OGL native support.

In the meantime I did it with the example from Spasi.

I googled a little bit but found nothing, did some one heared about further development in that direction?
I mean that javafx will extended in future to use opengl directly like for example QT at c++ side?
Or is there nothing about this on the horizont of the javafx roadmap?

Thanks