Mapping desktop to a texture?

[BrickFarmer]

Is it possible from within Java LWJGL3 to map the entire desktop image for texturing within OpenGL.  I'm specifically thinking about mapping that texture into a VR HMD.  I found Java Robot, but that seems to be for capturing a single frame, and sounds slow (possibly 1fps) which would not work for video playback, and also seems to drop the mouse pointer (although I could track that myself and draw it again over the texture).  Or is this something I'd need to write my own C/C++ library for?

[BrickFarmer]

I think I found a solution with JavaCV, seems quite fast for capture, I'm just not sure of an effcient way to convert the bufferedImage and update a texture...

https://github.com/bytedeco/javacv

[EDIT: I tried with glTexImage2D per frame, bad idea.  I tried glTexSubImage2D but failed to get that to work.  So I read about using an FBO and udating that, but then I'm not sure how I go from FBO to just a smaller textured quad in my scene.  ie, i'll have to the update a regular texture anyway and be back to square one, or am I missing something with that approach?  I know this is not now strickly LWJGL related, unless there is a better way in LWJGL ]

[Cornix]

I dont think there is another way then using a texture. Can you show us your code for uploading the BufferedImage to the ogl texture? Maybe we can help you improve it.

[BrickFarmer]

I'm using the following code that I also used for loading textures from disk:

https://github.com/WhiteHexagon/example-lwjgl3-rift/blob/master/src/main/java/com/sunshineapps/riftexample/thirdparty/TextureLoader.java#L167

Then I just overwrite the texture data like this.

        BufferedImage image = grabber.fetchImage();
        if (image != null) {
            ByteBuffer textureBuffer = loader.convertImageData(image, desktopWall);
            int srcPixelFormat = GL11.GL_RGB;
            int dstPixelFormat = GL11.GL_RGBA;
          
            desktopWall.bind();
            GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, dstPixelFormat, loader.get2Fold(image.getWidth()), loader.get2Fold(image.getHeight()), 0, srcPixelFormat, GL11.GL_UNSIGNED_BYTE, textureBuffer);
//         //   GL11.glTexSubImage2D(GL11.GL_TEXTURE_2D, 0, 0, 0, loader.get2Fold(image.getWidth()), loader.get2Fold(image.getHeight()), srcPixelFormat, GL11.GL_UNSIGNED_BYTE, textureBuffer);
//                
            drawPlaneWallFront();
        }

I havent got very far with the alternative FBO approach so far since I suspect it will fall foul of the same perforamnce issue.  btw the above code runs, and I get something from the desktop displayed but the performance is maybe 2 fps and its stracted due to I suspect due the ^2 texture, the actual capture seems fast, so it is probably the convert or blit but my profiler expired ...

[Cornix]

Well, what does your "loader.convertImageData(image, desktopWall);" do?
You should try to profile your code to find out which parts are slow and which are fast. The question is what is the bigger problem: Reading from the BufferedImage or writing to the ogl texture?
Or maybe the conversion of the format? or maybe the screen capture?

You have to find out which of these is really causing the slowdowns.


If uploading the texture data is the problem you may try to do it in parallel. Have a separate thread write the texture data and use several textures as a ring buffer.