Binding LWJGL and Swing/AWT (easily)

[gregorypierce]

So I've gone ahead and taken a stab at binding these two together. The process is really trivial once you have the framebuffer that you want to draw to the screen. Since we're not binding this to an image (and we don't really need to in this case), we can dump the raw pixels from glReadPixels into the graphics draw call.

The following code compiles but I haven't begun testing it and it has a glaring error in that it doesn't make a call to LWJGL for rendering the scene or anything but that's a small issue. Just wanted to post this so people can start to chime in if they want. I expect to have a complete adapter class done and ready to go into CVS by the end of the weekend so we can get past the "no Swing/AWT" functionality problem

import org.apache.log4j.*;

import org.lwjgl.opengl.Pbuffer;
import org.lwjgl.opengl.PixelFormat;
import org.lwjgl.opengl.GL11;
import org.lwjgl.BufferUtils;

import javax.swing.*;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.awt.event.ComponentListener;
import java.awt.event.ComponentEvent;
import java.nio.ByteBuffer;

public class LWJGLAdapter extends JComponent implements ComponentListener
{
    public static Logger logger = Logger.getLogger("LWJGLAdapter");

    private Pbuffer                     pbuffer;
    private ByteBuffer                  glFrameBuffer;

    public LWJGLAdapter()
    {
        try
        {
            // create an OpenGL LWJGL PBuffer
            //
            pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );

            // make the PBuffer current
            //
            pbuffer.makeCurrent();

            // create a frame buffer to hold the frame from LWJGL
            //
            glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 3 );

        }
        catch( Exception e )
        {
            logger.error("Adpater error ", e );
        }
    }

    protected void paintComponent( Graphics g )
    {
        // read the contents of the pbuffer into the frame buffer
        //
        GL11.glReadPixels(0, 0, getWidth(), getHeight(), GL11.GL_RGB, GL11.GL_BYTE, glFrameBuffer );

        // use the components paint command to draw the framebuffer
        //
        g.drawBytes( glFrameBuffer.array(), 0, 0, 0, 0 );
    }

    /**
     * Invoked when the component's size changes.
     */
    public void componentResized(ComponentEvent event)
    {
        // recreate the pbuffer for the new component size
        try
        {
            pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );
            pbuffer.makeCurrent();
            glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 3);
        }
        catch( Exception e )
        {
            logger.error("Unable to recreate the OpenGL components after the component resized", e );
        }
    }

    /**
     * Invoked when the component's position changes.
     */
    public void componentMoved(ComponentEvent e)
    {

    }

    /**
     * Invoked when the component has been made visible.
     */
    public void componentShown(ComponentEvent event)
    {
        try
        {
            // just in case we've broken some textures or in case some other GL thing became current
            // while we were hidden
            pbuffer.makeCurrent();
        }
        catch( Exception e )
        {
            logger.error("Unable to make the pbuffer current after component shown", e);
        }
    }

    /**
     * Invoked when the component has been made invisible.
     */
    public void componentHidden(ComponentEvent e)
    {

    }
}

[princec]

Er, Gregory, Graphics.drawBytes draws a string. You're going to have to do something funky with Images whether you like it or not :/

Cas

[Matzon]

ROFL

[gregorypierce]

Yes, I am going to draw a string that represents the frame - don't question my weird science!

[spasi]

I just saw this

http://www.javagaming.org/cgi-bin/JGNetForums/YaBB.cgi?board=jogl;action=display;num=1106324571;start=0#0

and felt kind of sorry...

I really would like to see a fully working AWT/SWT/whatever binding to LWJGL, so that we can bury JOGL once and for all. I can't believe how many are wasting time with it.

[Chman]

The jME team (it was renanse I think) has already made it... And it works well ! (except that you can't use PBuffers).

You could take a look at the code... it could be helpful.

Chman

[princec]

Oooh! So jME has an AWT  "LWJGLComponent" does it? That I could bung in, say, an Applet?

Cas

[Chman]

It might work in an applet
JME use a thing called Headless Rendering and it uses a lot of JME specific thingies. About the core part, it simply use a rendering to texture process and draw the images on a Canvas or JCanvas...

The only problem is that you need a video card that can make a fast render to texture.

Chman

[princec]

Ahh, no good. That's what Gregory's trying to do, and it's not really going to be fast enough. I need GL rendering direct to a backbuffer context.

Cas

[gregorypierce]

If you're rendering directly to the backbuffer you're a heavyweight component and you might as well just use a native window.

[Chman]

Ahh, no good. That's what Gregory's trying to do, and it's not really going to be fast enough. I need GL rendering direct to a backbuffer context.

Cas

Not so slow
Here I get about 100 fps... With 10 or 10000 cubes I get the same number of fps, rendering the scene is as fast as rendering it to screen, it's just the render-to-texture process that slow the thing up...

I'm currently running on an ATI 9800Pro@XT.

I've run some JME + Swing test on a Geforce 4 MX and it runs pretty well

I think this method could be a "wait for" a true rendering to an AWT/Swing canvas...

Chman

[gregorypierce]

Yeah. Its slower than a native render but its not THAT slow. It is more than usable. Its just the conversion of the framebuffer into something that can be painted in a bufferedimage takes a tad of time.

No biggie though. Its either that or have only native windows

[gregorypierce]

import org.apache.log4j.*;

import org.lwjgl.opengl.Pbuffer;
import org.lwjgl.opengl.PixelFormat;
import org.lwjgl.opengl.GL11;
import org.lwjgl.BufferUtils;

import javax.swing.*;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.awt.image.WritableRaster;
import java.awt.event.ComponentListener;
import java.awt.event.ComponentEvent;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;

public class LWJGLAdapter extends JComponent implements ComponentListener
{
    public static Logger logger = Logger.getLogger("LWJGLAdapter");

    private Pbuffer                     pbuffer;
    private IntBuffer                   glFrameBuffer;
    private BufferedImage               bufferedImage;
    private RenderingThread             renderingThread;

    class RenderingThread extends Thread
    {
        private int             frameRate = 30;
        private int             sleepTime;


        public RenderingThread()
        {
            this.sleepTime = (int)(1000/frameRate);
        }

        public RenderingThread( int frameRate )
        {
            this.frameRate = frameRate;
            this.sleepTime = (int)(1000/frameRate);
        }

        public void run()
        {
            try
            {
                sleep( sleepTime );
                repaint();
            }
            catch( Exception e )
            {
                logger.error("Failed to repaint screen due to exception ", e );
            }
        }

        public int getFrameRate()
        {
            return frameRate;
        }

        public void setFrameRate(int frameRate)
        {
            this.frameRate = frameRate;
        }
    }

    public LWJGLAdapter()
    {
        try
        {
            // create an OpenGL LWJGL PBuffer
            //
            pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );

            // make the PBuffer current
            //
            pbuffer.makeCurrent();

            // create a frame buffer to hold the frame from LWJGL
            //
            glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 4 ).asIntBuffer();

            // create a buffered image that will hold the frame buffer contents
            //
            bufferedImage = new BufferedImage( getWidth(), getHeight(), BufferedImage.TYPE_INT_RGB );

            renderingThread = new RenderingThread( 60 );

            // everything should be ready to render before this call is made. The component should have bee
            // added and packed so that it has a valid width() and height()
            //
            renderingThread.start();
        }
        catch( Exception e )
        {
            logger.error("Adpater error ", e );
        }
    }

    public int getFrameRate()
    {
        return renderingThread.getFrameRate();
    }

    public void setFrameRate(int frameRate)
    {
        renderingThread.setFrameRate( frameRate );
    }

    protected void paintComponent( Graphics g )
    {
        // read the contents of the pbuffer into the frame buffer
        //
        GL11.glReadPixels(0, 0, getWidth(), getHeight(), GL11.GL_RGB, GL11.GL_UNSIGNED_BYTE, glFrameBuffer );

        // dump the contents of the frame buffer into the buffered image - there should be a faster way to do
        // this with the writable raster which should give us direct access
        //
        bufferedImage.setRGB( 0, 0, getWidth(), getHeight(), glFrameBuffer.array(), 0, getWidth() );

        // use the components paint command to draw the framebuffer
        //
        g.drawImage( bufferedImage, 0, 0, getWidth(), getHeight(), getBackground(), null );
    }

    /**
     * Invoked when the component's size changes.
     */
    public void componentResized(ComponentEvent event)
    {
        // recreate the pbuffer for the new component size
        try
        {
            pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );
            pbuffer.makeCurrent();
            glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 4).asIntBuffer();
        }
        catch( Exception e )
        {
            logger.error("Unable to recreate the OpenGL components after the component resized", e );
        }
    }

    /**
     * Invoked when the component's position changes.
     */
    public void componentMoved(ComponentEvent e)
    {

    }

    /**
     * Invoked when the component has been made visible.
     */
    public void componentShown(ComponentEvent event)
    {
        try
        {
            // just in case we've broken some textures or in case some other GL thing became current
            // while we were hidden
            pbuffer.makeCurrent();
        }
        catch( Exception e )
        {
            logger.error("Unable to make the pbuffer current after component shown", e);
        }
    }

    /**
     * Invoked when the component has been made invisible.
     */
    public void componentHidden(ComponentEvent e)
    {

    }
}

[gregorypierce]

I've been working with Cas a lot today and he's been very helpful getting me through the more interesting parts of converting the byte[] into something that we can actually draw.

/*
* Copyright (c) 2005 Your Corporation. All Rights Reserved.
*/
package com.sojournermobile.client;


import org.apache.log4j.*;

import org.lwjgl.opengl.Pbuffer;
import org.lwjgl.opengl.PixelFormat;
import org.lwjgl.opengl.GL11;
import org.lwjgl.BufferUtils;

import javax.swing.*;
import java.awt.*;
import java.awt.color.ColorSpace;
import java.awt.image.*;
import java.awt.event.ComponentListener;
import java.awt.event.ComponentEvent;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.nio.ByteOrder;

public class TestPBuffer extends JComponent
{
   public static Logger logger = Logger.getLogger("TestPBuffer");

   private Pbuffer                     pbuffer;
   private ByteBuffer                  glFrameBuffer;
   private byte[]                      frameBytes;
   private BufferedImage               bufferedImage;
   private RenderingThread             renderingThread;
   private boolean                     isRenderable = false;
   private WritableRaster              writableRaster;

   private int                         width;
   private int                         height;

   class RenderingThread extends Thread
   {
       private int             frameRate = 5;
       private int             sleepTime;


       public RenderingThread()
       {
           this.sleepTime = (int)(1000/frameRate);
       }

       public RenderingThread( int frameRate )
       {
           this.frameRate = frameRate;
           this.sleepTime = (int)(1000/frameRate);
       }

       public void run()
       {
           try
           {
               sleep( sleepTime );
               repaint();
           }
           catch( Exception e )
           {
               logger.error("Failed to repaint screen due to exception ", e );
           }
       }

       public int getFrameRate()
       {
           return frameRate;
       }

       public void setFrameRate(int frameRate)
       {
           this.frameRate = frameRate;
       }
   }

   public TestPBuffer( int width, int height )
   {
       this.width = width;
       this.height = height;
       try
       {
           // create an OpenGL LWJGL PBuffer
           //
           pbuffer = new Pbuffer( width, height, new PixelFormat(), null, null );

           // make the PBuffer current
           //
           pbuffer.makeCurrent();

           // create a buffered image that will hold the frame buffer contents
           //
           ColorModel colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB), true, false, ColorModel.OPAQUE, DataBuffer.TYPE_BYTE);
           writableRaster = colorModel.createCompatibleWritableRaster( width, height );
           bufferedImage =  new BufferedImage(colorModel, writableRaster, false, null);

           renderingThread = new RenderingThread();

           isRenderable = true;

       }
       catch( Exception e )
       {
           logger.error("Adpater error ", e );
       }
   }

   public void start()
   {
       if ( isRenderable )
       {
           // everything should be ready to render before this call is made. The component should have been
           // added and packed so that it has a valid width() and height()
           //
           renderingThread.start();
       }
   }

   public int getFrameRate()
   {
       return renderingThread.getFrameRate();
   }

   public void setFrameRate(int frameRate)
   {
       renderingThread.setFrameRate( frameRate );
   }

   protected void paintComponent( Graphics g )
   {
       g.setColor( Color.BLACK );
       g.fillRect(0, 0, width, height);
       
       
       try
       {
           pbuffer.makeCurrent();
       }
       catch( Exception e ){}

       if ( glFrameBuffer == null )
       {
           // create a frame buffer to hold the frame from LWJGL
           //
           glFrameBuffer = BufferUtils.createByteBuffer( width * height * 4 );
       }

       renderScene();

       GL11.glReadPixels(0, 0, width, height, GL11.GL_RGBA, GL11.GL_UNSIGNED_BYTE, glFrameBuffer );

       glFrameBuffer.get( ((DataBufferByte)writableRaster.getDataBuffer()).getData() );
       glFrameBuffer.rewind();


       // draw the image
       g.drawImage( bufferedImage, 0, 0, width, height, getBackground(), null );
   }

   public void renderScene()
   {
       GL11.glClear(GL11.GL_COLOR_BUFFER_BIT | GL11.GL_DEPTH_BUFFER_BIT);          // Clear The Screen And The Depth Buffer
   }

   /**
    * Invoked when the component's size changes.
    */
   public void componentResized(ComponentEvent event)
   {
       // recreate the pbuffer for the new component size
       try
       {
           //pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );
           //pbuffer.makeCurrent();
           //glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 3);
       }
       catch( Exception e )
       {
           logger.error("Unable to recreate the OpenGL components after the component resized", e );
       }
   }

   /**
    * Invoked when the component's position changes.
    */
   public void componentMoved(ComponentEvent e)
   {

   }

   /**
    * Invoked when the component has been made visible.
    */
   public void componentShown(ComponentEvent event)
   {
       try
       {
           // just in case we've broken some textures or in case some other GL thing became current
           // while we were hidden
           pbuffer.makeCurrent();
       }
       catch( Exception e )
       {
           logger.error("Unable to make the pbuffer current after component shown", e);
       }
   }

   /**
    * Invoked when the component has been made invisible.
    */
   public void componentHidden(ComponentEvent e)
   {

   }

   public static void main( String[] args )
   {
       BasicConfigurator.configure();

       try
       {
           JFrame frame = new JFrame();
           frame.setSize( 640, 480 );
           TestPBuffer client = new TestPBuffer(640,480);

           frame.getContentPane().add( client );
           //frame.pack();
           frame.show();

           frame.setDefaultCloseOperation( JFrame.DISPOSE_ON_CLOSE );

           client.start();
       }
       catch( Exception e )
       {
           logger.error("Error building the client ", e );
       }
   }
}

I expect the next code update will be the final and we'll have something that will allow Swing and actually browser compatibility.

[gregorypierce]

I've been working with Cas a lot today and he's been very helpful getting me through the more interesting parts of converting the byte[] into something that we can actually draw.

/*
 * Copyright (c) 2005 Your Corporation. All Rights Reserved.
 */
package com.sojournermobile.client;


import org.apache.log4j.*;

import org.lwjgl.opengl.Pbuffer;
import org.lwjgl.opengl.PixelFormat;
import org.lwjgl.opengl.GL11;
import org.lwjgl.BufferUtils;

import javax.swing.*;
import java.awt.*;
import java.awt.color.ColorSpace;
import java.awt.image.*;
import java.awt.event.ComponentListener;
import java.awt.event.ComponentEvent;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.nio.ByteOrder;

public class TestPBuffer extends JComponent
{
    public static Logger logger = Logger.getLogger("TestPBuffer");

    private Pbuffer                     pbuffer;
    private ByteBuffer                  glFrameBuffer;
    private byte[]                      frameBytes;
    private BufferedImage               bufferedImage;
    private RenderingThread             renderingThread;
    private boolean                     isRenderable = false;
    private WritableRaster              writableRaster;

    private int                         width;
    private int                         height;

    class RenderingThread extends Thread
    {
        private int             frameRate = 5;
        private int             sleepTime;


        public RenderingThread()
        {
            this.sleepTime = (int)(1000/frameRate);
        }

        public RenderingThread( int frameRate )
        {
            this.frameRate = frameRate;
            this.sleepTime = (int)(1000/frameRate);
        }

        public void run()
        {
            try
            {
                sleep( sleepTime );
                repaint();
            }
            catch( Exception e )
            {
                logger.error("Failed to repaint screen due to exception ", e );
            }
        }

        public int getFrameRate()
        {
            return frameRate;
        }

        public void setFrameRate(int frameRate)
        {
            this.frameRate = frameRate;
        }
    }

    public TestPBuffer( int width, int height )
    {
        this.width = width;
        this.height = height;
        try
        {
            // create an OpenGL LWJGL PBuffer
            //
            pbuffer = new Pbuffer( width, height, new PixelFormat(), null, null );

            // make the PBuffer current
            //
            pbuffer.makeCurrent();

            // create a buffered image that will hold the frame buffer contents
            //
            ColorModel colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB), true, false, ColorModel.OPAQUE, DataBuffer.TYPE_BYTE);
            writableRaster = colorModel.createCompatibleWritableRaster( width, height );
            bufferedImage =  new BufferedImage(colorModel, writableRaster, false, null);

            renderingThread = new RenderingThread();

            isRenderable = true;

        }
        catch( Exception e )
        {
            logger.error("Adpater error ", e );
        }
    }

    public void start()
    {
        if ( isRenderable )
        {
            // everything should be ready to render before this call is made. The component should have been
            // added and packed so that it has a valid width() and height()
            //
            renderingThread.start();
        }
    }

    public int getFrameRate()
    {
        return renderingThread.getFrameRate();
    }

    public void setFrameRate(int frameRate)
    {
        renderingThread.setFrameRate( frameRate );
    }

    protected void paintComponent( Graphics g )
    {
        g.setColor( Color.BLACK );
        g.fillRect(0, 0, width, height);
        
        
        try
        {
            pbuffer.makeCurrent();
        }
        catch( Exception e ){}

        if ( glFrameBuffer == null )
        {
            // create a frame buffer to hold the frame from LWJGL
            //
            glFrameBuffer = BufferUtils.createByteBuffer( width * height * 4 );
        }

        renderScene();

        GL11.glReadPixels(0, 0, width, height, GL11.GL_RGBA, GL11.GL_UNSIGNED_BYTE, glFrameBuffer );

        glFrameBuffer.get( ((DataBufferByte)writableRaster.getDataBuffer()).getData() );
        glFrameBuffer.rewind();


        // draw the image
        g.drawImage( bufferedImage, 0, 0, width, height, getBackground(), null );
    }

    public void renderScene()
    {
        GL11.glClear(GL11.GL_COLOR_BUFFER_BIT | GL11.GL_DEPTH_BUFFER_BIT);          // Clear The Screen And The Depth Buffer
    }

    /**
     * Invoked when the component's size changes.
     */
    public void componentResized(ComponentEvent event)
    {
        // recreate the pbuffer for the new component size
        try
        {
            //pbuffer = new Pbuffer( getWidth(), getHeight(), new PixelFormat(), null, null );
            //pbuffer.makeCurrent();
            //glFrameBuffer = BufferUtils.createByteBuffer( getWidth() * getHeight() * 3);
        }
        catch( Exception e )
        {
            logger.error("Unable to recreate the OpenGL components after the component resized", e );
        }
    }

    /**
     * Invoked when the component's position changes.
     */
    public void componentMoved(ComponentEvent e)
    {

    }

    /**
     * Invoked when the component has been made visible.
     */
    public void componentShown(ComponentEvent event)
    {
        try
        {
            // just in case we've broken some textures or in case some other GL thing became current
            // while we were hidden
            pbuffer.makeCurrent();
        }
        catch( Exception e )
        {
            logger.error("Unable to make the pbuffer current after component shown", e);
        }
    }

    /**
     * Invoked when the component has been made invisible.
     */
    public void componentHidden(ComponentEvent e)
    {

    }

    public static void main( String[] args )
    {
        BasicConfigurator.configure();

        try
        {
            JFrame frame = new JFrame();
            frame.setSize( 640, 480 );
            TestPBuffer client = new TestPBuffer(640,480);

            frame.getContentPane().add( client );
            //frame.pack();
            frame.show();

            frame.setDefaultCloseOperation( JFrame.DISPOSE_ON_CLOSE );

            client.start();
        }
        catch( Exception e )
        {
            logger.error("Error building the client ", e );
        }
    }
}

I expect the next code update will be the final and we'll have something that will allow Swing and actually browser compatibility.