2D LoS rendering and stencil buffer - help needed!

[Muhwu]

Hello,

I just recently decided to move my 2D game to use LWJGL due to Java2D simply being sluggish and a bit ugly in quite mundane tasks. I already have a Line-of-sight algorithm for determining a visible polygon. What I want is:

The characters line of sight rendered as a clear clipped field of vision.
The rest of the map rendered in a darker shade or perhaps grayscale.

I have been trying to find a reasonable way to do this. I'm currently experiencing with Stencil buffers, but the results are indefinite. The rendering code for the field of vision works splendidly, but this solution forces me to render the map twice.

map.render();
        
GL11.glDisable(GL11.GL_TEXTURE_2D);
        
// Render gray overlay on the map
GL11.glBegin(GL11.GL_QUADS);
GL11.glColor4d(0.25, 0.25, 0.25, 0.25);
GL11.glVertex2d(m_Viewpoint.getX(), m_Viewpoint.getY());
GL11.glVertex2d(m_Viewpoint.getX()+Display.getWidth(), m_Viewpoint.getY());
GL11.glVertex2d(m_Viewpoint.getX()+Display.getWidth(), m_Viewpoint.getY()+Display.getHeight());
GL11.glVertex2d(m_Viewpoint.getX(), m_Viewpoint.getY()+Display.getHeight());
GL11.glEnd();
        
// Clear stencil buffer & enable
GL11.glClearStencil(0);
glClear(GL11.GL_STENCIL_BUFFER_BIT);
GL11.glEnable (GL11.GL_STENCIL_TEST);

GL11.glStencilFunc (GL11.GL_ALWAYS, 0x1, 0x1);
GL11.glStencilOp(GL11.GL_KEEP, GL11.GL_KEEP, GL11.GL_REPLACE);
GL11.glColorMask(false, false, false, false);  

// Render FoV to stencil buffer
GL11.glColor4d(1, 1, 1, 1);

GL11.glBegin(GL11.GL_TRIANGLE_FAN);
GL11.glVertex2d(mPlayer.getX(), mPlayer.getY());
for(KPoint p : mPlayer.getVisionPolygon().points) {
  GL11.glVertex2d(p.x, p.y);
}
GL11.glVertex2d(mPlayer.getVisionPolygon().points.get(0).x, mPlayer.getVisionPolygon().points.get(0).y);
GL11.glEnd();
GL11.glColor4d(1.0, 1.0, 1.0, 1.0);
GL11.glEnable(GL11.GL_TEXTURE_2D);
		
GL11.glStencilFunc (GL11.GL_EQUAL, 0x1, 
GL11.glColorMask(true, true, true, true);
        
map.render()

objects.render()

Basically it does exactly what it's supposed and the performance is still alright, but that is partly due to the simple nature of the test scenario as well. Is there a simple, easier way to achieve the same results without having to render the map twice? I can intuitively think of some sort of alpha mask rendering (same idea as in this, but instead of using binary stencil buffer, i would have a proper alpha mask to draw under - i.e. fov as 0,0,0,0 and everything else as 25,25,25,25 [using the same colors as in above code]). The stencil buffer could then be used to clip the sprites ? In this case, however, I would be rendering my FoV to two places (and I don't even know how exactly I should be approaching such alpha mask problem). Is there an easy way to do it?

I'm a newbie here and haven't worked with OpenGL in years, so all feedback/help is welcome.

[CodeBunny]

How about using render to texture and shaders?

Have two textures that you render the scene to:

One texture contains the scene as you normally would render it.

Once you have rendered that, switch to the other texture, and render a mask for line of sight. (An easy way to do this would be to clear the texture to black, disable texturing, and draw in opaque white the shape representing line-of-sight. Assuming you can easily create the geometry (I'd imagine a GL_TRIANGLE_FAN would be appropriate), it's a snap to render).

Then, use multi-texturing to bind both textures at once, then use a shader to modify the resulting image based upon the two sets of data.

Voila! Depending on your shader, you can have any affect you want applied to the out-of-sight areas.

What's interesting about this method is that you can put line-of-sight data into one color channel (say, the alpha channel) and then put other useful image data into the color channels. (I use a similar method for very simple but fast lighting in my 2D games.)

[Muhwu]

That sounds reasonable. I already used TRIANGLE_FAN in the code above to render to stencil buffer. As it appears, for some reason the above code is only working on my desktop computer. For some reason on GeForce GTX 460M it does not work. I have no clue as to why, I'd imagine the feature is available for both that and my ATI Radeon HD 4800. Don't really know what else could stop it from working really. :/

Either way, it looks like I'll have to go with the multi-texturing way, if I clear the texture out with an alpha color, I'll already have a darker shade with no need to actually use shaders, right? Or just render these two surfaces on top of each other, one transparent. This would work too I guess.

I'm still slightly bedazzled by why it does not work on the other hardware setup tho.

[Muhwu]

Actually now that I'm doing this in practice, I notice another short-coming of this approach. In the previous versions, the stencil buffer was also used to clip the sprites drawn into the scene. I only want to be able to shade the map darker, but the sprites existing in it should not be rendered, except for the parts that are visible in the vision polygon. This worked perfectly in my setting but since Stencil buffer for some reason does not work on my laptop, I will have to look for another option.

Is it possible to "clear" a polygon from a surface? It seems to be unreasonably difficult to find information on the subject. I created a frame buffer object and rendered my FOV there after clearing it with a darker color, but needless to say, it just renders as a lighter color instead of actually rendering an area transparent. I suppose this is where the shaders with the mask would come to play.

Either way, it does not solve the problem with clipping sprites which was quite elegantly done in the other version. I actually think that rendering the scenario twice is fast enough anyway since it's a 2D game and I'm not running into performance issues as of yet. Don't optimize what doesn't need optimizing, right? Anyone have any clue as to why it wouldn't work ? There's obviously no error data as it only doesn't apply the effect. Is there anyway to test for the Stencil buffer functionality and whether it exists at all or whether there's some gimmick to make it work?

[Muhwu]

Alrighty, after a LOT of work, I managed to find out that the Display.create function indeed kinda requires the pixel format as a parameter as the defaults appear to be system specific. The stencil buffer was initialized with value 0 in my other system.

Either way, I'm going with the stencil buffer solution now as it appears to satisfy my needs and solve most problems which would occur with other solutions.