Shadowmapping with FBO results in black texture [screenshot]

[Bvsemmer]

Hi,

I added shadowmap rendering to a hobby-engine of mine but it doesn't make use of an offscreen buffer (I just render the scene to the screen, copy the depth-values to a texture, clear the screen and rerender the scene again).
Since I just heard about EXT_Framebuffer_Object i'd like to take my shadow-renderer a step further and use an offscreen buffer to fill my depth-texture.

I did some research about Framebuffer_Objects and made a working realtime Environmental Mapping demo, making use of an FBO.  
Screen:


Though, ofcourse, this doesn't use a depth-texture yet and the next step, to use an FBO to do shadowmapping, goes not that smooth.
Setting everything up seems to work but the shadowtexture is always completely black.

Here is a screenshot from the working shadowmap renderer (the white lines are the light-frustum):


Here is a screenshot of the shadowmap renderer which uses an FBO:


Would someone be so kind to look what I'm doing wrong please?  I'm a bit in the dark.

Here is how I create my render-target depthtexture:

public int initRenderTexture()
{
	IntBuffer shadowMapTexture = BufferUtils.createIntBuffer(1);

	ByteBuffer tex = ByteBuffer.allocateDirect(shadowMapSize * shadowMapSize);
	GL11.glGenTextures(shadowMapTexture); 
	int texID = shadowMapTexture.get(0);

	System.out.println(texID);
	GL11.glBindTexture(GL11.GL_TEXTURE_2D, texID); // Bind texture
	GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL11.GL_DEPTH_COMPONENT, shadowMapSize, shadowMapSize, 0, GL11.GL_DEPTH_COMPONENT, GL11.GL_UNSIGNED_BYTE, tex);
	
	GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MIN_FILTER, GL11.GL_NEAREST);
	GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MAG_FILTER, GL11.GL_NEAREST);
	
	GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_S, GL11.GL_CLAMP);
	GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_WRAP_T, GL11.GL_CLAMP);

	return texID;
}

Here I create the FBO:

id = BufferUtils.createIntBuffer(1);
EXTFramebufferObject.glGenFramebuffersEXT(id);
fbID = id.get(0);
		
id = BufferUtils.createIntBuffer(1);
EXTFramebufferObject.glGenRenderbuffersEXT(id);
rbID = id.get(0);

Here I setup the FBO (depthID is the ID of the depth-texture):

EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, fbID);
GL11.glReadBuffer(GL11.GL_NONE);
GL11.glDrawBuffer(GL11.GL_NONE);
EXTFramebufferObject.glFramebufferTexture2DEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT,
											   EXTFramebufferObject.GL_DEPTH_ATTACHMENT_EXT,
											   GL11.GL_TEXTURE_2D,
											   depthID,
											   0);
EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, 0);

EXTFramebufferObject.glBindRenderbufferEXT(EXTFramebufferObject.GL_RENDERBUFFER_EXT, rbID);
EXTFramebufferObject.glRenderbufferStorageEXT(EXTFramebufferObject.GL_RENDERBUFFER_EXT,
		   GL11.GL_DEPTH_COMPONENT,
		   512,
		   512);
EXTFramebufferObject.glBindRenderbufferEXT(EXTFramebufferObject.GL_RENDERBUFFER_EXT, 0);

EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, fbID);
EXTFramebufferObject.glFramebufferRenderbufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, EXTFramebufferObject.GL_DEPTH_ATTACHMENT_EXT,
		EXTFramebufferObject.GL_RENDERBUFFER_EXT, rbID);
EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, 0);

I use this to render my scene to the FBO:

EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, fbID);
engine.render();
EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, 0);

I'm using an ATI card, by the way.  (I read ATI only supports GL_DEPTH_COMPONENT16 but changing the depth-texture to that format doesn't solve the problem).

Thanks a lot in advance!

[spasi]

Hi Bvsemmer,

You're using both a depth texture and an FBO depth render buffer. You're binding the render buffer last, so all rendering goes there and nothing goes to the texture. Remove the render buffer (it's not needed) and it should work ok.

[Bvsemmer]

Hi Spasi,

I removed the renderbuffer but the problem unfortunatly still remains.

This is the situation now:

FBO creation:

id = BufferUtils.createIntBuffer(1);
		EXTFramebufferObject.glGenFramebuffersEXT(id);
		fbID = id.get(0);

FBO setup:

EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, fbID);
		GL11.glReadBuffer(GL11.GL_NONE);
		GL11.glDrawBuffer(GL11.GL_NONE);
		EXTFramebufferObject.glFramebufferTexture2DEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT,
													   EXTFramebufferObject.GL_DEPTH_ATTACHMENT_EXT,
													   GL11.GL_TEXTURE_2D,
													   depthID,
													   0);
		EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, 0);

I can use the depthTexture (the one linked to the FBO) just like it's a regular texture without further ado, right?  Cause I'm still using my old code to actually render the shadowmap and project it over the scene.

[spasi]

Yes, you can use the depth texture as you did before, no problem.

Try some of the following:

1. Set the internal texture format to GL_DEPTH_COMPONENT16 explicitly.
2. Use glTexImage2D with a (ByteBuffer)null as the last parameter, instead of a random buffer.
3. Check the FBO for completeness. Are you sure no errors occur?
4. Disable color rendering with glColorMask.
5. Are you calling glViewport before rendering to the FBO?

[Bvsemmer]

Thanks again for the reply, Spasi.

1. Set the internal texture format to GL_DEPTH_COMPONENT16 explicitly.

I did, but with no effect:

GL11.glTexImage2D(GL11.GL_TEXTURE_2D, 0, GL14.GL_DEPTH_COMPONENT16, shadowMapSize, shadowMapSize, 0, GL11.GL_DEPTH_COMPONENT, GL11.GL_UNSIGNED_BYTE, (ByteBuffer)null);

2. Use glTexImage2D with a (ByteBuffer)null as the last parameter, instead of a random buffer.

See above

3. Check the FBO for completeness. Are you sure no errors occur?

After FBO Setup glCheckFramebufferStatusEXT() returns GL_FRAMEBUFFER_COMPLETE_EXT.  That seems a-ok.

4. Disable color rendering with glColorMask.

That's something I already did, but didn't mention here:

GL11.glColorMask(false, false, false, false);
EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, fbID);

RenderSceneToShadowMap();	
	
EXTFramebufferObject.glBindFramebufferEXT(EXTFramebufferObject.GL_FRAMEBUFFER_EXT, 0);
GL11.glColorMask(true, true, true, true);

5. Are you calling glViewport before rendering to the FBO?

I do, and I set the viewPort to the shadowMapSize, like this:

GL11.glViewport(0, 0, shadowMapSize, shadowMapSize);

I used the color to make a clearer distinction between the [code]- and

-parts.

edit: I forgot the mention that I messed around with my working Shadowmap code and when I remove "glCopyTexSubImage2D()" (which copies the contents of the screen to the depthTexture) I get the same result there.  
Apparently rendering to the texture linked to the FBO doesn't work, though it seems I linked it ok.

[spasi]

Some more:

1. Try GL_CLAMP_TO_EDGE instead of GL_CLAMP when setting up the texparams.
2. Try checking for FBO errors *after* rendering the scene too. I don't remember why, but I do this in my code. It may catch something.

[Bvsemmer]

Some more:

1. Try GL_CLAMP_TO_EDGE instead of GL_CLAMP when setting up the texparams.
2. Try checking for FBO errors *after* rendering the scene too. I don't remember why, but I do this in my code. It may catch something.

Nothing.

May I ask why I don't need a renderTarget here?  It seems renderTargets keep coming back when I read about using FBO's and a depth-texture (like here and here)  
Perhaps I don't fully understand what renderBuffers are for though.

edit: I think I found a solution to my last question.  According to this arcticle, RenderTargets can be rendered to, but can't be used as texture images (and thus not usable for ShadowMapping), is that correct?

Anyway, that still doesn't explain why I can't get my depthTexture working  Very wierd, indeed.
I'm using these two methods to compare the depth-values in the scene to the depth-values in the depthTexture (after I bind the depthTexture).

GL11.glTexParameteri(GL11.GL_TEXTURE_2D, ARBShadow.GL_TEXTURE_COMPARE_MODE_ARB, ARBShadow.GL_COMPARE_R_TO_TEXTURE_ARB);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, ARBShadow.GL_TEXTURE_COMPARE_FUNC_ARB, GL11.GL_LEQUAL);

Those are still valid here?

[spasi]

edit: I think I found a solution to my last question.  According to this arcticle, RenderTargets can be rendered to, but can't be used as texture images (and thus not usable for ShadowMapping), is that correct?

Yes, exactly. Put another way, you only have one depth attachment available and you use the depth texture for that. So, you get your depth values there and the depth texture also works as a normal z-buffer, so you're not losing anything.

Anyway, that still doesn't explain why I can't get my depthTexture working  Very wierd, indeed.
I'm using these two methods to compare the depth-values in the scene to the depth-values in the depthTexture (after I bind the depthTexture).

GL11.glTexParameteri(GL11.GL_TEXTURE_2D, ARBShadow.GL_TEXTURE_COMPARE_MODE_ARB, ARBShadow.GL_COMPARE_R_TO_TEXTURE_ARB);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, ARBShadow.GL_TEXTURE_COMPARE_FUNC_ARB, GL11.GL_LEQUAL);

Those are still valid here?

Yes, everything should work as before. I find it weird too, so I'm guessing some other state must be causing the trouble. Sorry for not helping further, I'll let you know if I think of something.

[spasi]

Some things you can try:

- Render to a normal color FBO to at least make sure everything is rendered fine.

- Set the depth texture's texparams GL_TEXTURE_COMPARE_MODE => GL_NONE and GL_DEPTH_TEXTURE_MODE => GL_LUMINANCE and render a fullscreen quad over your scene to see if anything gets rendered to it (and then reset the texparams).

[Bvsemmer]

Rendering to a GL_COLOR_ATTACHMENT0_EXT framebuffer works without a problem.  Here I render the scene to my framebuffer and map it on a quad.
No problems here:


When I render to a GL_DEPTH_ATTACHMENT_EXT framebuffer the depthTexture is, again, completely black:


Just out of curiosity, did you already manage to get Shadowmapping working with a FBO?

Thanks again for your time.

[spasi]

Just out of curiosity, did you already manage to get Shadowmapping working with a FBO?

Yup, you can see some oldish screenshots here.

Everything worked fine immediately when I switched to FBO, even on ATI cards (well, after setting depth to 16bits :wink:).

[Bvsemmer]

Yup, you can see some oldish screenshots here.

That seems to work fine, indeed.  Some nice screenshots there btw!

Everything worked fine immediately when I switched to FBO, even on ATI cards (well, after setting depth to 16bits :wink:).

I thought about driver issues, but after installation of two other Catalyst drivers (one recent, one older) the problem still remains.

Btw, as a side note: is it normal that in the most recent ATI driver "glCopyTexSubImage2D()" performs extremely bad?  My ShadowApp without the FBO went from +/-400fps to a humiliating 5fps, with that driver installed!

Anyway, I'm going to try my code on a friend's pc tonight who has an nVidia card and try to get it to work there.  If that doesn't work, well ... keep looking I guess

[Bvsemmer]

Anyway, I'm going to try my code on a friend's pc tonight who has an nVidia card and try to get it to work there.

No luck, the same problem there.
I guess it's a 99% chance now it's a mistake in my code, but I'm starting to run out of ideas now :?

[spasi]

Some nice screenshots there btw!

Thanks!

Btw, as a side note: is it normal that in the most recent ATI driver "glCopyTexSubImage2D()" performs extremely bad?

It isn't normal, but it's not surprising either. :wink: I remember having the same problem before switching to FBO, so I was using glCopyTexImage2D instead (with good enough performance). That was on an NV and it was also working fine on (at that time) older drivers. :?

I guess it's a 99% chance now it's a mistake in my code, but I'm starting to run out of ideas now :?

Yeah, me too. You could try posting more extensive code bits, I might be able to spot something.

[Bvsemmer]

Oh my god, I just found the problem.  I didn't realise you needed to clear the FBO as well each frame.  Argh, man i've been looking at this for days!  Anyway, it works flawlessly now.

Thanks a bunch for your help though Spasi, I'm much obliged