Shadow mapping using LWJGL and shaders not working

[Lotho]

I'm trying to use this tutorial to add shadow mapping to my 3D environment. As i need to translate it to java I'm obviously doing something wrong in the translation. Here is the environment I've set up:



Here's after applying the shadow map (for some reason everything is in shadow):




What I believe is the problem is how the shadowmap texture is created and/or how it's sent in to the fragment shader.

It's quite a lot of code, I'll try to post the relevant parts here:


Initializing:

public static void setUp() {
		setUpStates();
		setUpBasicShaders();
		setUpShadowShaders();
		VBOHandler.setUpGroundVBOs();
		VBOHandler.setUpHouseVBOs();
		VBOHandler.setUpShadowBuffer();
		setUpCamera();
	}

rendering:

public static void render() {
		VBOHandler.updateShadowBuffer(shadowShaderProgram, depthMatrix, depthMatrixID);
		
		Window.bindAsRenderTarget();
		clearScreen();
		glLoadIdentity();
		
		VBOHandler.updateShadowBias(shaderProgram, depthMatrix, depthBiasMatrixID, shadowMapID);
		
		camera.updateCamera();
		
		VBOHandler.updateGround(shaderProgram, modelMatrix, modelMatrixID);
		VBOHandler.updateHouse(shaderProgram, modelMatrix, modelMatrixID);
		
		glUseProgram(shaderProgram);
		
		viewMatrix.store(matrix44Buffer);
		matrix44Buffer.flip();
		glUniformMatrix4(viewMatrixID, false, matrix44Buffer);
		
		glUseProgram(0);

		checkInput();
	}

setting up shaders:

private static void setUpBasicShaders() {
		ArrayList<Integer> shaderList = new ArrayList<>();
		shaderList.add(Framework.loadShader(GL_VERTEX_SHADER, "basicShader.vs"));
		shaderList.add(Framework.loadShader(GL_FRAGMENT_SHADER, "basicShader.fs"));

		shaderProgram = Framework.createProgram(shaderList);

		projectionMatrixID = glGetUniformLocation(shaderProgram, "projectionMatrix");
		viewMatrixID = glGetUniformLocation(shaderProgram, "viewMatrix");
		modelMatrixID = glGetUniformLocation(shaderProgram, "modelMatrix");
		depthBiasMatrixID = glGetUniformLocation(shaderProgram, "depthBiasMatrix");
		shadowMapID = glGetUniformLocation(shaderProgram, "shadowMap");
		
		projectionMatrix = new MatrixHandler();
		viewMatrix = new MatrixHandler();
		modelMatrix = new MatrixHandler();

		matrix44Buffer = BufferTools.reserveFloatData(16);
	}
	private static void setUpShadowShaders() {
		ArrayList<Integer> shaderList = new ArrayList<>();
		shaderList.add(Framework.loadShader(GL_VERTEX_SHADER, "shadowShader.vs"));
		shaderList.add(Framework.loadShader(GL_FRAGMENT_SHADER, "shadowShader.fs"));

		shadowShaderProgram = Framework.createProgram(shaderList);
		
		depthMatrix = new MatrixHandler();
		depthMatrixID = glGetUniformLocation(shadowShaderProgram, "depthMatrix");
	}

shadowShader.vs:

#version 330 core

layout(location = 0) in vec3 position;

uniform mat4 depthMatrix;

void main(){
	gl_Position =  depthMatrix * vec4(position,1);
}

shadowShader.fs:

#version 330 core

layout(location = 0) out float fragmentdepth;

void main(){
	fragmentdepth = gl_FragCoord.z;
}

basicShader.vs:

#version 330

layout(location = 0) in vec3 position;
layout(location = 1) in vec3 color;

out vec3 theColor;
out vec4 shadowCoord;

uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;
uniform mat4 modelMatrix;
uniform mat4 depthBiasMatrix;

void main()
{
	gl_Position = projectionMatrix * viewMatrix * modelMatrix * vec4(position, 1);
	shadowCoord = depthBiasMatrix * vec4(position, 1);
	theColor = color;
}

basicShader.fs:

#version 330

in vec3 theColor;
in vec4 ShadowCoord;

out vec4 outputColor;

uniform sampler2DShadow shadowMap;

void main()
{
	float visibility = texture( shadowMap, vec3(ShadowCoord.xy, (ShadowCoord.z)/ShadowCoord.w) );
	visibility += 0.5f;
	vec3 tempColor = visibility * theColor;
	outputColor = vec4(tempColor, 1);
}

Setting up shadowBuffer

public static void setUpShadowBuffer() {
		//create texture
		shadowBuffer = glGenFramebuffers();
		glBindFramebuffer(GL_FRAMEBUFFER, shadowBuffer);
		
		depthTexture = glGenTextures();
		glBindTexture(GL_TEXTURE_2D, depthTexture);
		
		glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT16, 1024, 1024, 0, GL_DEPTH_COMPONENT, GL_FLOAT, (ByteBuffer)null);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
		
		glFramebufferTexture(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, depthTexture, 0);
		glDrawBuffer(GL_NONE);
		if(glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
			System.err.println("Error setting up shadowbuffer");
		}
		//Draw objects
		
		int vboVertexHandle;
		int vboIndexHandle;
		FloatBuffer vertexData;
		IntBuffer indexData;
		
		House house = new House();
		Ground ground = new Ground();
		
		float[] vertexDataValues;
		int[] indexDataValues;

		float[] houseVertexDataValues;
		int[] houseIndexDataValues;
		
		float[] groundVertexDataValues;
		int[] groundIndexDataValues;
		
		houseVertexDataValues = house.getVertices();
		houseIndexDataValues = house.getIndices();
		
		groundVertexDataValues = ground.getVertices();
		groundIndexDataValues = ground.getIndices();
		for (int i = 0; i < groundIndexDataValues.length; i++) {
			groundIndexDataValues[i] += houseIndexDataValues[houseIndexDataValues.length - 1] + 1;
		}
		
		vertexDataValues = new float[houseVertexDataValues.length + groundVertexDataValues.length];
		indexDataValues = new int[houseIndexDataValues.length + groundIndexDataValues.length];
		
		System.arraycopy(houseVertexDataValues, 0, vertexDataValues, 0, houseVertexDataValues.length);
		System.arraycopy(groundVertexDataValues, 0, vertexDataValues, houseVertexDataValues.length, groundVertexDataValues.length);
		
		System.arraycopy(houseIndexDataValues, 0, indexDataValues, 0, houseIndexDataValues.length);
		System.arraycopy(groundIndexDataValues, 0, indexDataValues, houseIndexDataValues.length, groundIndexDataValues.length);
		
		shadowIndices = indexDataValues.length;
		
		vertexData = BufferTools.reserveFloatData(vertexDataValues.length);
		vertexData.put(vertexDataValues);
		vertexData.flip();

		vboVertexHandle = glGenBuffers();
		glBindBuffer(GL_ARRAY_BUFFER, vboVertexHandle);
		glBufferData(GL_ARRAY_BUFFER, vertexData, GL_STATIC_DRAW);
		glBindBuffer(GL_ARRAY_BUFFER, 0);
		
		indexData = BufferTools.reserveIntData(indexDataValues.length);
		indexData.put(indexDataValues);
		indexData.flip();
		
		vboIndexHandle = glGenBuffers();
		glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, vboIndexHandle);
		glBufferData(GL_ELEMENT_ARRAY_BUFFER, indexData, GL_STATIC_DRAW);
		glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);
		
		shadowArrayObject = glGenVertexArrays();
		bindArray(shadowArrayObject);

        glBindBuffer(GL_ARRAY_BUFFER, vboVertexHandle);
        glEnableVertexAttribArray(0);
        glVertexAttribPointer(0, 3, GL_FLOAT, false, 0, 0);
        glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, vboIndexHandle);

        unBindArray();
	}

Updating shadowbuffer:

public static void updateShadowBuffer(int shadowShaderProgram, MatrixHandler depthMatrix, int depthMatrixID) {
		matrix44Buffer = BufferTools.reserveFloatData(16);
		
		glBindFramebuffer(GL_FRAMEBUFFER, shadowBuffer);
        glViewport(0,0,1024,1024);
        
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
        
        MatrixHandler depthProjectionMatrix = new MatrixHandler();
        MatrixHandler depthViewMatrix = new MatrixHandler();
        MatrixHandler depthModelMatrix = new MatrixHandler();
		
        depthProjectionMatrix.initOrthographicMatrix(-10, 10, -10, 10, -10, 40);
		depthViewMatrix.lookAt(new Vector3f(-5f, 15, 20), new Vector3f(0, 0, 0), new Vector3f(0, 1, 0));
		depthModelMatrix.setIdentity();
		
		Matrix4f.mul(depthViewMatrix, depthModelMatrix, depthMatrix);
		Matrix4f.mul(depthProjectionMatrix, depthMatrix, depthMatrix);
		
		glUseProgram(shadowShaderProgram);
		depthMatrix.store(matrix44Buffer);
		matrix44Buffer.flip();
		glUniformMatrix4(depthMatrixID, false, matrix44Buffer);
		
		bindArray(shadowArrayObject);
		glDrawElements(GL_TRIANGLES, getShadowIndices(), GL_UNSIGNED_INT, 0);
		unBindArray();
		
		glUseProgram(0);
	}

updating shadow bias:

public static void updateShadowBias(int shaderProgram, MatrixHandler depthMatrix, int depthBiasMatrixID, int shadowMapID) {
		matrix44Buffer = BufferTools.reserveFloatData(16);
		
		MatrixHandler biasMatrix = new MatrixHandler();
		MatrixHandler depthBiasMatrix = new MatrixHandler();
		biasMatrix.setBias();
		Matrix4f.mul(biasMatrix, depthMatrix, depthBiasMatrix);
		
		glUseProgram(shaderProgram);
		
		depthBiasMatrix.store(matrix44Buffer);
		matrix44Buffer.flip();
		glUniformMatrix4(depthBiasMatrixID, false, matrix44Buffer);
		
		glActiveTexture(GL_TEXTURE0);
		glBindTexture(GL_TEXTURE_2D, depthTexture);
        glUniform1i(shadowMapID, 0);
		
		glUseProgram(0);
	}

[quew8]

Sorry but I for one am not going to look through all that code. A) It would take me a lot of time and B) I'm just as likely to miss a simple typo (which is generally all the problem is) as you are.

What I will do is give you some debugging tips. Firstly, try rendering the shadow map texture directly to the screen (you can write a simple grey scale type fragment shader) and see what it looks like. Then try using a pre-baked shadow map to see what results you get. You can try no shadows, simple gradients, striped patterns, all sorts of things to figure out what is going on.

Basically the problem with debugging OpenGL is that the intermediate results are all inaccessible on the GPU. But this isn't the case when you are working with textures - you can jump in the middle of the process and review what is coming out and change what is going in. Make  use of it.

[Lotho]

Sorry but I for one am not going to look through all that code. A) It would take me a lot of time and B) I'm just as likely to miss a simple typo (which is generally all the problem is) as you are.

Yea, I thought that for myself too, "noone will read all this" but didn't come up with a better sulotion since I have no idea where the problem lies.

What I will do is give you some debugging tips. Firstly, try rendering the shadow map texture directly to the screen (you can write a simple grey scale type fragment shader) and see what it looks like. Then try using a pre-baked shadow map to see what results you get. You can try no shadows, simple gradients, striped patterns, all sorts of things to figure out what is going on.

Basically the problem with debugging OpenGL is that the intermediate results are all inaccessible on the GPU. But this isn't the case when you are working with textures - you can jump in the middle of the process and review what is coming out and change what is going in. Make  use of it.

Thank you for these tips! I've been debugging for a long time now and found many faults but just came to a halt, which is why I posted here. Now I know how to keep going. Will get back to it when I've been looking into it further.

[Lotho]

Solved this by adding these 2 lines when setting up the shadowbuffer:

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_COMPARE_FUNC, GL_LEQUAL);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE);

I also had a typo in the fragmentshader where the in-parameter shadowCoord had a capital S instead of a lower case s.

However a couple of new questions came up:

• What is the best way to deal with shadow acne? (I tried culling front faces for the shadow map but this still caused acne on the other side of the object)


• If I I render several objects and want to move them around separately do I need to create separate VBOs for each object? (I'm only going to have about 10 objects in the scene)


• When creating the shadow map I need all vertices in the same buffer, do I make a separate buffer containing every objects' vertices for that, or do I create a new shadow map for each object? (I will only be dealing with directional light)