Ray Picking

[Daslee]

I'm not sure what you mean by

add some float values to ray direction

. But this is most likely a problem with the way you create the ray. Could I see that code?

I add 0.75f to ray direction x and y values and then ray is almost correct, but only then when camera is not moving. And here is my code to get picking ray:

public Vector3 getPickingRay(float cursorX, float cursorY){	
		IntBuffer viewport = ByteBuffer.allocateDirect((Integer.SIZE/8)*16).order(ByteOrder.nativeOrder()).asIntBuffer();
		FloatBuffer modelview = ByteBuffer.allocateDirect((Float.SIZE/8)*16).order(ByteOrder.nativeOrder()).asFloatBuffer();
		FloatBuffer projection = ByteBuffer.allocateDirect((Float.SIZE/8)*16).order(ByteOrder.nativeOrder()).asFloatBuffer();
		FloatBuffer pickingRayBuffer = ByteBuffer.allocateDirect((Float.SIZE/8)*3).order(ByteOrder.nativeOrder()).asFloatBuffer();
		FloatBuffer zBuffer = ByteBuffer.allocateDirect((Float.SIZE/8)*1).order(ByteOrder.nativeOrder()).asFloatBuffer();
		glGetFloat(GL_MODELVIEW_MATRIX, modelview);
		glGetFloat(GL_PROJECTION_MATRIX, projection);
		glGetInteger(GL_VIEWPORT, viewport);
		float winX = (float) cursorX;
		// convert window coordinates to opengl coordinates (top left to bottom left for (0,0)
		float winY = (float) viewport.get(3) - (float) cursorY;
		
		// now unproject this to get the  vector in to the screen
		// take the frustrm and unproject in to the screen
		// frustrum has a near plane and a far plane
		
		// first the near vector
		gluUnProject(winX, winY,  0, modelview, projection, viewport, pickingRayBuffer);		
		Vector3 nearVector = new Vector3(pickingRayBuffer.get(0),pickingRayBuffer.get(1),pickingRayBuffer.get(2));
		
		pickingRayBuffer.rewind();
		
		// now the far vector
		gluUnProject(winX, winY,  1, modelview, projection, viewport, pickingRayBuffer);
		Vector3 farVector = new Vector3(pickingRayBuffer.get(0),pickingRayBuffer.get(1),pickingRayBuffer.get(2));
		
		//save the results in a vector, far-near
		return farVector.sub(nearVector).normalize();
	}

abcdef gave me this code. And here how I use it:

for(Object obj : world.visibleCollidingObjects) obj.selected = false;
		Vector3 destination = getPickingRay(Display.getWidth() / 2, Display.getHeight() / 2);
		Ray rayPick = new Ray(new Vector3(control.position.x, control.position.y, control.position.z), destination);				
		
		int index = -1;
		float tmin = Float.MAX_VALUE;
		t=tmin;
		float minDepth = Float.MAX_VALUE;
		for(int i=0; i<world.visibleCollidingObjects.size(); i++){
			float depth = 0;
			if(rayIntersection(rayPick, world.visibleCollidingObjects.get(i))){
				depth = world.visibleCollidingObjects.get(i).z;
			}else{
				depth = Float.MAX_VALUE;
			}
			if(depth < minDepth){
				minDepth = depth;
				index = i;
			}
		}
		if(index != -1) world.visibleCollidingObjects.get(index).selected = true;
		selectedObjectIndex = index;

[quew8]

The only things I can think of right now are:
1) You are applying transforms to your drawn objects in rendering but not in picking.
2) You don't have the correct projection / modelview / viewport matrices setup when calling getPickingRay. Essentially are you calling it after calling glInitMatrix but before all the glFrustum, glTranslate and glRotate.

These seem unlikely and I'm probably missing something. I'll do something else for a few hours then take another look, maybe I'll see something. On a side note, in your second code extract, you call getPickingRay inside the loop, despite it being the same each time. I don't know for sure but I don't see it as being a very computationally "fast" function.

[Daslee]

Look what I found. On this line:

Ray rayPick = new Ray(new Vector3(control.position.x, control.position.y, control.position.z), destination);

As ray begin I'm using camera position and picking works good at yaw > 90 and yaw < 180. But if I set ray begin Y position to player Y position, not camera:

Ray rayPick = new Ray(new Vector3(control.position.x, this.y-1f, control.position.z), destination);

Then picking works good at yaw > 270 and yaw < 360. So maybe something's bad with ray begin position?

[quew8]

This is an FPS game. What is the difference between player position and camera position?

[Daslee]

In my game there is difference, because I would only use camera position, but I have player position for collision detection. Player has its own cube, but it's not rendering. http://img831.imageshack.us/img831/4193/53435298.png

But actually that couldn't be the problem, because in other game where I'm testing ray picking, I'm sure that camera position always stays in 0, 0, 0 and when creating picking ray I'm also using 0, 0, 0 as begin position. Now thinking about that, what you said:

1) You are applying transforms to your drawn objects in rendering but not in picking.

So I'm applying transforms in rendering, but not elsewhere. If that could be the problem, so where I should apply transforms elsewhere?

[quew8]

You need to apply the transforms to your cubes before you test them for picking. Are these transforms the viewing transforms (as in translating to camera position then rotating by camera's pitch and yaw)? If they are then it would probably be better to transform the camera position by the opposite of these. But that's up to you. Probably best to get it working before worrying too much about performance. 

[Daslee]

For first person camera control I'm using this code: http://www.lloydgoodall.com/tutorials/first-person-camera-control-with-lwjgl/ So you can look at it's source code how it works. And in my game I never using glRotatef except for camera. Cubes are only translated to their x,y,z position (when rendering) and then calling display list for cube.

[quew8]

Couldn't look at the code, your link is broken. Googling lloyd goodall didn't give anything programmesque either. So going only from your description:
To start with you're going to need to need a different picking ray for each cube you test.
rayStart = cameraPosition + cubePosition .
rayEnd = farVector + cubePosition.

farVector by the way is the variable from your getPickingRay method you posted earlier:

gluUnProject(winX, winY,  1, modelview, projection, viewport, pickingRayBuffer);
Vector3 farVector = new Vector3(pickingRayBuffer.get(0),pickingRayBuffer.get(1),pickingRayBuffer.get(2));

You must make sure however that when you call getPickingRay(), the only transformations you have given OpenGl are the viewing transformations and projection transformations.

[Daslee]

So in getPickingRay method I changed return to this:

gluUnProject(winX, winY,  1, modelview, projection, viewport, pickingRayBuffer)
return new Vector3(pickingRayBuffer.get(0),pickingRayBuffer.get(1),pickingRayBuffer.get(2));

And now checking for picking:

Vector3 farVec = getPickingRay(Display.getWidth() / 2, Display.getHeight() / 2);
			Ray rayPick = null;
			Vector3 cubePos = null;
			
			int index = -1;
			float minDepth = Float.MAX_VALUE;
			for(int i = 0; i < world.objects.size(); i++) {
				cubePos = new Vector3(world.objects.get(i).x, world.objects.get(i).y, world.objects.get(i).z);
				rayPick = new Ray(new Vector3(control.position.x, control.position.y, control.position.z).add(cubePos), farVec.add(cubePos));
				float depth = 0;
				if(rayIntersection(rayPick, world.objects.get(i))){
					depth = world.objects.get(i).z;
				}else{
					depth = Float.MAX_VALUE;
				}
				if(depth < minDepth){
					minDepth = depth;
					index = i;
				}
			}
			if(index != -1) world.objects.get(index).selected = true;
			selectedObjectIndex = index;

But rayIntersection method wants ray direction vector to be as normalized or..? Because now rayPick direction values sometimes can be ~1000, and it doesn't works.

[quew8]

It shouldn't matter that the ray isn't normalized, in fact I don't think it was before. Is it not working at all now?
And could you post the code that calls this method. Essentially anywhere where you do any matrix stuff.

[Daslee]

Look at my old getPickingRay method. return was:

farVector.sub(nearVector).normalize();

And it worked, but now it doesn't, just if I would stay in center of world and then just sensitive, when I move mouse just per 1 pixels, so it picks through 20 cubes far.

And I'm doing matrix only before ray picking check when rendering world objects.

world.render(); //Here
		for(int i=0; i<world.objects.size(); i++) world.objects.get(i).selected = false;
		
		
		if(world.rayPicking){
			Vector3 destination = getPickingRay(Display.getWidth() / 2, Display.getHeight() / 2);
			
			Ray rayPick = new Ray(new Vector3(control.position.x, control.position.y, control.position.z), destination);
			int index = -1;
			float minDepth = Float.MAX_VALUE;
			for(int i = 0; i < world.objects.size(); i++) {
				float depth = 0;
				if(rayIntersection(rayPick, world.objects.get(i))){
					depth = world.objects.get(i).z;
				}else{
					depth = Float.MAX_VALUE;
				}
				if(depth < minDepth){
					minDepth = depth;
					index = i;
				}
			}
			if(index != -1) world.objects.get(index).selected = true;
			selectedObjectIndex = index;
		}

And after this code is doing matrix with player camera "lookThrough" method.

public void lookThrough(){
		glRotatef(pitch, 1.0f, 0.0f, 0.0f);
        glRotatef(yaw, 0.0f, 1.0f, 0.0f);
        glTranslatef(-position.x, -position.y, -position.z);
	}