gluProject/gluUnproject problem (bug??)

[curmoISU]

I'm using gluProject and gluUnproject pretty extensively in a project and I am having some extremely strange behavior that I just can't solve.

Most of the time, gluProject and gluUnproject work perfectly for me.  However, about a week ago I happened to place my camera
at:

Position: (0.0, -20.0, 0.0)
Forward: (normalised): (0.0, 1.0, 0.0)
Up (normalised):  (0.0, 0.0, -1.0)

At that position, projections start failing (for example: gluUnproject projects the thing slightly behind the camera) ..  

As I move the camera around (forward/back or strafe around) it intermittently works perfectly (At certain positions), sometimes unprojects behind the camera and sometimes just completely messes it up and I get artifacts all over the place.

The odd thing is, however, if I rotate the up vector and then move slightly, bang - everything (projections) start working perfectly.

Now, normally I only grab the matrices (modelview/projection/viewport) when their respective matrices change, but for testing purposes (trying to fix this bug) I do glGet's on all 3 of them immediately before doing a gluUnproject of gluProject...

I need help!!  I've burnt a week on this and I can't figure it out.  I can't find any relationship between positions/rotations that work and those that don't.  I've tried keeping a list of all the positions and that do and don't, along with the projection and modelview matrices, but I just can't find a correlation.  I can post sample code if necessary..

The only thing that I've -thought- might be causing the problem is that my normalise method for the up/forward sometimes get slight rounding errors (6.14e-17 instead of 0.0, etc), but I played around with that trying to force things to be -exactly- length 1 and that didn't really help...

======================================
I decided to add some code:

When the viewport/window changes the projection matrix is setup as follows:

       
GL11.glMatrixMode(GL11.GL_PROJECTION);
GL11.glLoadIdentity(); // Reset The Projection Matrix

 // Calculate The Aspect Ratio Of The Window
GLU.gluPerspective(
	(float)fovY,
	(float)width / (float)height,
	(float)nearClipPlane,
	(float)farClipPlane);

Then, in the normal rendering loop:

       
GL11.glMatrixMode(GL11.GL_MODELVIEW); 
GL11.glViewport(x, y, width, height);

GL11.glClearColor((float)clearColor.getX(), (float)clearColor.getY(), (float)clearColor.getZ(), 1.0f);
GL11.glClear (GL11.GL_COLOR_BUFFER_BIT | GL11.GL_DEPTH_BUFFER_BIT);

GL11.glLoadIdentity ();											// Reset The Modelview Matrix

//Look where the current actor is looking
GLU.gluLookAt((float)cameraActor.getPosition().getX(), (float)cameraActor.getPosition().getY(), (float)cameraActor.getPosition().getZ(),
					((float)(cameraActor.getPosition().getX()+cameraActor.getForward().getX())), ((float)(cameraActor.getPosition().getY()+cameraActor.getForward().getY())),((float)(cameraActor.getPosition().getZ()+cameraActor.getForward().getZ())),
					(float)cameraActor.getUp().getX() , (float)cameraActor.getUp().getY(),(float)cameraActor.getUp().getZ());

My project/unproject code then looks like (unproject is here for the example, project is almost identical):

float[] projArray = new float[3];
recalcViewMatrices();        
        
GLU.gluUnProject((float)point2D.getX(), (float)point2D.getY(), (float)point2D.getZ(), modelViewArray, projViewArray, intViewArray, projArray);
	    

and the recalcViewMatrices is as follows (Remember, this is for testing only, I normally don't get all 3 of these for every projection only when they change, but i am when trying to debug this)

   

       
FloatBuffer modelView = BufferUtils.createFloatBuffer(16); 
FloatBuffer projView = BufferUtils.createFloatBuffer(16);
IntBuffer viewport = BufferUtils.createIntBuffer(16);
			
GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, modelView);
GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, projView);
GL11.glGetInteger(GL11.GL_VIEWPORT, viewport);
	
modelViewArray = getFloatBufferAs2DArray(modelView);
projViewArray = getFloatBufferAs2DArray(projView);
intViewArray = getIntBufferAsArray(viewport);
needViewArrayRecalc = false;

[Fool Running]

Can you give any camera locations/positions that worked?
Have you tried using glTranslate, glRotate, etc. instead of whatever and see if everything works then?

The odd thing is, however, if I rotate the up vector and then move slightly, bang - everything (projections) start working perfectly.

Does moving around work after that? or just that initial change?

[curmoISU]

Moving around after that does work for the most part, yes.  It still has the occassional position it gets to that hiccups.  I really shouldn't say position either, I can stay at one position and rotate around the up vector and sometimes projections work, sometimes they don't - however, I can keep the same orientation and just move the position too and sometimes that works, and sometimes that doesn't.  The only surefire thing seems to be to rotate the up vec. slightly around the forward vec and then move.

I wrote my own gluLookAt function (Because I thoguht maybe gluLookat had the bug because simple glTranslatef's seemed to work perfectly) that basically builds the rotation matrix, and does a glMultMatrix  and then translates the position.  It has the same problem.  I'm really wondering if its a floating point precision problem with glMultMatrix ??  I don't know, i'm at a loss.

I'll be able to get my list of working vs. non working positions and post them here tonight.

[curmoISU]

Here are examples working and non-working modelview matrices.  I am also including the projection matrix used (it doesn't change, so its the same for working an non working)

My test case is to unproject point <10, 10, 0.1>  to 3D space.  My near and far clip planes for gluPerspective are 0.1 (tried 0.1 -> 2) and 1000.0 respectively.  These are the matrices that end up being used during that unprojection.


Projection Matrix: (from glGet)
1.81066, 0.0, 0.0, 0.0,
0.0, 2.4142134, 0.0, 0.0,
0.0, 0.0, -1.0001999, -1.0,

0.0, 0.0, -0.20002, 0.0,

Viewport Matrix: (from glGet sometimes/by hand sometimes)
0, 0, 1024, 768

Now the modelview matrix - this changes of course as the camera moves.  It is positioned using gluLookAt.  This one seems to be the problem.  The first 2 working examples unproject the test point correctly, the other 2 examples do it incorrectly (behind the camera or just randomly weird).  The xE-17 #'s are floating oint inprecision during
my normalise method that I discussed in my first post....

Working (2 examples)

1)
-0.9820042, 0.0, -0.1888589, 0.0,
0.1888589, 6.123234E-17, -0.9820042, 0.0,
1.1564272E-17, -1.0, -6.013042E-17, 0.0,
3.777178, 1.2246469E-15, -16.540085, 1.0

2)
-0.92490906, 0.0, 0.3801884, 0.0,
-0.3801884, 6.123234E-17, -0.92490906, 0.0,
-2.3279827E-17, -1.0, -5.6634343E-17, 0.0,
-4.537527, 1.2246469E-15, -43.622128, 1.0


Non-working (2 examples)

1)
-1.0, 0.0, 0.0, 0.0,
0.0, 6.123234E-17, -1.0, 0.0,
0.0, -1.0, -6.123234E-17, 0.0,
0.0, 1.2246468E-15, -20.0, 1.0,

2)
-0.96638995, 0.0, 0.25708055, 0.0,
-0.25708055, 6.123234E-17, -0.96638995, 0.0,
-1.5741644E-17, -1.0, -5.917432E-17, 0.0
12.658388, 1.2246469E-15, -21.327799, 1.0

[curmoISU]

I still haven't been able to figure this out.  I've tried everything!  Does anyone have any ideas?  This is really a show stopper and has been for nearly 2 weeks...

[Matzon]

sorry, can't help - but if you believe this to be a opengl question you might want to try http://opengl.org otherwise, an example to prove the error might be in order?

[tomb]

The glu functions used in LWJGL was ported from c to java. Have a look at the source if you suspect there is a bug. It is part of the lwjgl source that can be downloaded on sourceforge. Try comparing it to c source like MESA to see if you can find something wrong.