gluUnproject Inaccurate

[fabian]

This probably sound stupid, but:
I've previously used gl picking for a tile-based game, and this works perfectly, but now I am trying to get the 3d location of where a user clicked. I tried this:

IntBuffer viewport = BufferUtils.createIntBuffer(16);
            FloatBuffer modelview = BufferUtils.createFloatBuffer(16);
            FloatBuffer projection = BufferUtils.createFloatBuffer(16);
            FloatBuffer winZ = BufferUtils.createFloatBuffer(1);
            float winX, winY;
            FloatBuffer position = BufferUtils.createFloatBuffer(3);
            GL11.glGetFloat( GL11.GL_MODELVIEW_MATRIX, modelview );
            GL11.glGetFloat( GL11.GL_PROJECTION_MATRIX, projection );
            GL11.glGetInteger( GL11.GL_VIEWPORT, viewport );
            System.out.println("M X:"+Mouse.getX()+" Y:"+Mouse.getY());
            winX = (float)Mouse.getX(); winY = (float)viewport.get(3) - (float)Mouse.getY();
            GL11.glReadPixels((int)winX, (int)winY, 1, 1, GL11.GL_DEPTH_COMPONENT, GL11.GL_FLOAT, winZ);
            GLU.gluUnProject(winX, winY, winZ.get(0), modelview, projection, viewport, position);
            System.out.println("Position: "+position.get(0)+","+position.get(1)+","+position.get(2));
            cam.setX(position.get(0));
            cam.setY(position.get(1));
            cam.setZ(position.get(2));

However, This sends the camera ridiculously far away from where I actually clicked; It generally ends up floating above a tile, a few hundred away from where was actually clicked.

Can someone give me some help with this? It's probably easy, but I haven't done it before.

[Mickelukas]

My class works fine and looks like this, maybe it will help you figure it out I use a set z though instead of reading it out (because I can, and because it saves me from drawing the scene twice), so that part won't work in your case.

public final class Math3D {

    private static FloatBuffer modelview = BufferUtils.createFloatBuffer(16); //The model view for caching

    private static FloatBuffer projection = BufferUtils.createFloatBuffer(16); //The projection for caching

    private static IntBuffer viewport = BufferUtils.createIntBuffer(16); //The view port for caching

    private final static FloatBuffer tempBuffer = FloatBuffer.allocate(3); //A temporary buffer

    private final static double[] tempVec = new double[3]; //A temporary double array

    //Generate the private variables before calling the below functions that rely on them
    public static void generateArrays() {
        modelview.clear();
        projection.clear();
        viewport.clear();
        GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, modelview);
        GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, projection);
        GL11.glGetInteger(GL11.GL_VIEWPORT, viewport);
    }

    //Get the position inside the application from a passed position on the applet together with depth, realPos gets set to the return value
    public static void getPoint3DfromPoint2D(final Point2DInteger p, final double zPos, final Point3DFloat realPos) {
        if (p == null) {
            return;
        }
        //Get the coordinates at the far plane
        tempBuffer.clear();
        GLU.gluUnProject(p.getX(), p.getY(), 1f, modelview, projection, viewport, tempBuffer);
        //Get the vector that goes from the UnProject to the camera
        tempVec[0] = Camera.getX() - tempBuffer.get(0);
        tempVec[1] = Camera.getY() - tempBuffer.get(1);
        tempVec[2] = Camera.getZ() - tempBuffer.get(2);
        //Calculate X and Y based on a known Z
        tempBuffer.put(0, (float)((zPos - tempBuffer.get(2)) / tempVec[2] * tempVec[0] + tempBuffer.get(0)));
        tempBuffer.put(1, (float)((zPos - tempBuffer.get(2)) / tempVec[2] * tempVec[1] + tempBuffer.get(1)));
        tempBuffer.put(2, (float)zPos);
        realPos.setLocation(tempBuffer.get(0), tempBuffer.get(1), tempBuffer.get(2));
    }

[fabian]

Well, the tiles are based off a heightmap, so does that mean that I can get the z position? Or is that Y?
Edit: Aaah, Z = the distance from the camera / the depth?
Edit 2: Fixed it. It'd seem you need to render it with depth testing turned off, then do it. Also, I needed to limit the amount of clicks, but that was because I was moving the camera. Thanks for the help though Mick.
Edit 3: Or not, It worked 2ce and has broken...

I've found out it's to do with the cameras rotation, due to the camera simply transforming the entire scene.
I was working on a way to reverse the camera, so I devised this method to reverse the rotation after receiving the rotated coordinates:

    public static Point3f rotate(Point3f point, float pitch, float yaw, float roll) {
        Point3f newPoint = new Point3f(point.getX(), point.getY(), point.getZ());
        //PITCH
        newPoint.setZ((float)(((newPoint.getZ()*Math.cos(pitch))-(newPoint.getX()*Math.sin(pitch)))));
        newPoint.setX((float)((newPoint.getZ()*Math.sin(pitch))+(newPoint.getX()*Math.cos(pitch))));
        //YAW
        newPoint.setX((float)((newPoint.getX()*Math.cos(yaw))-(newPoint.getY()*Math.sin(yaw))));
        newPoint.setY((float)((newPoint.getX()*Math.sin(yaw))+(newPoint.getY()*Math.cos(yaw))));
        //ROLL
        newPoint.setY((float)((newPoint.getY()*Math.cos(roll))-(newPoint.getZ()*Math.sin(roll))));
        newPoint.setZ((float)((newPoint.getY()*Math.sin(roll))+(newPoint.getZ()*Math.cos(roll))));
        return newPoint;
    }

However, this still doesn't work.
From the correct angle (180 yaw, any pitch, 0 roll) it gets it correctly without the rotation, with only the translation reversal.
Any ideas?

[fabian]

Ok, Now I'm confused as hell.
The entire viewport height thing appears to have been wrong. After quoting the viewport.get(3)-Mouse.getY and just using Mouse.getY it works.