I had a similar problem when I first got into unProject().
One problem was that I wasn't giving unProject the correct modelview, projection and viewport matrices.
Another stranger problem, which might just be some sort of newbie error on my part, was here:
public static float getZDepthAt(float worldx, float worldy, float worldz)
{
float[] resultf = new float[3];
project( worldx, worldy, worldz, resultf);
return ((int)(resultf[2] * 10000F)) / 10000f; // truncate to 4 decimals
}
I was doing the same thing as you, I projected a world point to get the equivalent Z depth value. I found that whenever the depth value went to more than 4 decimal places my Z coords became erratic. I tried truncating the Z depth to 4 decimals and got the results I wanted.
I hesitate to bring it up because it's a long shot, but maybe theres a precision issue at work.