I'm not sure why the values are like that, but it doesn't look like you're allocating the buffer correctly.
Luckily, I used gluProject fairly recently; here's the copy & paste of the code that worked for me:
//Triangle points
FloatBuffer pos1 = BufferUtils.createFloatBuffer(3);
FloatBuffer pos2 = BufferUtils.createFloatBuffer(3);
FloatBuffer pos3 = BufferUtils.createFloatBuffer(3);
FloatBuffer model = BufferUtils.createFloatBuffer(16);
FloatBuffer proj = BufferUtils.createFloatBuffer(16);
IntBuffer view = BufferUtils.createIntBuffer(16);
glGetFloat(GL_MODELVIEW_MATRIX,model);
glGetFloat(GL_PROJECTION_MATRIX,proj);
glGetInteger(GL_VIEWPORT,view);
gluProject(-50,-50,0,model,proj,view,pos1);
gluProject( 50,-50,0,model,proj,view,pos2);
gluProject( 0, 50,0,model,proj,view,pos3);
pos1.get(0); //x
pos1.get(1); //y
pos1.get(2); //z
pos2.get(0); //x
pos2.get(1); //y
pos2.get(2); //z
pos3.get(0); //x
pos3.get(1); //y
pos3.get(2); //z