First of all, buffers are very simple. Think of them as just a container of an array of values that we use to communicate with openGL. The method you have posted is not one I'm familiar with and after a quick look through I can't see any errors in the code however I cannot see how it would be very efficient and would advise you try out openGls selection mode for which I will write you some sample code. Be warned It is a long time since I have done this and there might be an error in my code.
public int getIndexOfSelectedObject(int mouseX, int mouseY) {
int bufferSize = 128; //This number is the size of the buffer for openGl to write your results to. I chose 128 because I'm not sure how
//many objects your likely to be drawing. If the maximum number of objects potentially under the mouse is n then
//the size should be n * (3 + sizeOfNameStack). In this example there will be 1 name per object therefore the
//name stack size is 1 and the size should be n * 4.
IntBuffer resultBuffer = BufferUtils.createIntBuffer(bufferSize);
glSelectBuffer(resultBuffer); //Tells openGl to write results to this buffer.
glRenderMode(GL_SELECT);
glInitNames(); //Initialize openGLs name stack into which we will put the name of each object (i will use its index in your array)
glPushName(0); //Adds 1 to the size of the name stack (it starts off as 0).
IntBuffer currentViewport = BufferUtils.createIntBuffer(4);
glGetInteger(GL_VIEWPORT, currentViewport); //If you are storing the values you use for glViewport then you can put them in instead
//You must have setup the correct projection matrix before you do the next line.
FloatBuffer currentProjectionMatrix = BufferUtils.createFloatBuffer(16);
glGetFloat(GL_PROJECTION_MATRIX, currentProjectionMatrix);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPickMatrix(mouseX, mouseY, 1, 1, currentViewport);
glMultMatrix(currentProjectionMatrix);
for(int i = 0; i < world.visibleCollidingObjects.length; i++) { // I'm presuming visibleCollidingObjects is an array and not one of the
// collections
glLoadName(i) //Replaces the name on the top of the name stack with the argument.
world.visibleCollidingObjects[i].drawForSelection(); //You should write a new method that only gives position data, no normals,
// colours, textures etc.
}
int numberOfHitRecords = glRenderMode(GL_RENDER); //Switches back to regular mode and returns the number of hit records written.
//If this returns a negative number you need to supply a larger selection buffer.
//Each hit record contains: the number of names on the stack at the time (always 1 for us), the minimum z value of the object, the
//maximum z value of the object, a list of the names on the stack (just the objects name for us).
int minimumZValue = Integer.MAX_VALUE;
int nameOfClosest = -1;
for(int i = 0; i < numberOfHitRecords; i++) {
int stackSize = resultBuffer.get(); //Unused by us. If not 1, somethings gone wrong.
int thisMinimumZValue = resultBuffer.get();
int maxZValue = resultBuffer.get(); //Unused by us.
int name = resultBuffer.get();
if(thisMinimumZValue < minimumZValue) { //This means this result is closer to the viewer that the previous and is therefore
// preferable.
minimumZValue = thisMinimumZValue;
nameOfClosest = name;
}
glMatrixMode(GL_PROJECTION); //These lines just return the projection matrix to what it was before. Not necessary but neater.
glLoadMatrix(currentProjectionMatrix);
return nameOfClosest ; //Which is the index of the selected object.
}
}
I checked through this a few times, but as I say it's been a while since I last used this and I apologize in advance if I got something wrong.