When implementing this:
http://lwjgl.org/wiki/index.php?title=Using_Shaders_to_Calculate_Model%27s_Position

I noticed a problem. The rotation seems to rotate about the origin rather than the position the objects have been translated to.

Short of doing the matrix math all by myself, can anyone offer some help as to how to properly calculate, in the shader, the rotation about the new position rather than the origin?

Obviously, I use the mouse and keyboard for movement; currently I'm doing three glRotate calls: one to set yaw (heading) directly, and the other two translate the mouse pitch (y movement) into a combination of roll and pitch depending on the yaw (heading). The basic formula is roll = cameraPitch*sin(yaw*PI) and pitch = cameraPitch*cos(yaw*PI). That just means you roll the scene if the player is facing parallel to the GL X axis, and you pitch it if they are facing parallel to the GL Z axis. The percent by which you pitch or roll at intermediate positions is determined by the circle functions.

This works, but I'm uncertain as to where GLRotate is done; I'd prefer to do the work in the shader if possible since rotating a fair number of vertexes can be computationally intensive.

Looking at the shader code, (I've successfully linked this up, so there doesn't seem to be a problem getting the floats into the uniform.)

uniform vec3 pos;
uniform vec3 rot;
varying vec4 vertColor;

void main(){
    mat4x4 position=mat4x4(1.0);
    position[3].x=pos.x;
    position[3].y=pos.y;
    position[3].z=pos.z;
    mat4x4 heading=mat4x4(1.0);
    heading[0][0]=cos(rot.y);
    heading[0][2]=-(sin(rot.y));
    heading[2][0]=sin(rot.y);
    heading[2][2]=cos(rot.y);
    mat4x4 pitch=mat4x4(1.0);
    pitch[1][1]=cos(rot.x);
    pitch[1][2]=sin(rot.x);
    pitch[2][1]=-(sin(rot.x));
    pitch[2][2]=cos(rot.x);
    mat4x4 roll=mat4x4(1.0);
    roll[0][0]=cos(rot.z);
    roll[0][1]=sin(rot.z);
    roll[1][0]=-(sin(rot.z));
    roll[1][1]=cos(rot.z);

    gl_Position= gl_ModelViewProjectionMatrix*position*heading*pitch*roll*gl_Vertex;
    vertColor = vec4(0.6,0.5,0.3,1.0f);
}

I don't use his vertColor nonsense (I pass through gl_Color to gl_frontColor) but I have noticed that if position is different, heading, pitch and roll seem to rotate the scene around the origin rather than the position. Obviously this isn't evident if you haven't moved.

To reiterate my question, what changes do I need to make to this matrix math to factor in the current position into the rotations? With his math a particular yaw value (for instance, .5 radians) still rotates the vertices to the same position no matter what the camera position is. Thus you get the weird effect of rotating objects 'into' you when you approach them to one side and try to look around.

My thought was something like:

uniform vec3 pos;
uniform vec3 rot;
varying vec4 vertColor;

void main(){
    mat4x4 position=mat4x4(1.0);
    position[3].x=pos.x;
    position[3].y=pos.y;
    position[3].z=pos.z;
    mat4x4 heading=mat4x4(1.0);
    heading[0][0]=cos(rot.y);
    heading[0][2]=-(sin(rot.y));
    heading[2][0]=sin(rot.y);
    heading[2][2]=cos(rot.y);
    heading[3][0]=pos.x;
    heading[3][1]=pos.y;
    heading[3][2]=pos.z;
    mat4x4 pitch=mat4x4(1.0);
    pitch[1][1]=cos(rot.x);
    pitch[1][2]=sin(rot.x);
    pitch[2][1]=-(sin(rot.x));
    pitch[2][2]=cos(rot.x);
    pitch[3][0]=pos.x;
    pitch[3][1]=pos.y;
    pitch[3][2]=pos.z;
    mat4x4 roll=mat4x4(1.0);
    roll[0][0]=cos(rot.z);
    roll[0][1]=sin(rot.z);
    roll[1][0]=-(sin(rot.z));
    roll[1][1]=cos(rot.z);
    roll[3][0]=pos.x;
    roll[3][1]=pos.y;
    roll[3][2]=pos.z;

    gl_Position= gl_ModelViewProjectionMatrix*position*heading*pitch*roll*gl_Vertex;
    vertColor = vec4(0.6,0.5,0.3,1.0f);
}

Also ultimately I'm trying to make the math as efficient as possible, but first I'd just like it to work the way it 'feels' it should vis a vis moving around in a 3d scene.

In matrix transformations, the order of operations is important.

Unless I'm completely mistaken, this:
gl_Position= gl_ModelViewProjectionMatrix*position*heading*pitch*roll*gl_Vertex;will cause the transformations applied by 'position*heading*pitch*roll' be performed in screen space.

Seems fishy. If you reorder the operations, you first apply all transformations in model-space, then convert to screen-space:
gl_Position= gl_ModelViewMatrix*position*heading*pitch*roll*gl_ProjectionMatrix*gl_Vertex;

Ah! I will give that a try. But am I re-doing a math operation that is already done by doing it this way?

Doesn't exactly work... causes the rotation to occur around a center point fixed ahead, and flattens out all objects.

Try:

gl_Position= gl_ModelViewProjectionMatrix*heading*pitch*roll*position*gl_Vertex;
I'm not sure what you're trying to achieve, but it's a bad idea doing 4 extra matrix multiplications in the shader. The model transformation should be done on the CPU and you should only use the final matrix in the shader.

Model transformation? You mean moving the vertices around -- ? I was doing on the CPU, but wanted to know if I could offload the work to the GPU. If it's simpler to use 3 glRotates to get the scene into the right position, I'm all for it, but I want to make sure I have a rudimentary vertex shader that offloads a fair amount of work onto the GPU leaving more juice for abstract calculations.

Ah, this seems to have removed a lot of necessity for multiplying by PI to get proper forward motion and roll + pitch amounts.

Hats off to you gentlemen!

When I say model transformation I mean the position/orientation of the model in world space, not the vertices moving around (if it's an animated model). If you have 4 parameters that define that final transformation, in your case heading/pitch/roll/position, then you can build a matrix on the CPU that is the product of all 4. This is called the Model matrix in OpenGL. The corresponding camera transformation is called the View matrix. Multiply the Model with the View matrix and you get the ModelView matrix. Multiply again with the Project matrix and you get the ModelViewProjection matrix.

If you do all of that on the CPU and simply set the result to the current MVP matrix, all you have left to do in the shader is:

gl_Position= gl_ModelViewProjectionMatrix * gl_Vertex;
That's a lot of calculations saved compared to what you're doing. The GPU might be very good at quickly doing stuff per-vertex or per-fragment, but it still has limits. You trade a bit of CPU work, which is done per-model and not per-vertex, for much quicker shader execution. You also have less uniforms to update per render-call.

Hm, but this just seems to be a question of how much the GPU can do, not so much a categorical distinction. Translating pieces of models of things such as player arms when its root connection rotates is no different than rotating the scene, it's just a question of how many translations are being done. What you seem to be telling me is that you shouldn't do camera rotation in GPU, but doesn't doing or not doing that relate merely to the complexity of the scene, not so much to the category of the activity being done? So for instance, Crysis would have too many vertices in its scenes to be doing both camera translations and all the other translations (sure), but what about games with simpler visual styles such as Voxatron?

Of course it depends on the game you're making. You could get away with it in simpler games, but the fact remains that matrix multiplication is expensive and you probably want your simple game to run on simple (slow) hardware. Matrix-vector multiplication is 4 dot products, matrix-matrix is 16.

Character parts are usually animated with bones in a skeletal hierarchy. In that case, the MVP matrix will contain the transformation of the skeletal hierarchy root. You'd also have a uniform array that contains the bone transformations relative to the base pose for the current frame. This is a valid case of doing 2 matrix-vector multiplications per-vertex, because you really want to make a single draw call for the whole character model, not one per model part.

Voxatron uses some kind of voxel rendering, which I'm not familiar with. But I do know there are no vertices or models in the classical sense in voxel engines, so what we talk about here doesn't apply there.

The bottom line is, try to upload a single transformation matrix per draw call. It's easy to do and it pays off in performance. Also keep in mind that uniforms take resources. In the skeletal animation case, you need every spare uniform to hold the bone matrices. This creates problems when you have too many bones and the GPU has too few available uniform slots. If you hit the limit you have to break up your model and do multiple draw calls, which is costly in both performance and programming work you have to do.

Well, I'm trying to deal with models in a unique way. When I got into this first (I was doing research) I found that the world of 3d models is a kind of hell all of its own. To this end, I decided (and based on the requirements for customizability of characters and monsters) to create voxel-part-skeletal models. The game creates a model cache from png files that declare the color and position of the voxels in each skeletal piece. The animation of models are then contained in logic sets attached to the different model classes according to hierarchy and type. For example, while the PNG file may be indentical, you would get a different result if you applied it to the modelclass 'jock kid' versus the modelclass 'nerd girl' (for example.) Being all extensions of the same hierarchy starting with Biped which implements the interface Drawable, you can ask for the same animations (walk, punch, jump, etc) but the result varies according to the implementation of that motion (stored as rotations of joints and maybe distortions of the pieces connected to them such as slight stretches that may be required to imitate common gestures) however we provide a 'default' animation logic set for each at least for things like Biped meaning some actions will result in the same animation of differently shaped parts. This is done on a case-by-case basis.

Typically, most 'blocky' or simple games also have simple animation, but I'm wondering how expensive it is to make, instead of the models complex, the gestures reasonably complex. The modelcaching should simplify the shape resulting from the voxels from the PNG, so models may turn out to be fairly simple. Considering that I'm not merely replaying a series of pre-recorded model frames but synchronously moving the parts --we need 5 different run speeds, so calculating the movement rate against the actual speed instead of playing a preset model animation that has to be carefully synched with the possible movement speed is optimal.

This doesn't begin to consider the need for hit box calculation based on model part movement, time distortion, and then the other thread which handles world simulation. Movement will be precise and complex, with for instance tracking how fast the player changed direction to see if they would need an acrobatic feat to actually go through with that move.

A small update in case someone stumbles upon this thread.

The rest of that vertex shader is correct, but the last line (the several multiplications) needs to be:

gl_Position= gl_ModelViewProjectionMatrix*roll*pitch*heading*position*gl_Vertex;
This is so that you start with the verticies, move them to where the camera is, change their yaw (heading) *first*, then their pitch, and lastly, their roll. If it isn't done in this order you need to do extra calculations just to get the motion you see to agree with what you think you should be getting with the mouse. My supposition is that different camera schemes (such as a flying or swimming camera) may benefit from differing orders of rotations, but this one produces the 'Quake style' camera motion we usually expect.