Billboards - Best practices [SOLVED]

[Source_of_Truth]

DEAR FORUM USER, IF THIS IS THE FIRST TIME THAT YOU ARE VISTING THIS THREAD, BETTER JUMP AHEAD TO THIS POST. Thanks 

I have implemented a spherical billboard via the vertex shader by applying additional rotations to the object being rendered. Basically, that means that I first compute the horizontal and vertical angles an object must be rotated to "face the camera", generating rotation matrices for them and then just updating gl_Position accordingly. For simplicity, assume it code looks like this:

gl_Position = projectionMatrix * viewMatrix * translationMatrix * rotationLeftRight * rotationUpDown * vec4(inVertexPosition, 1.0);

Low, let's since we are doing billboards, the object is a textued quad for now.

Consider for example this well-know face of evil which can be seen below. Let's say its located at (0,0,0) while our camera is at (0,0,1), looking towards it, aka into (0,0,-1). We can see that it is being rendered correctly (never mind the texture filter ).



Now, moving the camera to the left and updwards, from (0,0,-1) to something like (1,1,0) for example. We rotate the camera back towards more or less the origin and we arrive at the next image:



We see that the object remains facing towards us, while the terrain, rendered with a standard, non-billboarding shader, obviously does not (never mind that the object appears a little larger, I was not too precise when taking these example shots).

Now, not moving the camera but only rotating it to the right to face the edge of the map, the perspective matrix kicks in, "destroying" our billboarding sprite's effect, as seen in below:



Illustrating the problem in an easier to see manner, I have created this handy image, in which I have set transparent fragments to red, thus making the quad visible  .



Mathematically speaking, everything going on here is obviously correct, both the spherical billboard as well as the perspective matrix is doing its job. From a sprite's perspective however, said matrix is introducing a unwanted rotation once the object is no longer the center. Basically, it has tilted to the left as the camera went looking to the right, as that is the "job" of the perspective matrix, creating perspective. However, rendering the billboards utterly separate with a different perspective matrix "detaches" them from the rest of the scene and creates a whole new set of challenges, for example in regards to the depth buffer.

One possiblity to fix this would be to apply an additional rolling-rotation based on the horizontal viewing angle difference but when that thought came to my mind, I decided should stop and first ask around.

My question: I am approaching this in an entirely incorrect way?

[KaiHH]

You haven't really told how you actually want it to look like. But from your post, it seems, you want the character rectangle to always align with the camera plane (that is, be perpendicular to the camera view direction), so as to mimic the effect of the character simply having been rendered using a 2D projection without any 3D rotations.

You can achieve this by simply eliminating the rotational component of the view matrix.

So, assuming V is your view matrix (world -> camera space), M is your model matrix (model -> world) without any billboard rotations and P is your projection matrix, then the resulting matrix 'R' you need is:

R = P * V * M * r(V)^-1

where r(X) is the upper-left (rotational) part of the given matrix X without any translational component. And X^-1 is the inverse of X, which for orthonomal matrices (rotation without scaling) is simply the transpose of the matrix.
If you need to have your view matrix always separate you need to use the above equation.

However, if you want to factor-in the billboard transformations into V, then V will simply be the camera translation without any rotational component, leaving you with P, V (without any rotational component) and M when rendering your billboard character with P * V * M.

[Source_of_Truth]

be perpendicular to the camera view direction

Exactly, I want to align the quad with the horizon regardless of camera angle, while keeping the "rotate it towards the camera's position"-part.

[KaiHH]

This was not what I meant with the rectangle (from now on be simply called 'P') being perpendicular to the camera view direction or perpendicular to the camera near/flar plane's normal.
What you are talking about is basically having P's rotation constrained to the world's Y axis, but no rotation applied on the X or Z axes.
Or in other words: P will always "stand straight" in the world but will only rotate around the world's Y axis such that P's normal vector is collinear to the camera's view direction or the camera's near/far plane normal.

This can be achieved with:

R = P * V * M * (r(V)^-1)[col1 = (0, 1, 0, 0)]

where (X)[colN = V] is: X where the column with index N (0..3) is replaced by vector V.
This will constrain the rotation only around the Y axis.
So it's basically the same as the "always perpendicular to the camera's direction" but with the constraint of only allowing rotation around the Y axis of the object which is the additional operation on the inverse of r(V).

[Source_of_Truth]

This was not what I meant with the rectangle (from now on be simply called 'P') being perpendicular to the camera view direction or perpendicular to the camera near/flar plane's normal.

No, this would simply be a cylindrical billboard. After reading above posts again, I think I have been going about this thread the wrong way. Instead of talking tech, I am going to prepare an example that describes what I actually want, then we can discuss how, if at all, one can achieve it. Please note that your posts are much appreciated anyway. After all, this is basically the problem:

You haven't really told how you actually want it to look like.

Stand-by 

[Source_of_Truth]

Alright , here goes:

We have a standard 3d world from standard geometry (untextured here for contrast), rendered in the "classical" sense. By that, I mean that the projection matrix is a symmetric perspective projection frustum transformation. And it looks like this at a FOV of 75 degrees rendered at more or less Full HD:



Now, as you can see, the above screenshots shows terrain only. The idea was to marry this full 3d terrain with an old-school sprite based look, that means all objects on the terrain should not be made out of actual geometry but consist only of pre-rendered or drawn images, like in the good old days . The thing about these classic games is, that they ever only had sprites of an object at exactly one particular vertical angle:

* in games like Doom or Wolfenstein, all sprites were drawn to accommodate a player looking at them from the same elevation as the object the sprite is representing
* in games like Myth: The Fallen Lords or the early Total War games, all sprites were drawn to accommodate a player looking at them from a sort of 50 degrees angle above the object the sprite is representing

While Doom's raycasting engine was involved enough to "simulate" elevation differences, the game's actual vertically in the level design had to be limited in order to not betray the effect (and at various points in even the original levels, the illusion broke down hard due to elevation differences). In the RTS-games mentioned above, both camera movement and degrees of possible rotation had to be controlled and restricted a lot for the very same reason.

Again, the idea was to achieve a similar old-school look but avoid these restrictions, with our nice, modern gigabytes of RAM. As such, I have spritesheets that not only show the object at different rotations but also seen from different angles! Below an examplary image, to illustrate what I mean by that (this is not a real nor a complete spritesheet and it is not pixel accurate either):



This means, I had to come up with a way to combine the actual 3D geometry and the pseudo-3D like the programmers of old. In contrast to Doom's utterly fake ray-caster however, we have modern OpenGL and after doing some investigation into possible techniques, I arrived at using spherical billboards and switching textures based on the angle. The reason: I obviously need a "canvas" to render the texture onto and that canvas should always be perpendicular to the camera's look-at angle. Instead of trying to reconcile two utterly different notions of what space and perspective is, I thought rendering it all in the standard way and just "rotating the rectangular sprite-objects to face me" should solve the problem quite easily and early attempts looked promising. However, in practice, the effect breaks down. Consider the following:



Here is a nice line of tree sprites, never mind the transparency-problems. I have chosen trees in this example because their trunk's will illustrate the problem quite nicely later. Our camera is level towards this group, that means the billboard-effect in question is currently not a spherical but a cylindrical one, as there is no vertical rotation to speak of.

If we move and rotate the camera around the group, everything looks fine. In the example below, I rotated counter-clockwise (see the mountain in the background). The billboards continue to face the camera and the texture-switch has exchanged the sprite-texture for a horizontal and vertical viewing angle of about 0 degrees with another simulating the horizontal viewing angle of about 45 degrees, but a still level vertical one.



Going back to the original position, we now move the camera upwards. Again, the textures are exchanged, the spherical billboard continues to face us but the effect breaks down hard  .



The relevant sprite sheet for a vertical angle of about 45 degrees looks like this:



We can see that all the trunks are perfectly vertical and nothing there is angled. Now, there are two possibilites here:

1.) I simply have a bug in the billboarding shader!

While this is always possible, visual investigation seems to indicate otherwise. Doing the same trick as in the OP above, namely just coloring discarded transparent pixels as red, once can see the rectangles and they appear to behave correctly:



I had to space out the trees of the example, to make this image visually digestable but otherwise, everything is the same. It is important to mentioned that I am currently using what I call a stepped spherical billboard, that means I am not facing the camera exactly. Instead, the billboard can only rotate in the exact number of sprite angles for that axis. Since the tree as sprites for 8 possible horizontal angles, that means the billboard is either rotated 0 degrees, 45, 90, and so forth. This can be nicely observed in the picture.

2.) We are facing the opposite of an optical illusion!

Weirdly enough, the picture with the non-red trees looks terrible while the one with the red-background trees looks fine, at least to me. This is seemingly due to the fact that our brain latches on to the now visualized rectangle, deems that to be okay enough and no tree looks terribly bend or angled anymore to me. So basically, everything looks terrible to our tiny lizard brain until does stops doing so.


I thought about this entire problem for a while now and I think that everything we can observe here is actually doing the exact job that I programmed it to do, especially the perspective projection matrix. But obviously, it really sucks and looks terrible. Combining 2d and 3d in a way that looks both good and interesting can be really hard, a statement supported by the various blog posts on the internet regarding this topic.

I am looking for input here on how to improve the look of this and look forward to discussing if my entire approach is wrong, which might very well be the case.

[KaiHH]

Okay, in this situation the problem is that you only have eight pre-rendered impostors for only eight discrete directions and it will only ever look correct if your camera direction precisely matches the direction from which a particular impostor image was generated.
On top of that, it will also only look correct when the perspective projection the impostor was rendered with matches the one of the camera in the scene. For example, things will look incorrect when you move too close to the impostor when it was generated/rendered with a perspective/camera farther away.

I'd suggest another approach: Measure by how much a given impostor becomes "incorrect" and then regenerate it on the fly in-game using true impostors [1].
Also, with the way you are currently making use of the statically pre-generated set of impostors: will the switch from one image at a given angle to another one for a different angle not be very noticeable and distracting during gameplay?

Also: You'd probably need correct depth information for close-by objects for e.g. light/shadow mapping. For this I'd also recommend true impostors to retain depth information.

[1]: https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-effects/chapter-21-true-impostors

[Source_of_Truth]

Only have eight pre-rendered impostors for only eight discrete directions and it will only ever look correct if your camera direction precisely matches the direction from which a particular impostor image was generated.
On top of that, it will also only look correct when the perspective projection the impostor was rendered with matches the one of the camera in the scene. For example, things will look incorrect when you move too close to the impostor when it was generated/rendered with a perspective/camera farther away.

All of that is true but expected. In fact, the goal is not to have a realistic scene. Normally, the sprites are either rendered using an orthogonal projection or are drawn by hand (aka they are cartoonish).

Will the switch from one image at a given angle to another one for a different angle not be very noticeable and distracting during gameplay?

Yes it will and it is desired. Again, I really appreciate you trying to help but we seem to be constantly miscommunicating .

A true imposter, while certainly be a cool thing (I know said book pretty well), is really not the thing I am aiming for.

The issue is basically this here, what could be nicknamed the "dancing tree", because this breaks all the gestalt-principles and destroys and "feel" of the scene:


This is obviously a threshold-problem in regards to the fixed, "stepped" angle of the billboard. One idea that I am toying around with is to eliminate the "stepped" nature of the billboard rotation in the horizontal axis and employ a continous rotation but the question I am after is, if anyone else has an idea on how to produce a ClassicDoom-like sprite-based look with vertical angle support, without resorting to render-object-to-texture or other framebuffer-related trick (or even the imposter).

EDIT: As suspected, everything is perfectly correct here. The code performs exactly as intended, just that my expectation was a different one.