picking and transparency

[tomster]

Hello,

I am thinking about using opengl picking vs. ray picking.

My program is a pretty simple 2D program so picking will probably be sufficient and i won't have to get into reverse projecting and intersection detections.

There is one thing i couldn't find the answer to and i'd like to know before i choose.

When i have a polygon that has a texture on it, and part of the polygon is transparent (due to 0 value in the texture alpha channel), when i click on the transparent part will it get picked or not (obviously i'd like it to not get picked).
Also, in this same case how would ray trace picking help?

Thanks for your help

[Kai]

Hi,

When i have a polygon that has a texture on it, and part of the polygon is transparent (due to 0 value in the texture alpha channel), when i click on the transparent part will it get picked or not (obviously i'd like it to not get picked).

If you have alpha testing (glAlphaFunc, glEnable) enabled, rejected fragments (specified via glAlphaFunc) will not be pickable.

Note however that the selection mode in OpenGL is marked deprecated as of 3.0 and should be replaced by a custom ray/object intersection method.

Also, in this same case how would ray trace picking help?

If you have transparency in textures, I know of no way to get this done with manual ray-intersection, because that would require to reproduce the whole rasterizer process by hand.

[tomster]

Hey again,

I did as you said. Doesn't work though. The transparent areas still get picked when i click on them.

Do i have to enable alpha testing while in selection mode? Its enabled in the opengl init method (i enabled it and disabled blending - just in case).

Could it be the texture image?
It a png file and it gets rendered fine (transparency wise) so i figure it's alpha channel is fine.

I am thinking about color coding instead of picking but how would i render all the texture in one color while still not rendering the transparent parts?

thanks

[Kai]

I'm so sorry, I dug in it again, and found that while drawing primitives in selection mode, they do not reach the rasterizer, but will be intercepted after clipping. I was sure that the selection is based on the depth of the fragment being written into the framebuffer, which, when the fragment is being rejected, would not get updated, which would make sense. But I was wrong.

So, what can you do about it:
You can still use alpha testing but you cannot use the selection mode anymore. This way, you lose the ability to identify objects, because you cannot assign names to them anymore. What you could do instead is color-code the objects.
So you assign to your pickable objects a unique color in RGB-space while rendering them and afterwards you have to read-back the framebuffer contents and check the color code.
The good news is that you can still use the projection matrix and viewport that you used in selection mode.

Since you are now really rendering something that you do not want to be visible to the user, you might want to use the framebuffer object extension to create youself a renderable texture to which you can render while doing the picking.

[tomster]

ok, lemme see if i got what you're saying correctly.

not reaching the rasterizer means that the fragments that would have been rejected (had they reached the rasterizer) are still there (where ever "there" is for selection processing) and therefore my transparent fragments are still there and get picked, right?. This would account for the results I am getting.

Assuming that is correct - ok, i will use color coding. Shouldn't be a problem as I doubt i will be rendering more primitives then i have available colors (at least not around the mouse click point so it should be ok). However, how do i render just the non transparent areas without using the texture? and if i do use the texture how do i get them all the non rejected fragments to be in one color?

The last part i didn't get at all. What is a renderable texture? That's something i am not familiar with.
Why not draw to the regular buffer, not flip it when i'm done, check the color and then clear it and draw the regular stuff?

[Kai]

not reaching the rasterizer means that the fragments that would have been rejected (had they reached the rasterizer) are still there (where ever "there" is for selection processing) and therefore my transparent fragments are still there and get picked, right?

Yes, but before the rasterizer, there are no fragments. The selection mode just asks the clipping algorithm whether or not any polygon of a named object would generate fragments.

However, how do i render just the non transparent areas without using the texture? and if i do use the texture how do i get them all the non rejected fragments to be in one color?

The first way I can think of of doing this is to use a small custom fragment shader (GLSL) that only blend in the texture's alpha channel and uses the RGB-color-code and not the texture's color.
This way you have full control of the shading process and you do not have to worry about lighting and all the other fixed-function stuff, which you don't need and hence would not need to disable with many state-changing commands.

The last part i didn't get at all. What is a renderable texture? That's something i am not familiar with.
Why not draw to the regular buffer, not flip it when i'm done, check the color and then clear it and draw the regular stuff?

Yes, of course, you are right. I just thought that rendering not to the framebuffer (but to a special texture) would be more clean.
If you want to learn about "offscreen-rendering" which would be the name for "render into a texture", you can have a look at http://www.songho.ca/opengl/gl_fbo.html.

[tomster]

All right

thanks a lot, you've been a big help .
Ill look into that custom shader and see what works.

One last question. is it possible to create an "alpha texture"?
Something like use the alpha channel of the image to create a texture that when applied only has alpha information and leaves the color as is?

[Kai]

One last question. is it possible to create an "alpha texture"?

I thought about it, too, while writing the last response and it occurred to me that a texture can have alpha or not, but always has color, which is bad for what you are trying to do. This was the time I came up with the shader part, because there you have total control over everything.
So the final line in your fragment shader code would look something like this:

gl_FragColor = vec4(colorCode.rgb, texture2D(textureWithAlpha, gl_TexCoord[0]).a);

with "textureWithAlpha" being of type "sampler2D" and sampling your texture whose alpha channel you are interested in, and "colorCode" being a "vec3" containing the current color-code.

[Kai]

If you are interested in learning about the shader stuff (which is real fun!), I suggest http://www.lighthouse3d.com/opengl/glsl/.
I also could not resist in writing the fragment shader code for you (which is not really hard to do), here it is:

uniform vec3 colorCode;
uniform sampler2D textureWithAlpha;
gl_FragColor = vec4(colorCode, texture2D(textureWithAlpha, gl_TexCoord[0]).a);

You know have to bind your texture with alpha to a texture unit (0 per default) and assign the colorCode-variable in the shader your custom color-code (The lighthouse's site describes how to do that).
If you follow the setting-up code in the lighthouse's shader tutorial, you will pretty soon have a working version of alpha-enabled picking :-).

[tomster]

awsome!

thanks a lot.
(just to be sure: we are still talking about some sort of color coding method and not actual picking in selection mode right?)

I will go over all that tonight. I must get some actual work done now or i will have a lot more time for my pet projects when im unemployed!!

[Kai]

we are still talking about some sort of color coding method and not actual picking in selection mode right?

Yes, render your pickable objects like usual (no selection mode involved).
The rendered vertices will go through the standard vertex transformation and the rasterizer generates fragments which will go through your custom fragment shader and afterwards you can use glReadPixels to read-back the framebuffer contents.
You still have to leave alpha-testing enabled for this to work, of course!