Is there a way to Anti-alias Transparent Textures ?

[Jehan]

Hi everyone, thank you for answers ...

I'm trying to make a 3D game.. with 2D entities implemented in this 3D world...

With my 3D model entities, I can set up antialias for edges of my polygons like this :

ContextAttribs attribs = new ContextAttribs(3,3)
				.withForwardCompatible(true)
				.withProfileCore(true);
		
		try
		{
			Display.setDisplayMode(new DisplayMode(WIDTH, HEIGHT));
			Display.create(new PixelFormat().withDepthBits(24), attribs);
			Display.setTitle("display");
			GL11.glEnable(GL13.GL_MULTISAMPLE);
		}

But my 2D entities, are just pictures(with an alpha channel) bound in a simple vertex quad.
for example my picture is a picture of a tree surrounded by transparency.
I'm not really using blending but I discard "alpha < 0.5" pixels of my picture in my fragment shader like this :

vec4 textureColour = texture(modelTexture,pass_textureCoords);
if (textureColour.a<0.5){
	discard;
}

that way I can see my tree in my 3D world and the transparency around the tree is transparent.
so the rendering is like this : (see attachment)

But for exemple the edge of my tree trunk  (the transition from drawing to transparency) is very hard,
and I would like to make it softer !

so my question is :
does LWJGL offer a way to do anti-aliasing on the edge of things drawn in pictures as it does for the edges of polygons ?
or do I have to use blending with GL_BLEND ?

Thankyou ! and Have a good Day !

[Jehan]

[KaiHH]

There is a great post on the problem of alpha tested aliasing and techniques to improve it. The article mentions Unity but the ideas can be implemented in any shader language: https://medium.com/@bgolus/anti-aliased-alpha-test-the-esoteric-alpha-to-coverage-8b177335ae4f

[Jehan]

Hi KaiHH,

thank you for your help.

So I used the alpha to coverage technique as it is written in your link.

To be able to use this technique I must first use the anti-aliasing multi-sample (4x at least),
and I have to render directly to the screen, not to an FBO.

First, I delete these lines in fragment shader : (alpha testing)

if (textureColour.a<0.5)
{
	discard;
}

I enable GL13.GL_SAMPLE_ALPHA_TO_COVERAGE and disable it before and after rendering my triangles.
like this:

GL11.glEnable(GL13.GL_SAMPLE_ALPHA_TO_COVERAGE);
// render
GL11.glDisable(GL13.GL_SAMPLE_ALPHA_TO_COVERAGE);

The result is as follows:



Then I use the function fwidth () in my fragment shader to average colors.
The fwidth () function returns the sum of the pixelââ,¬â,,¢s partial derivatives for a value, usually described as the equivalent of abs (ddx (value)) + abs (ddy (value)). Those two functions, ddx () and ddy (), are partial derivative functions.

Like this:

textureColour.a = (textureColour.a - 0.4) / max(fwidth(textureColour.a), 0.0001) + 0.5;

0.4 and 0.5 are taken arbitrarily.

The result is as follows:



thirdly, I add this line (before the last one) to manage the cases of disappearance of the intensity of the triangle when the mipmaping level is high (lowest resolutions of pictures)

like this:

float mipmapLevel = textureQueryLod(modelTexture, pass_textureCoords).x;
		
textureColour.a *= 1 + max(0, mipmapLevel) * 0.0;

so this :

if (textureColour.a<0.5)
{
	discard;
}

became :

float mipmapLevel = textureQueryLod(modelTexture, pass_textureCoords).x;
		
textureColour.a *= 1 + max(0, mipmapLevel) * 0.0;
		
textureColour.a = (textureColour.a - 0.4) / max(fwidth(textureColour.a), 0.0001) + 0.5;

in my fragment shader with alpha to coverage enabled.

Hope this post can be useful for others ...