Hi i am working on my own game engine using LWJGL 3 and OpenGL 4.6. But currently I am stumped by a graphical glitch which happens when I render transparent grass sprites, which are inside the fog.
It seems the fog doesn't get properly mixed in my shader.

The screenshot:


Here is the last part of my fragment shader.
//colorTotal holds the combined colors from all light sources
//I have tried forcing the w component to the texture's alpha
colorTotal = vec4(colorTotal.x, colorTotal.y, colorTotal.z, texture(texture_sampler, texcoord).w);
colorTotal += texture(texture_sampler, texcoord) * vec4(MaterialAmbientColor, colorTotal.w);

colorTotal = vec4(clamp(colorTotal.x, 0, 1), clamp(colorTotal.y, 0, 1), clamp(colorTotal.z, 0, 1), clamp(colorTotal.w, 0 ,1));

float fogAmount = smoothstep(60.0f, 120.0f, v_fogDepth);
vec4 u_fogColor = vec4(0.8, 0.8, 1, 1);
outColour = mix(colorTotal, u_fogColor, fogAmount);

And these are the settings I use for blending
GL11.glEnable(GL11.GL_BLEND);
GL11.glBlendFunc(GL11.GL_SRC_ALPHA, GL11.GL_ONE_MINUS_SRC_ALPHA);

All objects in my scene are sorted by distance. Opaque from front to back, and transparent sprites from back to front.

The problem essentially is that you blend/mix your (RGB, A) of the texture with (RGB, 1) of your fog color. This means that when the fogAmount is large (assume 1), then your grass quad essentially has alpha=1, so is fully opaque and thus you see the fog color that would only have been valid for opaque sections of the texture.
So, you must only blend the RGB channels between texture and fogColor, NOT the alpha channel.

Once you fix that, you'll bump into the next issue that your fog on rendered geometry is inconsistent with (missing) fog on the background. So, you will see grey fogged grass in front of perfectly clear background sky. This is of course physically implausible, since the light coming from the background (around the grass blades) is affected by the same fog as the light coming from the grass blades themselves (in fact, the background/sky light is affected by even more fog, since it travels a further distance than the light coming from the grass blades). So, given your current fog model, your background/sky should just be gray.

Thank you for the explanation. I have changed the fog color so it uses the alpha channel from the texture and now it works.

So, you must only blend the RGB channels between texture and fogColor, NOT the alpha channel.

//colorTotal holds the combined colors from all light sources
//I have tried forcing the w component to the texture's alpha
colorTotal = vec4(colorTotal.x, colorTotal.y, colorTotal.z, texture(texture_sampler, texcoord).w);
colorTotal += texture(texture_sampler, texcoord) * vec4(MaterialAmbientColor, colorTotal.w);

colorTotal = vec4(clamp(colorTotal.x, 0, 1), clamp(colorTotal.y, 0, 1), clamp(colorTotal.z, 0, 1), clamp(colorTotal.w, 0 ,1));

float fogAmount = smoothstep(60.0f, 120.0f, v_fogDepth);
vec4 u_fogColor = vec4(0.8, 0.8, 1, texture(texture_sampler, texcoord).w);
outColour = mix(colorTotal, u_fogColor, fogAmount);