Stencil Test - Acting a little weird.

[quew8]

So I'm trying to overlay a 2D box on top of the rest of my geometry. To save on rendering times, I want to draw the overlay first, drawing into the stencil as I do. Then when I draw the rest of it, I use the stencil test to see if the overlay has already been drawn there.

First of all I tried this:

void init() {
    glEnable(GL_STENCIL_TEST);
    glClearStencil(0);
    glStencilFunc(GL_LESS, 1, 0xFFFFFFFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}

void render() {
    glClear(GL_STENCIL_BUFFER_BIT);
    

    glPushAttrib(GL_STENCIL_BUFFER_BIT);

    glStencilFunc(GL_ALWAYS, 1, 0xFFFFFFFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    drawOverlay();

    glPopAttrib();


    drawRestOfScene();
}

NB This is just the important stuff pertaining to the question.

However, if I do this then only the overlay gets drawn and nothing else. To get it to work as expected I must change the regular stencil function to GL_GREATER like so:

void init() {
    glEnable(GL_STENCIL_TEST);
    glClearStencil(0);
    glStencilFunc(GL_GREATER, 1, 0xFFFFFFFF); //This line is the only change
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}

void render() {
    glClear(GL_STENCIL_BUFFER_BIT);
    

    glPushAttrib(GL_STENCIL_BUFFER_BIT);

    glStencilFunc(GL_ALWAYS, 1, 0xFFFFFFFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    drawOverlay();

    glPopAttrib();


    drawRestOfScene();
}
   

And that works as I would expect the first snippet to.

Whilst it is working, I would very much like to know why.

Thank you.

[broumbroum]

you have to carefully read the glstencilfunc operating functions. ref & mask is compared to stencil & mask . that's it, ref & mask equal to 1, so that it can never be less than < stencil & mask, which equals to "at most" 1, GL_LESS will always fail because of your ref equals to 1. (and not a hex value like the mask you chose)

edit : just put "1" in place of 0xF. for convenience.

[quew8]

Sorry, I don't really understand what you're saying.

To begin with stencil is cleared to 0, the reference value is 1 and the mask is 0xFFFFFFFF (Or 1 you're quite right). So:

    (0 & 1) < (1 & 1)
    0 < 1
    true

Then when the overlay has been drawn in that pixel, the stencil value becomes 1, so:

    (1 & 1) < (1 & 1)
    1 < 1
    false

Where am I going wrong here?

[quew8]

Scrap this, I'm officially an idiot. I just reread the glStencilFunc docs as you suggested.

Turns out the reference value gets compared to the stencil buffer value whereas I assumed (because I'm an idiot) that stencil buffer value got compared to the reference value.

Ie. Its (ref & mask) < (stencil & mask)
not as I assumed (stencil & mask) < (ref & mask)

Somehow that seemed more intuitive to me.

Thank you for your answer.

[broumbroum]

oh yeah..
About the hex values, which I haven't tried anyway, they ought to be used when letting multiple stencils overlap, so that a stencil value like 0x00F00F will return something greater than ref 0x00000F & mask 0xFFFFFF... Thereby GL_LESS would make sense.. '

[quew8]

Ah, I see what you're getting at.

Whilst I know what all the bitwise operations do, and how they're often used and stuff, they still confuse the hell out of me.