How to select polygons that fall into a frustum?

[gibson]

I made selective rays (6 planes were obtained). It is now necessary to color the triangles that are inside this truncated pyramid. How do you advise it? Tell me not the geometry, but how it is done programmatically in the shader. Given: the matrix P, the matrix of the view, the matrix the matrix of the model M, the coordinates of the vertices of the planes (their 8 is origin and the dir (used the function unproject)), the coordinates of the vertices of the model and its normals I got.

[KaiHH]

I am assuming that you extracted the six frustum planes from an existing view-projection matrix.
If that is the case and if your question was how to color all the fragments of the rendered model that fall inside of that frustum, then just do this:
1. transform the model-space vertices of the model by the model matrix and the camera's view-projection matrix as you would normally do
2. also transform the model-space vertices by the model matrix and the view-projection matrix describing your additional frustum
3. output the homogeneous vector from that 2. step as interpolated varying/out variable from your vertex shader to your fragment shader
4. in your fragment shader, color the fragment when their x, y and z coordinates all are within -w to w. This is basically what OpenGL does with your camera-transformed vertex positions written to gl_Position, just that it performs clipping based on that.

In the case where you really only have the six planes, simply only transform the model-space vertices only by the model matrix in step 2 and perform a point/plane distance test with all the six faces and output those six distances to the fragment shader as interpolated float values. Then, inside the fragment shader, test whether any of those distances is negative, in which case that fragment got effectively clipped by that plane.

[gibson]

Many thanks, like would understand!!! But is there a function in Lwigl that finds A B C D?

I am assuming that you extracted the six frustum planes from an existing view-projection matrix.
If that is the case and if your question was how to color all the fragments of the rendered model that fall inside of that frustum, then just do this:
1. transform the model-space vertices of the model by the model matrix and the camera's view-projection matrix as you would normally do
2. also transform the model-space vertices by the model matrix and the view-projection matrix describing your additional frustum
3. output the homogeneous vector from that 2. step as interpolated varying/out variable from your vertex shader to your fragment shader
4. in your fragment shader, color the fragment when their x, y and z coordinates all are within -w to w. This is basically what OpenGL does with your camera-transformed vertex positions written to gl_Position, just that it performs clipping based on that.

In the case where you really only have the six planes, simply only transform the model-space vertices only by the model matrix in step 2 and perform a point/plane distance test with all the six faces and output those six distances to the fragment shader as interpolated float values. Then, inside the fragment shader, test whether any of those distances is negative, in which case that fragment got effectively clipped by that plane.

[KaiHH]

But is there a function in Lwigl that finds A B C D?

Are you kidding? You just wrote that function in GLSL. I quote your code from the other thread:

float distancePointPlane(vec3 point,mat3 matr) {
        float v1Y0Y = matr[1][1] - matr[0][1];
        float v2Z0Z = matr[2][2] - matr[0][2];
        float v2Y0Y = matr[2][1] - matr[0][1];
        float v1Z0Z = matr[1][2] - matr[0][2];
        float v2X0X = matr[2][0] - matr[0][0];
        float v1X0X = matr[1][0] - matr[0][0];
        float a = v1Y0Y * v2Z0Z - v2Y0Y * v1Z0Z;
        float b = v1Z0Z * v2X0X - v2Z0Z * v1X0X;
        float c = v1X0X * v2Y0Y - v2X0X * v1Y0Y;
        float d = -(a * matr[0][0] + b * matr[0][1] + c * matr[0][2]);
        float s=0.0;
        return distancePointPlane(a, point[1], point[2], a, b, c, d);
    }

It should be very very easy to convert that to Java.

[gibson]

I do not want to duplicate code that is probably already written

But is there a function in Lwigl that finds A B C D?

Are you kidding? You just wrote that function in GLSL. I quote your code from the other thread:

float distancePointPlane(vec3 point,mat3 matr) {
        float v1Y0Y = matr[1][1] - matr[0][1];
        float v2Z0Z = matr[2][2] - matr[0][2];
        float v2Y0Y = matr[2][1] - matr[0][1];
        float v1Z0Z = matr[1][2] - matr[0][2];
        float v2X0X = matr[2][0] - matr[0][0];
        float v1X0X = matr[1][0] - matr[0][0];
        float a = v1Y0Y * v2Z0Z - v2Y0Y * v1Z0Z;
        float b = v1Z0Z * v2X0X - v2Z0Z * v1X0X;
        float c = v1X0X * v2Y0Y - v2X0X * v1Y0Y;
        float d = -(a * matr[0][0] + b * matr[0][1] + c * matr[0][2]);
        float s=0.0;
        return distancePointPlane(a, point[1], point[2], a, b, c, d);
    }

It should be very very easy to convert that to Java.

[gibson]

Problems in rendering.  A whole model stands out, not a pixel

#version 330

layout(location = 0) in vec3 vertexPos;
layout(location = 1) in vec3 normal;
layout (location = 2) in vec2 texCoord;

out float f;
out vec3 normal_modelspace;
out vec3 vertex_modelspace;
out vec2 TexCoord;
out vec4 vertexColor;
uniform mat4 P;
uniform mat4 V;
uniform mat4 M;

uniform vec4[6] planes;


void main() {
    TexCoord = texCoord;
    vertex_modelspace = (M * vec4(vertexPos.xyz, 1.0)).xyz;
    vertexColor = vec4(0.5f, 0.0f, 0.0f, 1.0f);
    gl_Position = P * V * vec4(vertex_modelspace.xyz, 1.0);
    normal_modelspace = (M * vec4(normal.xyz, 1.0)).xyz;
  	vec3 EyeDirection_cameraspace = vec3(0,0,0) - (V * M * vec4(vertexPos,1)).xyz;
      f=1.0;
  	  int i=0;
       while(i<6 && f==1.0){
        float denom = sqrt(planes[0].x * planes[0].x + planes[0].y * planes[0].y + planes[0].z * planes[0].z);
        if(float((planes[0].x * vertexPos.x + planes[0].y * vertexPos.y + planes[0].z * vertexPos.z + planes[0].w) / denom)>=0){
         f=0.0;
        }
        i++;
     }
}

#version 330 core

in vec3 normal_modelspace;
in vec3 vertex_modelspace;
in vec2 TexCoord;
in vec4 vertexColor;
in float f;
out vec4 color;

uniform vec3 light_worldspace;

uniform sampler2D ourTexture;

void main() {
  vec3 n = normalize(normal_modelspace);
  vec3 l = normalize(light_worldspace - vertex_modelspace);
  float cosTheta = clamp( dot( n, l), 0,1 );
  float ambient = 0.05;
    color = texture(ourTexture, TexCoord)*(cosTheta + ambient);
    if(f==0.0){
      color = vertexColor;
    }
}

But is there a function in Lwigl that finds A B C D?

Are you kidding? You just wrote that function in GLSL. I quote your code from the other thread:

float distancePointPlane(vec3 point,mat3 matr) {
        float v1Y0Y = matr[1][1] - matr[0][1];
        float v2Z0Z = matr[2][2] - matr[0][2];
        float v2Y0Y = matr[2][1] - matr[0][1];
        float v1Z0Z = matr[1][2] - matr[0][2];
        float v2X0X = matr[2][0] - matr[0][0];
        float v1X0X = matr[1][0] - matr[0][0];
        float a = v1Y0Y * v2Z0Z - v2Y0Y * v1Z0Z;
        float b = v1Z0Z * v2X0X - v2Z0Z * v1X0X;
        float c = v1X0X * v2Y0Y - v2X0X * v1Y0Y;
        float d = -(a * matr[0][0] + b * matr[0][1] + c * matr[0][2]);
        float s=0.0;
        return distancePointPlane(a, point[1], point[2], a, b, c, d);
    }

It should be very very easy to convert that to Java.

[KaiHH]

No, you still do not output the point/plane distance from the vertex shader to the fragment shader.
You have to output the point/plane distance and not some 1.0 or 0.0 value...
Why are you doing any f==1.0 and distance >= 0 comparison in the vertex shader??
It does not make any sense at all.
And you also do not use the world-space vertex position for that, but the model-space "vertexPos". I also told you in the initial answer that you have to transform by the model matrix...

Again: You have to output the computed distances from the WORLD-SPACE!! vertex position and use that interpolated distance values in the fragment shader...

One more time: You have to compute the distances and ONLY the distances between your world-space vertex position and the six planes and store that OF COURSE in six floats which you output to the fragment shader. Then do the >= 0.0 comparison in the fragment shader.

And yet another time: You have to compute the distances for each of the six planes and the world-space position (that is, the model-space input "vertexPos" transformed by the model matrix) and output those distances as "out" variables to the fragment shader. And inside of the fragment shader you do the distance >= 0.0 comparison for each of the six distances...

[gibson]

So?

#version 330

layout(location = 0) in vec3 vertexPos;
layout(location = 1) in vec3 normal;
layout (location = 2) in vec2 texCoord;

out float[6] dis;
out vec3 normal_modelspace;
out vec3 vertex_modelspace;
out vec2 TexCoord;
out vec4 vertexColor;
uniform mat4 P;
uniform mat4 V;
uniform mat4 M;

uniform vec4[6] planes;


void main() {
    TexCoord = texCoord;
    vertex_modelspace = (M * vec4(vertexPos.xyz, 1.0)).xyz;
    vertexColor = vec4(0.5f, 0.0f, 0.0f, 1.0f);
    gl_Position = P * V * vec4(vertex_modelspace.xyz, 1.0);
    normal_modelspace = (M * vec4(normal.xyz, 1.0)).xyz;
  	vec3 EyeDirection_cameraspace = vec3(0,0,0) - (V * M * vec4(vertexPos,1)).xyz;
     for(int i = 0;i<6;i++){
       float denom = sqrt(planes[0].x * planes[0].x + planes[0].y * planes[0].y + planes[0].z * planes[0].z);
       dis[i] = float((planes[0].x * vertex_modelspace.x + planes[0].y * vertex_modelspace.y + planes[0].z * vertex_modelspace.z + planes[0].w) / denom);
     }
}

#version 330 core

in vec3 normal_modelspace;
in vec3 vertex_modelspace;
in vec2 TexCoord;
in vec4 vertexColor;
in float[6] dis;
out vec4 color;

uniform vec3 light_worldspace;

uniform sampler2D ourTexture;

void main() {
  vec3 n = normalize(normal_modelspace);
  vec3 l = normalize(light_worldspace - vertex_modelspace);
  float cosTheta = clamp( dot( n, l), 0,1 );
  float ambient = 0.05;
    int i=0;
    while(i<6 && dis[i]>=0){
      i++;
    }
    if(i==6){
      color = texture(ourTexture, TexCoord)*(cosTheta + ambient);
    }
    else 
      color = vertexColor;
}

No, you still do not output the point/plane distance from the vertex shader to the fragment shader.
You have to output the point/plane distance and not some 1.0 or 0.0 value...
Why are you doing any f==1.0 and distance >= 0 comparison in the vertex shader??
It does not make any sense at all.
And you also do not use the world-space vertex position for that, but the model-space "vertexPos". I also told you in the initial answer that you have to transform by the model matrix...

Again: You have to output the computed distances from the WORLD-SPACE!! vertex position and use that interpolated distance values in the fragment shader...

One more time: You have to compute the distances and ONLY the distances between your world-space vertex position and the six planes and store that OF COURSE in six floats which you output to the fragment shader. Then do the >= 0.0 comparison in the fragment shader.

And yet another time: You have to compute the distances for each of the six planes and the world-space position (that is, the model-space input "vertexPos" transformed by the model matrix) and output those distances as "out" variables to the fragment shader. And inside of the fragment shader you do the distance >= 0.0 comparison for each of the six distances...