Ray intersection with a triangle using projection matrix

[Joshi Elijah]

Hello,

            I am working on ray triangle intersection from couple of days, Initially i took just transformationMatrix found out unprojectRay using joml unprojectRay method and using Intersectionf.intersectRayTriangle(origin, direction, vertices) i am able to detect the hit. Later i have introduced projectionMatrix with fov = 70 nearPlane = 0.1f farPlane = 1000f
Here is the projection matrix

1.428148 0.0 0.0 0.0
0.0 1.7236269 0.0 0.0
0.0 0.0 -1.0001999 -0.20002
0.0 0.0 -1.0 0.0

when i see the direction vector from unprojectray method everything is at infinity, and there is no hit for simple translation (0,0,-5f). Am struggling to get this work,Any help ?

[KaiHH]

I provided a self-contained (JUnit 3) test for you to verify that it's working and for you to check what you probably did wrong since you did not provide any information on how you create your "transformationMatrix", how you combine/multiply your "transformationMatrix" with your "projectionMatrix" and how you use/call unprojectRay and intersectRayTriangle:

import static org.joml.Intersectionf.*;
import java.lang.Math;
import org.joml.*;
import junit.framework.TestCase;
public class UnprojectAndIntersectTest extends TestCase {
  private static void assertComponentWiseEquals(Vector3f actual, Vector3f expected, float delta) {
    assertEquals(expected.x, actual.x, delta);
    assertEquals(expected.y, actual.y, delta);
    assertEquals(expected.z, actual.z, delta);
  }
  public void test() {
    // given
    int w = 800;
    int h = 800;
    Matrix4f m = new Matrix4f()
        .perspective(
            (float) Math.toRadians(70.0),
            (float) w / h,
            0.1f,
            1000.0f)
        .translate(0, 0, -5);
    Vector3f origin = new Vector3f();
    Vector3f dir = new Vector3f();

    // when
    m.unprojectRay(w / 2, h / 2, new int[] { 0, 0, w, h }, origin, dir);
    dir.normalize();
    float t = intersectRayTriangle(
        origin, 
        dir,
        new Vector3f(-1, -1, 0), 
        new Vector3f(1, -1, 0),
        new Vector3f(0, 1, 0), 
        1E-6f);
    Vector3f hit = new Vector3f(dir).mul(t).add(origin);

    // then
    assertEquals(4.9f, t, 1E-6f); // <- ray starts at near plane
    assertComponentWiseEquals(origin, new Vector3f(0, 0, 4.9f), 1E-6f);
    assertComponentWiseEquals(dir, new Vector3f(0, 0, -1), 1E-6f);
    assertComponentWiseEquals(hit, new Vector3f(0, 0, 0), 1E-6f);
  }
}

[Joshi Elijah]

This is how am creating transformationMatrix and the respective results, followed by projection matrix and the results

public static Matrix4f createTransformationMatrix(Vector3f translation, float rx, float ry,
			float rz, float scale) {
		Matrix4f matrix = new Matrix4f();
		matrix.setIdentity();
		Matrix4f.translate(translation, matrix, matrix);
		Matrix4f.rotate((float) Math.toRadians(rx), new Vector3f(1,0,0), matrix, matrix);
		Matrix4f.rotate((float) Math.toRadians(ry), new Vector3f(0,1,0), matrix, matrix);
		Matrix4f.rotate((float) Math.toRadians(rz), new Vector3f(0,0,1), matrix, matrix);
		Matrix4f.scale(new Vector3f(scale,scale,scale), matrix, matrix);
		return matrix;
	}

1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.0 0.0 1.0 -5.0
0.0 0.0 0.0 1.0

//ProjectionMatrix

public void createProjectionMatrix() {
		float aspectRatio = (float) Display.getWidth() / (float) Display.getHeight();
		float y_scale = (float) ((1f / Math.tan(Math.toRadians(FOV / 2f))) * aspectRatio);
		float x_scale = y_scale / aspectRatio;
		float frustum_length = FAR_PLANE - NEAR_PLANE;

		projectionMatrix = new Matrix4f();
		projectionMatrix.m00 = x_scale;
		projectionMatrix.m11 = y_scale;
		projectionMatrix.m22 = -((FAR_PLANE + NEAR_PLANE) / frustum_length);
		projectionMatrix.m23 = -1;
		projectionMatrix.m32 = -((2 * NEAR_PLANE * FAR_PLANE) / frustum_length);
		projectionMatrix.m33 = 0;
	}

1.428148 0.0 0.0 0.0
0.0 1.7236269 0.0 0.0
0.0 0.0 -1.0001999 -0.20002
0.0 0.0 -1.0 0.0

am multiplying ProjectionMatrix * transformationMatrix
Matrix4f mvp = ProjectionMatrix * transformationMatrix; // am supplying this as column major to joml mvp matrix.
Later am using this method to calculate the hit.

// Resultant MVP matrix that am supplying
1.4281 ,0, 0, 0
0, 1.723, 0, 0
0, 0, -1.00, -1
0, 0, 4.8005, 5

// int[] viewport = {0, 0, 700, 580};
// float[] vertices = {
//				-0.5f, 0.5f, 0,
//				-0.5f, -0.5f, 0,
//				0.5f, -0.5f, 0
//		};
public static void selectTriangle(Matrix4f mvp, int[] viewport, float[] vertices)
{
        float mouseX = Mouse.getX();
        float mouseY = Mouse.getY();

        org.joml.Vector3f origin = new org.joml.Vector3f();
        org.joml.Vector3f dir = new org.joml.Vector3f();
        
	mvp.unprojectRay(mouseX, mouseY,viewport,origin,dir);
		
        float minDist = Float.POSITIVE_INFINITY;
        
        float dist = Intersectionf.intersectRayTriangle
                (origin.x, origin.y,origin.z,
                dir.x,dir.y,dir.z, -0.5f, 0.5f, 0f,-0.5f, -0.5f, 0f,0.5f, -0.5f, 0f, 1E-6f);
    	//System.out.println(origin+" "+dir+" "+ dist);
        if (dist > 0.0f && dist < minDist) {
            System.out.println("Hit");
        }
    }

if am missing anything else please let me know happy to provide, Thanks

[KaiHH]

Your combined MVP matrix is completely wrong (it inverts the z axis and the far plane is at positive infinity).
When you multiply:

1.428148 0 0 0
0 1.7236269 0 0
0 0 -1.0001999 -0.20002
0 0 -1 0

with:

1 0 0 0
0 1 0 0
0 0 1 5
0 0 0 1

you will get:

1.428148  0  0  0
0  1.7236269  0  0
0 0 -1.0001999 -5.2010193
0 0 -1 -5

which works perfectly fine with unprojectRay() and intersectRayTriangle() and is not what you showed below your comment "// Resultant MVP matrix that am supplying".
So you did not arrive at your resulting MVP matrix by just multiplying the "projectionMatrix" with the "transformationMatrix".
I also recommend you DO NOT USE both the lwjgl_util.jar vector/matrix classes as well as JOML. Choose just one.
If you need help with migrating from the lwjgl_util.jar classes to JOML, see: https://github.com/JOML-CI/JOML/wiki/Migrating-from-LWJGL-2

Apart from that, I'll make unprojectRay() robust enough to handle far planes at infinity.

[Joshi Elijah]

yes your right i will stick to one library. Thanks kaiHH, it helped.