hello ,every one
Does JOML have the function that calculate the intersect point of a ray from camera to xy plane. I want to write a button class and test whether user click on it.
Thank you
Sure, please see org.joml.Intersectionf#intersectRayPlane(...) (https://joml-ci.github.io/JOML/apidocs/org/joml/Intersectionf.html#intersectRayPlane(org.joml.Rayf,org.joml.Planef,float))
This would be your method if you already have a ray and you know the plane equation coefficients (a, b, c and d) in `a*x + b*y + c*z + d = 0`. There is also a method to intersect a ray using the three-point form of the plane.
Btw.: to get the ray origin and direction of the mouse cursor from a given view-projection matrix, you can use org.joml.Matrix4f#unprojectRay(...) (https://joml-ci.github.io/JOML/apidocs/org/joml/Matrix4f.html#unprojectRay(float,float,int%5B%5D,org.joml.Vector3f,org.joml.Vector3f)) with the window coordinates of the mouse.
Take care to invert the y coordinate via (windowHeight - mouseY), where `this` Matrix4f is the view-projection matrix transforming a vector in world space (or rather the space you defined the plane in) into clip space.
EDIT: The above method is the most generic solution and assumes that your UI/button plane is any plane (such as the XY plane) in the world and your camera is free to move around in the world. If however your UI/button is always aligned with the camera (as is typical) and you render the UI/button with a simple orthographic projection, you can directly use the mouse coordinates and do not need any ray/plane intersection. If you have a linear mapping between mouse coordinates and UI coordinates that is not directly the pixels (such as with a ortho(0, windowWidth, 0, windowHeight) projection, then you can use projectionMatrix.unproject().
Thank you for detailed explain. So now I know I don't need ray trace calculation anymore . because I use ortho project whose bottom and right are exactly the same as the canvas height and width.