Sure, please see

org.joml.Intersectionf#intersectRayPlane(...)This would be your method if you already have a ray and you know the plane equation coefficients (a, b, c and d) in `a*x + b*y + c*z + d = 0`. There is also a method to intersect a ray using the three-point form of the plane.

Btw.: to get the ray origin and direction of the mouse cursor from a given view-projection matrix, you can use

org.joml.Matrix4f#unprojectRay(...) with the window coordinates of the mouse.

Take care to invert the y coordinate via (windowHeight - mouseY), where `this` Matrix4f is the view-projection matrix transforming a vector in world space (or rather the space you defined the plane in) into clip space.

EDIT: The above method is the most generic solution and assumes that your UI/button plane is any plane (such as the XY plane) in the world and your camera is free to move around in the world. If however your UI/button is always aligned with the camera (as is typical) and you render the UI/button with a simple orthographic projection, you can directly use the mouse coordinates and do not need any ray/plane intersection. If you have a linear mapping between mouse coordinates and UI coordinates that is not directly the pixels (such as with a ortho(0, windowWidth, 0, windowHeight) projection, then you can use projectionMatrix.unproject().