I think but not entirely sure that the Applet SecurityManager does not allow you to change System Properties (System.setProperty) unless its explicitly one of the exceptions. That could explain the current behaviour.
A possible solution (other than signing your applet) would be to have a publically exposed API in LWJGL e.g. Mouse.enableNegativeCoords(boolean). Technically its not just unlocking negative coords but any coords outside the Display, so could also be something like Mouse.clipCoords(boolean), Display.clipMouseCoords(boolean), etc.