ok, thx to matthiasmann on irc lwjgl, he narrowed the problem down a AWT Lock issue caused by switchApplet running on the EDT. Investigating a possible fix now using invokeLater.

I noticed a similar crash on MacOSX when you accept the cerfiticate temporarily. it would cause a crash on the second run in the same place (switch applet) when it checks the certificate of the downloaded jars it somehow conflicted with the first cerftificate check. I fixed it by creating a minimum delay between the first certficate accept prompt and the switch applet method.

edit:
I set the delay to the first time check plus 1000 milliseconds
Also the applet example doesn contain "<param name="separate_jvm" value="true">" this may help prevent at least the browser crashing (otherwise it kills firefox even when closing the browser window).

edit2:
after testing the applet using no cache, and running in a seperate vm, it seems that it really is a problem with multiple certificates, strange that no exception is thrown. I doubt it can be fixed short of making sure correct certificates are used.

after a few hours of fiddling with this it appears that it is indeed a JVM bug. It appears the culprit is a thread called SysExecutionThread which appears to be doing stuff outside the EDT and causing a deadlock. A bug has been submitted to Oracle and hopefully we'll see a fix for this soon. Will post a link here once I receive a number for the bug report.

On the bright side I think the dialog issue on mac's should be fixed now without adding a 1 second delay to startup.

So for now as mentioned above by bobjob just make sure you've got the right certificate and you'll be fine.