After trying Fraps and similar programs to record in-game videos I'm kind of disappointed and I'd like to implement my own recording capability directly in the game. Images are simple, I can just take a screenshot each frame; the game code will cope with the incurred delays.

But I would also like to record the sound. Is there a way to create a custom OpenAL device that could save the resulting sound mix as a WAV file? Synchronization with the images is not a problem, I can add a beep/flash to the start and sync it later in a video editing app. I just need a continuous sound output from an app that will definitely not run in real-time.

Any ideas?

Maybe you can grab the sound using JavaSound? I'm not sure this works as I never tried it.
AFAIK, there's no wav file writing capabilities in JavaSound, but I've written a little class that does it, if you need it (but it's also not difficult to write yourself)...