I think I finally figured out why running my OpenCL app with DISPLAY unset gives the ELFCLASS32 error somewhere around the loading of awt: lwjgl was never designed to run with a headless jre or with -Djava.awt.headless=true.
Before OpenCL, this would have never been tested because who runs OpenGL headless, but now that apps like mine legitimately exist, this bug shows up.