Keyboard input consistency/speed

[XeaLouS]

Hi, just a few questions regarding the speed of keyboard inputs in Windows.

Reading the changelog, LWJGL used to use directInput, and now uses windows messaging - this was so that it could handle things like foreign languages/imes,
as well as directinput being deprecated.

I am attempting to create an extremely input sensitive game. There are a few hurdles I need to overcome:

High accuracy "wait" timer. I have a delayed auto-repeat feature that needs to be within 1-2ms accuracy. Thankfully with some testing i did this works perfectly in LWGJL

Consistent/well-timed input. So far my testing hasn't been promising. I have done two tests:
The first test i have done is simple - in LWGJL 3.0.0a I do a glfwWaitEvent, record the frame number, mash 8 keys simultaneously and see how many frames passed. I have no graphics (that is, i am not rendering anything) so it could theoretically run at 100000000000fps. The "correct" result would be to see all 8 keys on different frame numbers - right? I consistently get all 8 keys on the same frame. This means one or some of the following:

• My keyboard doesn't send fast enough (e.g. it sends at 125hz and i'm mashing all 8 keys in a 8ms window). I tested with two keyboards. In USB mode, all 8 keys appear on the same frame. with a PS/2 keyboard, they appear on 8 separate frames.
• Windows OS doesn't pick up the keys fast enough (e.g. i send the keys 1ms apart but windows processes them in 8ms increments)
• Windows Messaging (OS -> Window) doesnt pick up keys fast enough
• LWJGL doesn't pick up the keys fast enough

Test 2: strum 8 keys:
The test here is to try and see the minimum time values between two keys - hopefully by strumming 8 keys you would occasionally get two keys that are 1-2ms away from each other.
Unfortunately for this test, i would CONSISTENTLY get keys that are 15-16ms away from each other - suggesting that somewhere it is only updating at 60hz.

Would there be a speed difference between using DirectInput vs WindowsMessaging for keyboardEvents?

[Kai]

Interestingly, what DirectInput did, was just sitting on top of windows messaging by receiving only the "raw input" and not the interpreted mouse and keyboard events.
For mouse inputs this allowed to ignore the mouse acceleration/ballistics and enable high-dpi mouse inputs.
And then most games provided a ballistics settings themselves.
That's also the reason why it was deprecated, because it did not really provide any real advantage over windows messaging or raw input.
Now, the fastest and the most efficient way possible (with Windows) is for you to also make use of Raw Input.
There is no other solution (for Windows) that I know of.
LWJGL 3 also has some decent native platform API support for Windows which you make use of.

[spasi]

There's nothing in LWJGL or GLFW that batches or delays event delivery, it is entirely up to the hardware and/or OS. On Windows 8.1 with a Vengeance K95 (USB) keyboard and doing "Test 2" as described above, I can get (between key events):

- 7.3 milliseconds with a single key. This is the speed at which I can physically spam a key (I think 5ms is the limit for Cherry MX switches).
- 9 microseconds with multiple keys.

[XeaLouS]

- 9 microseconds with multiple keys.

That is indeed very promising!

Could you find out if those multiple keys are processed on the same glfwWaitEvent() call though?
i.e.:

while ( glfwWindowShouldClose(window) == GL_FALSE ) {
    glfwSwapBuffers(window); // swap the color buffers

    glfwWaitEvents();
    frameCounter++;
}

and

private void invoke(long window, int key, int scancode, int action, int mods) {       
        if (action == GLFW_PRESS) {                
            System.out.println(frameCounter); //if this is the same for those "9 microsecond" differences then they are actually 0 microseconds apart...
            System.out.println(glfwGetTime());
        }

I just want to make sure that 9 microseconds isn't actually because Windows/LWJGL received them all on the same glfwWaitEvents, so technically the difference is actually 0 microseconds and calling glfwGetTime() while iterating through them produces the microseconds.

[spasi]

Indeed, 9 μs was multiple events in the same glfwWaitEvents call. But let me repeat that there isn't any buffering involved in LWJGL or GLFW; glfwWaitEvents on Windows translates to:

// glfwWaitEvents
WaitMessage
glfwPollEvents

// glfwPollEvents
while ( PeekMessage ) {
	TranslateMessage
	DispatchMessage // This eventually invokes GLFWKeyCallback in Java
}

By the time the first DispatchMessage returns, there's another event ready to be processed, and so on. Any buffering is happening at the OS level.

With vsync disabled and testing for the "frame counter", the minimum difference I see between glfwWaitEvents is 161 μs, which still is too low to have any practical impact.

[XeaLouS]

With vsync disabled and testing for the "frame counter", the minimum difference I see between glfwWaitEvents is 161 μs, which still is too low to have any practical impact.

Thanks! Yes i realise the buffering is happening at OS Level.

in the 161 Ã,µs example, does that mean there were two glwfWaitEvent calls within 161Ã,µs, both with actual keyboard events? (if so, this is most excellent... i must learn why it's happening on your machine and not mine)
i.e.

time = 0, glwfWaitEvents occured = 1, keyboardEvents occured = 1
time = 161Ã,µs, glwfWaitEvents occured = 2, keyboardEvents occured = 2

In my testing the minimum time between two glwfWaitEvent calls that had actual keyboard inputs was on the order of 8+ms implying that the OS was buffering inputs and only releasing them every 8ms. This DOES have a practical impact in my case.
i.e.

time = 0, glwfWaitEvents occurred = 1, keyboardEvents occured = 1
time = 4ms+, glwfWaitEvents occurred = 2, keyboardEvents occured = 2

[spasi]

in the 161 Ã,µs example, does that mean there were two glwfWaitEvent calls within 161Ã,µs, both with actual keyboard events?

Yes.

Could you please post the code you're using to test?

[XeaLouS]

package eve;

import org.lwjgl.Sys;
import org.lwjgl.glfw.*;
import org.lwjgl.opengl.*;
 
import java.nio.ByteBuffer;
 
import static org.lwjgl.glfw.Callbacks.*;
import static org.lwjgl.glfw.GLFW.*;
import static org.lwjgl.opengl.GL11.*;
import static org.lwjgl.system.MemoryUtil.*;

public class Eve {
 
    // We need to strongly reference callback instances.
    private GLFWErrorCallback errorCallback;
    private GLFWKeyCallback   keyCallback;    
    // The window handle
    private long window;
    private boolean canBlit = true;        
        
    
    public void run() {        
        System.out.println("Hello LWJGL " + Sys.getVersion() + "!");
 
        try {
            init();
            loop();
 
            // Release window and window callbacks
            glfwDestroyWindow(window);
            keyCallback.release();
        } finally {
            // Terminate GLFW and release the GLFWerrorfun
            glfwTerminate();
            errorCallback.release();
        }
    }
       
    private int frameCounter = 0;

    private void invoke(long window, int key, int scancode, int action, int mods) {        
        this.canBlit = true;        
        double time = glfwGetTime();
        if (action == GLFW_PRESS || action == GLFW_RELEASE) {                
             System.out.println(Integer.toString(this.frameCounter) + " " 
                                + Double.toString(time));
        } 

        
    }
    
    
    private void init() {        
        // Setup an error callback. The default implementation
        // will print the error message in System.err.
        glfwSetErrorCallback(errorCallback = errorCallbackPrint(System.err));
 
        // Initialize GLFW. Most GLFW functions will not work before doing this.
        if ( glfwInit() != GL11.GL_TRUE )
            throw new IllegalStateException("Unable to initialize GLFW");
 
        // Configure our window
        glfwDefaultWindowHints(); // optional, the current window hints are already the default
        glfwWindowHint(GLFW_VISIBLE, GL_TRUE); // the window will stay hidden after creation
        glfwWindowHint(GLFW_RESIZABLE, GL_FALSE); // the window will be resizable
 
        int WIDTH = 640;
        int HEIGHT = 480;
 
        // Create the window
        window = glfwCreateWindow(WIDTH, HEIGHT, "Hello World!", NULL, NULL);
        if ( window == NULL )
            throw new RuntimeException("Failed to create the GLFW window");
 
        // Setup a key callback. It will be called every time a key is pressed, repeated or released.
        
        keyCallback = GLFWKeyCallback (this::invoke);        
        glfwSetKeyCallback(window, keyCallback);        
        
        // Get the resolution of the primary monitor
        ByteBuffer vidmode = glfwGetVideoMode(glfwGetPrimaryMonitor());
        // Center our window
        glfwSetWindowPos(
            window,
            (GLFWvidmode.width(vidmode) - WIDTH) / 2,
            (GLFWvidmode.height(vidmode) - HEIGHT) / 2
        );
 
        // Make the OpenGL context current
        glfwMakeContextCurrent(window);
        // Enable v-sync
        glfwSwapInterval(0);
 
        // Make the window visible
        glfwShowWindow(window);
        
    }
 
    private void loop() {
        // This line is critical for LWJGL's interoperation with GLFW's
        // OpenGL context, or any context that is managed externally.
        // LWJGL detects the context that is current in the current thread,
        // creates the ContextCapabilities instance and makes the OpenGL
        // bindings available for use.
        GLContext.createFromCurrent();
 
        // Set the clear color
        glClearColor(1.0f, 0.0f, 0.0f, 0.0f);
 
        // Run the rendering loop until the user has attempted to close
        // the window or has pressed the ESCAPE key.
        while ( glfwWindowShouldClose(window) == GL_FALSE ) {
            //if (canBlit) {
            //    canBlit = false;
            //    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); // clear the framebuffer            
            //    glfwSwapBuffers(window); // swap the color buffers
            //}
            // Poll for window events. The key callback above will only be
            // invoked during this call.            
            glfwWaitEvents();
            frameCounter++;
        }
    }    

    public static void main(String[] args) {
        new Eve().run();
    }
 
}

and the output (here's an example of just strumming some keys. the differences are on the order of 11-18ms between frames.

24 1.7608934284545137
25 1.7755140140643044
26 1.7986285191705875
27 1.8458040812403615
27 1.845893493985027
28 1.8616863476641283
29 1.8777311297606096
30 1.9165336397248882
31 3.920571193077648
32 3.936549862167487
33 3.9563726967844906
33 3.956459197061903
34 3.9726941632700017
34 3.9728007595714594
35 4.004540245200618
35 4.0046721799671765
35 4.004735380506565
35 4.004791299877822
36 4.035453755116841
36 4.035540255394253
37 4.045537590486717
38 4.062469510108445
38 4.062548146724275
39 4.075545323087273
40 8.293066143587549
41 8.303020082917943
42 8.330322715933963
42 8.33041008995155
43 8.342230629544359
44 8.353185875116317
45 8.380053968018197
45 8.38013726458163
45 8.380195222679966
45 8.380252015791397
46 8.4103305213462
46 8.410454009957725
46 8.410530025353028
46 8.410584197243931
47 8.428188023060915
48 8.44018389318235

and more output for when i just mash 8 keys simultaneously:

103 6.9825298564299265
103 6.982800133391
103 6.982901778498128
103 6.982958280362835
103 6.983013908487367
103 6.983070410352074
103 6.983124873489704
104 6.9930700754183395
105 7.060313701847814
105 7.0605030122192565
105 7.060707467420413
105 7.060769211726176
105 7.060823092370355
105 7.060876099274359
105 7.060929106178362
105 7.060982404329091

everything is on the order of 10+ms between glwfWaitEvent calls - hardly the Ã,µs you got

[spasi]

Try this:

double time;
double min = Double.MAX_VALUE;

long lastFrame = -1;

private void invoke(long window, int key, int scancode, int action, int mods) {
	if ( action == GLFW_REPEAT )
		return;

	if ( lastFrame != frameCounter ) { // Ignore multiple events on the same frame
		lastFrame = frameCounter;

		double t = glfwGetTime();
		double diff = t - time; // Diff since last glfwWaitEvents
		time = t;

		if ( diff < min ) { // Print smallest diff seen so far
			min = diff;
			System.err.println(Math.round(diff * 1000.0 * 1000.0) + "us [" + frameCounter + "]");
		}
	}
}

Sample output (mashing multiple keys):

[Demo] Hello LWJGL 3.0.0a!
     [Demo] 805511us [3]
     [Demo] 11953us [4]
     [Demo] 11709us [5]
     [Demo] 3902us [6]
     [Demo] 3857us [25]
     [Demo] 3815us [29]
     [Demo] 3755us [32]
     [Demo] 3398us [57]
     [Demo] 180us [65]
     [Demo] 146us [72]
     [Demo] 82us [76]

[XeaLouS]

Yep, i'm able to reproduce your output.

So it's definitely possible to get sub-millisecond differences in inputs *sometimes* but it seems quite rare in my case.
Notice the weird jump from 9ms straight down to 0.1ms:

1372804us [30]
73922us [31]
53948us [488]
22910us [489]
16698us [493]
11762us [498]
9893us [513]
8906us [541]
219us [809]
193us [810]

It's probably due to having different hardware/OS.

You seem to be able to easily get 4ms~ whereas mine is 9ms~ and i have to spam REALLLLLLY hard to get those weird <1ms outliers.

Thanks!!

[Kai]

Hey. Sorry, I am rather curious: What kind of keyboard-destroying application are you developing that requires sub-millisecond accuracy for keyboard input handling, where people need to spam "REALLLLLLY hard" to make use of that?

[XeaLouS]

Hey. Sorry, I am rather curious: What kind of keyboard-destroying application are you developing that requires sub-millisecond accuracy for keyboard input handling, where people need to spam "REALLLLLLY hard" to make use of that?

The app isn't keyboard destroying at all - it only needs to handle about 12-15 keyboard presses per second. the sub-millisecond accuracy is the important part. There is a function in the game known as Delayed auto shift. Let delay = x milliseconds:

• On key down, action (A) happens
• If key is held down for at least x milliseconds, action (B) happens

Basically, the player wants either (A) to happen, or both (A) and (B) to happen.

Now this is where the keyboard accuracy is CRITICAL.

If the keyboard is accurate to 0ms (i.e. completely accurate...), and the Delay is x milliseconds then we know that:

• The player must press the button for between 0 and x milliseconds (non-inclusive) for the game to detect (A)
• The player must press the button for >= x milliseconds for the game to detect (A) + (B)

But if the keyboard is accurate to y ms, and the delay is x milliseconds, then we know that:

• The player must press the button for between y ms and x - y milliseconds (non inclusive) for the game to guarantee execute (A) but not (B)
  The reason for this is that the input could be detected up to y ms late, so you have to press it y ms early, and you have to release y ms early because if the release is detected y ms late,the game will execute B
  
• The player must press the button for > x + y to guarantee that (A) + (B) occurs.
  

Ok, so now that i've explained the gist of it, let me reveal x and y.

x is user configurable - in current implementations it is a multiple of 16ms because the game runs at 60fps. However, we have argued that the user doesn't need to actually see the result of (A) + (B) so realistically x can be a multiple of 1ms. Currently for most people, X is on the order of 60-80ms.

y is currently on the order of 10ms.

That means, with x = 67 (4 frames) and y = 10ms:

The user must press for at least 10ms (easy) and less than 57 ms (not easy) for (A)
The user must press for at least 77 (easy) to activate (A) + (B)

Notice that the error is 15% for (A) and 15% for (B)

As x (the delay) decreases, y will stay the same, so the error will increase. Also note that due to this error, there is a delay floor.

Let's say that the minimum time a player can accurately press a key as a tap (vs holding it) is 50ms. The minimum "controllable" value of x would be a  60ms if the accuracy is 10ms. If the accuracy was 0ms instead of 10ms, then we could set x to 50ms instead of 60ms.

Now you're probably wondering why it needs to be so accurate:
Here's a video. x = 4 frames (64~ms), and y = 16ms (old engine, doesn't properly time inputs). Note that a player capable of playing at x = 64, y = 16 would also be able to play at x = 48, y = 0!!!! This would probably lead to a huge increase in performance!!!

https://www.youtube.com/watch?v=v3TJXSFnVS4

So, what are (A) and (B)? (A) is tapping the left or right arrow. (B) is holding the left or right arrow. This activates DAS - which instantly shifts a piece to the wall.

[Kai]

I see. That tetris video looked amazing. This was not fast-forward, or was it?
Haven't played tetris at all, so dunno whether this is actually possible.

[XeaLouS]

I see. That tetris video looked amazing. This was not fast-forward, or was it?
Haven't played tetris at all, so dunno whether this is actually possible.

https://www.youtube.com/watch?v=4OFq31lek2g

it's real time