Mouse problem in 2.2.1

[Elviz]

LWJGL generally works well. However, I'm having a problem with 2.2.1 that I didn't have with 2.1.0.

Setup is the following:

• Windows XP
• Sun's JRE 6 Update 17
• Standalone LWJGL application (no applet)
• No AWT use from my side
• Windowed mode
• Using the event-based mouse API
• Didn't modify allowNegativeMouseCoords property

The issue now is that LWJGL sometimes gets into a state where it starts reporting mouse move events when the cursor is over the trim area of the window (i.e., title bar and border). In this state, clicking on the trim area or even outside the application (e.g., on the taskbar) will result in a mouse click event as if it had originated within the client area of my LWJGL window, with no way to tell that it didn't. In addition, a click on the window's close (or minimize) button seems to be intercepted somewhere as it does not cause the button to be pressed down.

It is not entirely clear what triggers the aforementioned behaviour. The best clue so far is that it seems related to mouse events accumulating while the application is busy executing some time-intensive code (for example, in response to a previous mouse click).

Test case is found below. Steps to reproduce (at least on my machine):

1. Click inside the window
2. Within the next second, wiggle the mouse a bit to create events
3. Observe printed-out mouse events when moving the cursor over the title bar
4. Try to press the window's close button

import org.lwjgl.*;
import org.lwjgl.input.*;
import org.lwjgl.opengl.*;

public final class MouseEventTest {
  public static void main(String[] args) {
    try {
      Display.setDisplayMode(new DisplayMode(640, 480));
      Display.create();
      Mouse  .create();
    } catch (LWJGLException ex) {
      ex.printStackTrace();
    }
    
    while (Display.isCreated()) {
      Display.update();
      
      if (Display.isCloseRequested()) {
        Mouse  .destroy();
        Display.destroy();
        
        continue;
      }
      
      while (Mouse.next()) {
        System.out.println(System.currentTimeMillis());
        System.out.println("eventNanoseconds=" + Mouse.getEventNanoseconds());
        System.out.println("eventX="           + Mouse.getEventX());
        System.out.println("eventY="           + Mouse.getEventY());
        System.out.println("eventDX="          + Mouse.getEventDX());
        System.out.println("eventDY="          + Mouse.getEventDY());
        System.out.println("eventDWheel="      + Mouse.getEventDWheel());
        System.out.println("eventButton="      + Mouse.getEventButton());
        System.out.println("eventButtonState=" + Mouse.getEventButtonState());
        System.out.println();
        
        if (Mouse.getEventButtonState()) {
          System.out.println("button down");
          System.out.println();
          
          try {
            Thread.sleep(1000);
          } catch (InterruptedException ex) {
          }
        }
      }
    }
  }
}

[Elviz]

The only workaround I have so far is to filter out mouse events whose coordinates are reported as lying on the edge of the client area when in windowed mode:

int x = Mouse.getEventX();
int y = Mouse.getEventY();

if (Mouse.isGrabbed()
    || Display.isFullscreen()
    || ((x > 0)
      && (x < Display.getDisplayMode().getWidth () - 1)
      && (y > 0)
      && (y < Display.getDisplayMode().getHeight() - 1))) {
  // process mouse event
}

This will eliminate some valid events, too, but most importantly, it doesn't solve the problem of the user intermittently being unable to press the window's close button or drag the window by its title bar.

I'd like to update my application from 2.1.0 to 2.2.1 to take advantage of the Windows icon fix, among other things, but this regression is currently stopping me.

[Matzon]

I am unable to reproduce the issue with getting mouse events in the titlebar area (using vista). However the other issues you appear to have, seems to be your own fault.

First of all, you need to either sync or yield in the general loop, so add: Display.sync(60);
And furthermore - if you sleep for 1 second on mouse down, then you wont process the messageloop nor anything else, causing you yo be unable to move the window.
I have "fixed" your example:

import org.lwjgl.*;
import org.lwjgl.input.*;
import org.lwjgl.opengl.*;

public final class MouseEventTest {
  public static void main(String[] args) {
    try {
      Display.setDisplayMode(new DisplayMode(640, 480));
      Display.create();
    } catch (LWJGLException ex) {
      ex.printStackTrace();
    }
    
    while (!Display.isCloseRequested()) {
      Display.sync(60);
      Display.update();
      
      while (Mouse.next()) {
        System.out.println(System.currentTimeMillis());
        System.out.println("eventNanoseconds=" + Mouse.getEventNanoseconds());
        System.out.println("eventX="           + Mouse.getEventX());
        System.out.println("eventY="           + Mouse.getEventY());
        System.out.println("eventDX="          + Mouse.getEventDX());
        System.out.println("eventDY="          + Mouse.getEventDY());
        System.out.println("eventDWheel="      + Mouse.getEventDWheel());
        System.out.println("eventButton="      + Mouse.getEventButton());
        System.out.println("eventButtonState=" + Mouse.getEventButtonState());
        System.out.println();
        
        if (Mouse.getEventButtonState()) {
          System.out.println("button down");
          System.out.println();
        }
      }
    }
    Display.destroy();
  }
}

[Elviz]

I am unable to reproduce the issue with getting mouse events in the titlebar area (using vista).

While I can reproduce it regularly, I cannot do so 100% reliably. You may need to try a number of times. Perhaps it's limited to Windows XP altogether. (Unfortunately, I don't have a Vista installation to check.)

Either way, the issue didn't exist in 2.1.0. It seems likely that it's caused by one of the changes in the Windows mouse code that happened between 2.1.0 and 2.2.1 (e.g., r3215, r3220, r3240).

if you sleep for 1 second on mouse down, then you wont process the messageloop nor anything else, causing you yo be unable to move the window.

That's true, of course, but I don't expect anything to be processed while sleeping, nor do I try to close or move the window during that period. The described issues arise when the time has passed and the game loop is in full swing again. The sleep call has only been inserted to simulate a situation in which the app is temporarily busier than usual (e.g., loading level data) because this seems to play some role in triggering the problem.

The bug I'm encountering is that, under certain circumstances related to mouse clicks, movements and timing, LWJGL's mouse behaviour gets into an abnormal state (kind of a pseudo-capture state). When this state has been reached, all of the following is true:

• Moving the mouse over the title bar or border of the window will cause mouse events to be reported. The mouse position will be clamped to (0,0)-(width-1,height-1) and the button state will be false. The expected behaviour is that no events will be reported.
• Clicking on the title bar or border or even the taskbar (which clearly isn't part of the LWJGL app) will cause a mouse event to be reported. The mouse position will be clamped to (0,0)-(width-1,height-1) and the button state will be true. The expected behaviour is that no event will be reported.
• If the click happens to be on one of the window buttons (close, minimize), the button will fail to be pressed down. If the click happens to be on the (rest of the) title bar, it will fail to initiate a drag. Both are really the same issue: the click doesn't get through to the Windows widget routines that usually handle these things.
• Once one mouse click has been made, things return to normal (until the same circumstances cause the abnormal state to be entered again).

you need to either sync or yield in the general loop, so add: Display.sync(60);

Why is that? Either vsync is on, in which case swapBuffers() will block, or vsync is off, in which case the game loop and the painting will run at maximum speed, with the frame rate determined by the capabilities of the graphics hardware. There is no "desired frame rate" I want to set.

I tried the changed example and was still able to reproduce the problem (after reinserting the sleep call, see above).

[Matzon]

Will try to see if I can reproduce the problem on my xp installation.
Please consider adding some debug code to lwjgl to see if you can pinpoint it. I am kinda thinking that if you move the mouse outside the window while sleeping for 1 sec, that some of the mouse tracking fails?

not sure though :/

[Matzon]

I can reproduce it on my XP box - not sure whats going on, can't seem to locate the problem

[Matzon]

I think I know what the problem is - I just not sure how to fix it yet ...
The problem appears to be the call to setCapture - and a missing releaseCapture.
I think that whats happening is that we're capturing in non-event code using isButtonDown and then releasing in handleMouseButton... the following changes may work - can you confirm?:

Index: java/org/lwjgl/opengl/WindowsDisplay.java
===================================================================
--- java/org/lwjgl/opengl/WindowsDisplay.java	(revision 3258)
+++ java/org/lwjgl/opengl/WindowsDisplay.java	(working copy)
@@ -672,15 +672,19 @@
 		if (mouse != null) {
 			mouse.handleMouseButton((byte)button, (byte)state, millis);
 			
-			// done with capture?
-			if(captureMouse != -1 && button == captureMouse && state == 0) {
-				nReleaseCapture();
-				captureMouse = -1;
+			if(!Mouse.isGrabbed()) {
+
+				// need to capture?
+				if(captureMouse == -1 && button != -1 && state == 1) {
+					captureMouse = button;
+					nSetCapture(hwnd);
+				}
 				
-				// force mouse update - else we will run into an issue where the
-				// button state is "stale" while captureMouse == -1 which causes
-				// handleMouseMoved to issue a setCapture.
-				Mouse.poll();
+				// done with capture?
+				if(captureMouse != -1 && button == captureMouse && state == 0) {
+					nReleaseCapture();
+					captureMouse = -1;
+				}				
 			}
 		}
 		
@@ -696,16 +700,6 @@
 	private void handleMouseMoved(int x, int y, long millis) {
 		if (mouse != null) {
 			mouse.handleMouseMoved(x, y, millis, shouldGrab());
-			
-			// Moving - while mouse is down?
-			// need to capture
-			if(!Mouse.isGrabbed()) {
-				int button = firstMouseButtonDown();
-				if(captureMouse == -1 && button != -1) {
-					captureMouse = button;
-					nSetCapture(hwnd);
-				}
-			}
 		}
 	}

what -I have done is basically move all the capture related stuff to the same method - seems to fix it on my end..

[Elviz]

Tried the patch, and the results are mixed. The good news is that so far I wasn't able to reproduce the original issue (as seen in MouseEventTest and my real application) again. Below you'll find some modified debug output.

Without the patch:

eventNanoseconds=31652078000000
eventX=341
eventY=237
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=true

button down  <--- first mouse click

nSetCapture

eventNanoseconds=31652265000000
eventX=341
eventY=237
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

eventNanoseconds=31653109000000
eventX=409
eventY=460
eventDX=68
eventDY=223
eventDWheel=0
eventButton=-1
eventButtonState=false

eventNanoseconds=31653125000000
eventX=413
eventY=479
eventDX=4
eventDY=22
eventDWheel=0
eventButton=-1
eventButtonState=false

(...)

eventNanoseconds=31654796000000
eventX=629
eventY=479
eventDX=0
eventDY=11
eventDWheel=0
eventButton=0
eventButtonState=true

button down  <--- second mouse click

nReleaseCapture
nReleaseCapture

eventNanoseconds=31655468000000
eventX=629
eventY=479
eventDX=0
eventDY=11
eventDWheel=0
eventButton=0
eventButtonState=false

eventNanoseconds=31655468000000
eventX=629
eventY=479
eventDX=0
eventDY=11
eventDWheel=0
eventButton=0
eventButtonState=false

With the patch applied:

nSetCapture

eventNanoseconds=30811312000000
eventX=322
eventY=279
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=true

button down  <--- first mouse click

nReleaseCapture
nReleaseCapture

eventNanoseconds=30811484000000
eventX=322
eventY=279
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

eventNanoseconds=30811484000000
eventX=322
eventY=279
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

eventNanoseconds=30812343000000
eventX=254
eventY=303
eventDX=-68
eventDY=24
eventDWheel=0
eventButton=-1
eventButtonState=false

eventNanoseconds=30812343000000
eventX=255
eventY=303
eventDX=1
eventDY=0
eventDWheel=0
eventButton=-1
eventButtonState=false

(...)

nSetCapture

eventNanoseconds=30829390000000
eventX=275
eventY=378
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=true

button down  <--- second mouse click

nReleaseCapture
nReleaseCapture

eventNanoseconds=30829609000000
eventX=275
eventY=378
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

eventNanoseconds=30829609000000
eventX=275
eventY=378
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

The bad news is that the patch appears to let the problem pop up in a different place, see the new test case below.

1. Click inside the window (-> mouse will be grabbed)
2. Press SPACE (-> mouse will be un-grabbed)
3. Bad mouse state reached again (try to press the window's close button)

import org.lwjgl.*;
import org.lwjgl.input.*;
import org.lwjgl.opengl.*;

public final class MouseGrabTest {
  public static void main(String[] args) {
    try {
      Display.setDisplayMode(new DisplayMode(640, 480));
      Display.create();
      Mouse  .create();
    } catch (LWJGLException ex) {
      ex.printStackTrace();
    }
    
    while (!Display.isCloseRequested()) {
      Display.sync(60);
      Display.update();
      
      while (Keyboard.next()) {
        if ((Keyboard.getEventKey() == Keyboard.KEY_SPACE)
            && Keyboard.getEventKeyState()) {
          Mouse.setGrabbed(false);
        }
      }
      
      while (Mouse.next()) {
        System.out.println("eventNanoseconds=" + Mouse.getEventNanoseconds());
        System.out.println("eventX="           + Mouse.getEventX());
        System.out.println("eventY="           + Mouse.getEventY());
        System.out.println("eventDX="          + Mouse.getEventDX());
        System.out.println("eventDY="          + Mouse.getEventDY());
        System.out.println("eventDWheel="      + Mouse.getEventDWheel());
        System.out.println("eventButton="      + Mouse.getEventButton());
        System.out.println("eventButtonState=" + Mouse.getEventButtonState());
        System.out.println();
        
        if (Mouse.getEventButtonState()) {
          System.out.println("button down");
          System.out.println();
          
          Mouse.setGrabbed(true);
        }
      }
    }
    
    Display.destroy();
  }
}

[Matzon]

think this is caused by the wrapped is mouse grapped - try and remove that.

[Elviz]

Indeed, removing the if(!Mouse.isGrabbed()) condition in the patched WindowsDisplay.handleMouseButton fixes it.

One more issue, though: two identical events are generated when a mouse button is released.

import org.lwjgl.*;
import org.lwjgl.input.*;
import org.lwjgl.opengl.*;

public final class MouseUpTest {
  public static void main(String[] args) {
    try {
      Display.setDisplayMode(new DisplayMode(640, 480));
      Display.create();
      Mouse  .create();
    } catch (LWJGLException ex) {
      ex.printStackTrace();
    }
    
    while (!Display.isCloseRequested()) {
      Display.sync(60);
      Display.update();
      
      while (Mouse.next()) {
        if (Mouse.getEventButton() != -1) {
          System.out.println("eventNanoseconds=" + Mouse.getEventNanoseconds());
          System.out.println("eventX="           + Mouse.getEventX());
          System.out.println("eventY="           + Mouse.getEventY());
          System.out.println("eventDX="          + Mouse.getEventDX());
          System.out.println("eventDY="          + Mouse.getEventDY());
          System.out.println("eventDWheel="      + Mouse.getEventDWheel());
          System.out.println("eventButton="      + Mouse.getEventButton());
          System.out.println("eventButtonState=" + Mouse.getEventButtonState());
          System.out.println();
          
          System.out.print  ("button " + Mouse.getEventButton() + " ");
          System.out.println(Mouse.getEventButtonState() ? "down" : "up");
          System.out.println();
        }
      }
    }
    
    Display.destroy();
  }
}

With 2.1.0:

eventNanoseconds=26028703000000
eventX=303
eventY=265
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=true

button 0 down

eventNanoseconds=26030531000000
eventX=303
eventY=265
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

button 0 up

With 2.2.1+patch: (printlns for nSetCapture and nReleaseCapture added)

nSetCapture

eventNanoseconds=27259015000000
eventX=306
eventY=264
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=true

button 0 down

nReleaseCapture
nReleaseCapture

eventNanoseconds=27259296000000
eventX=306
eventY=264
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

button 0 up

eventNanoseconds=27259296000000
eventX=306
eventY=264
eventDX=0
eventDY=0
eventDWheel=0
eventButton=0
eventButtonState=false

button 0 up

[Elviz]

I didn't study the mouse code in any detail, but it seems that the line captureMouse = -1; in the patched handleMouseButton should be moved before the call to nReleaseCapture. Otherwise, handleMouseButton will be entered a second time as a result of the WM_CAPTURECHANGED handler in doHandleMessage.

[Matzon]

I didn't study the mouse code in any detail, but it seems that the line captureMouse = -1; in the patched handleMouseButton should be moved before the call to nReleaseCapture. Otherwise, handleMouseButton will be entered a second time as a result of the WM_CAPTURECHANGED handler in doHandleMessage.

yeah - going to look into it this weekend - just a bit busy.

[Elviz]

In an attempt to move this forward, here is the updated patch. I haven't had any problems with this code so far. (Only tested it with my application, though.)

Index: java/org/lwjgl/opengl/WindowsDisplay.java
===================================================================
--- java/org/lwjgl/opengl/WindowsDisplay.java	(revision 3287)
+++ java/org/lwjgl/opengl/WindowsDisplay.java	(working copy)
@@ -690,15 +690,16 @@
 		if (mouse != null) {
 			mouse.handleMouseButton((byte)button, (byte)state, millis);
 			
+			// need to capture?
+			if(captureMouse == -1 && button != -1 && state == 1) {
+				captureMouse = button;
+				nSetCapture(hwnd);
+			}
+			
 			// done with capture?
 			if(captureMouse != -1 && button == captureMouse && state == 0) {
-				nReleaseCapture();
 				captureMouse = -1;
-				
-				// force mouse update - else we will run into an issue where the
-				// button state is "stale" while captureMouse == -1 which causes
-				// handleMouseMoved to issue a setCapture.
-				Mouse.poll();
+				nReleaseCapture();
 			}
 		}
 		
@@ -714,16 +715,6 @@
 	private void handleMouseMoved(int x, int y, long millis) {
 		if (mouse != null) {
 			mouse.handleMouseMoved(x, y, millis, shouldGrab());
-			
-			// Moving - while mouse is down?
-			// need to capture
-			if(!Mouse.isGrabbed()) {
-				int button = firstMouseButtonDown();
-				if(captureMouse == -1 && button != -1) {
-					captureMouse = button;
-					nSetCapture(hwnd);
-				}
-			}
 		}
 	}

[Matzon]

totally slipped my mind... this still happens in 2.3 or ? - if it does, I'll apply the patch and see how it fares.

[Elviz]

this still happens in 2.3 or ?

Yes.