I'm not sure if this is the best way, but the first thing I can think of is to make the 'scroll area' a few rows / columns of pixels, not only the first and last row / col. Then, you keep track of the last known mouse position. If the mouse exits the window, this value will not be updated and therefore will be still in the 'scroll area'. In every update of the game you check the mouse position you saved yourself, and scroll accordingly.