Performance issue when swapping buffers

[vesa1995]

At the end of my render code I call glfwSwapBuffers(), it is a GLFW function which swaps back/front frame buffer. I surround that call with my performance measuring code:

performance.start();
glfwSwapBuffers();
performance.end();

My update_interval is 0.008333333 sec, which is 1/120 sec (120 FPS). Once per minute, it takes about 10*update_interval seconds for glSwapBuffers() to unblock.
My vsync is off via glfwSwapInterval(0).

Can anyone please give me an idea why is this happening, if you need more information I will provide it.