Could this be it:
glBufferData(GL_ARRAY_BUFFER, buffer, GL_STREAM_DRAW);
The buffer at this point is fresh out of:
buffer = MemoryUtil.memAlloc(BUFFER_SIZE);
i.e. it is empty? The limit is 0? I don't know what the wrapper glBufferData does, but if the buffer is empty at this point, could it be like calling glBufferData(GL_ARRAY_BUFFER, 0, GL_STREAM_DRAW), which translates to allocate a buffer of 0 bytes.
Then, each glBufferSubData(GL_ARRAY_BUFFER, 0, buffer); which is called with a non-empty, flipped buffer will always require more space than 0, which will cause the driver to allocate another memory chunk on NVIDIA 1080?