Everything you upload to OpenGL via glBufferData will be pretty certainly committed to video memory.
When you don't want something to be stored on the graphics card, decommit the memory of the buffer object via:
glBindBuffer(GL_ARRAY_BUFFER, bufferId);
glBufferData(GL_ARRAY_BUFFER, 0, GL_STATIC_DRAW);
and possibly delete the buffer object's name via glDeleteBuffers(). But you don't need to do that, because you could recommit memory to it later on.
Then later, when some chunk is about to become visible, you recommit the data again to the buffer object via another call to glBufferData(), but now with the actual data.
However, an even better way would be to ask yourself why you need this rediculuous precision everywhere in the terrain, and whether you can employ some LOD mechanism to reduce the number of vertices. For example: if your terrain has medium/large areas of constant first order derivatives (i.e. its slope does not change), you can make a single triangle/quad out of that area.
Another example would be view-dependent LOD, in which you constantly adapt the tessellation of the terrain based on the distance to the viewer. One practical example of this is "Chunked Lod".
This will in the end save you the most memory and will massively improve rendering performance.
Have a look at this:
http://vterrain.org/LOD/Papers/