How do I switch to the latest version of LMDB

[joshua tang]

Hi Guys:
    when I using lwgjl 3.1.3 lmdb api to write certain data item to the database, sometimes I get SIGSEGV error. when switched to another LMDB java binding (lmdbjni), all works fine.

    the reason I believe is that lmdbjni using a more recent branch of native lmdb library.

   So my question is How can I customized the lwgjl library to use the latest native lmdb library (is it possible that I replace the native code in my local git repository to achieve that?)


Best regards,

[spasi]

LWJGL 3.1.3 includes lmdb 0.9.21.

The lmdb build that comes with LWJGL 3.1.5 is based on the latest commit of the master branch, plus a custom patch that improves incremental file growth performance on Windows.

Afaict, lmdbjni is still at 0.9.19.

Please try again with LWJGL 3.1.5 or 3.1.6-SNAPSHOT. If the problem persists, please prepare an MCVE and I'll gladly have a look and maybe help identify what's causing the crash.

[joshua tang]

Hi :

   Thanks for your quick response. I will switch to 3.15 to see what is the result. If still get the same problem I will give you detailed information about the issue I encountered.

Thanks again.

[joshua tang]

Hi Guy:

    I have attached the file dumped when the JVM crash. 

    I am writing a DUP_SORT mode database, using mdb_cursor_put. 

    there have been a lot of records successfully written. I found the problematic record that cause the issue, write it to the database again, the issue happens again.
    But when clean all the db, then write the problematic record, it can be successfully written. That is strange.

[spasi]

It would help if you could reproduce the crash in a simple program. Share it here and I'll try to debug it.

[joshua tang]

import org.lwjgl.PointerBuffer;
import org.lwjgl.util.lmdb.LMDB;
import org.lwjgl.system.MemoryStack;
import org.lwjgl.system.MemoryUtil;
import org.lwjgl.util.lmdb.MDBVal;

import java.io.*;
import java.nio.IntBuffer;


public class CrashLmdb {
    static long env;
    static int db;

    public static void main(String[] args) throws IOException {
        try (MemoryStack stack = MemoryStack.stackPush()) {
            File dir = new File("/home/joshua/Data1");
            PointerBuffer envPointer = stack.mallocPointer(1);
            LMDB.mdb_env_create(envPointer);
            env = envPointer.get(0);
            LMDB.mdb_env_set_mapsize(env, 1024 * 1024 * 1024);
            LMDB.mdb_env_set_maxreaders(env, 512);
            LMDB.mdb_env_set_maxdbs(env, 512);
            LMDB.mdb_env_open(env, dir.getPath(), 0, 0644);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            IntBuffer dbPointer = stack.mallocInt(1);
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            LMDB.mdb_dbi_open(txn, "mydb", LMDB.MDB_CREATE|LMDB.MDB_DUPSORT, dbPointer);
            LMDB.mdb_txn_commit(txn);
            db = dbPointer.get(0);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            PointerBuffer cursorPointer = stack.mallocPointer(1);
            long cursor = cursorPointer.get(0);
            LMDB.mdb_cursor_open(txn, db,cursorPointer);
            MDBVal key = MDBVal.callocStack(stack);
            MDBVal value = MDBVal.callocStack(stack);
            key.mv_data(stack.ASCII("AA ABE  AMS  ", false));
            value.mv_data(stack.ASCII("001AA ABE  AMS  KMZ     00100009", false));
            LMDB.mdb_cursor_put(cursor, key, value, 0);
            LMDB.mdb_cursor_close(cursor);
            LMDB.mdb_txn_commit(txn);
        }

    }
}


just run the above code will cause a  JVM crash, please point the db directory to a clean directory. Hope this could help you help me.

[spasi]

The bug is in these 3 lines:

PointerBuffer cursorPointer = stack.mallocPointer(1);
long cursor = cursorPointer.get(0);
LMDB.mdb_cursor_open(txn, db,cursorPointer);

You're dereferencing the cursor pointer before calling mdb_cursor_open. It's trivially fixed by doing:

PointerBuffer cursorPointer = stack.mallocPointer(1);
LMDB.mdb_cursor_open(txn, db,cursorPointer);
long cursor = cursorPointer.get(0);

[joshua tang]

Hi Guy:
 
     I made a mistake in the previous program. you are right about it.  this time I give you the right code, and a data.txt file.  I try to load all data.txt file into db, the last record seems cause the SIGSEGV error.

     please run the following code against the attached data.txt file.  Appreciate your help.

import org.lwjgl.PointerBuffer;
import org.lwjgl.util.lmdb.LMDB;
import org.lwjgl.system.MemoryStack;
import org.lwjgl.system.MemoryUtil;
import org.lwjgl.util.lmdb.MDBVal;

import java.io.*;
import java.nio.IntBuffer;


public class CrashLmdb {
    static long env;
    static int db;

    public static void main(String[] args) throws IOException {
        try (MemoryStack stack = MemoryStack.stackPush()) {
            File dir = new File("/home/joshua/Data1");
            PointerBuffer envPointer = stack.mallocPointer(1);
            LMDB.mdb_env_create(envPointer);
            env = envPointer.get(0);
            LMDB.mdb_env_set_mapsize(env, 1024 * 1024 * 1024);
            LMDB.mdb_env_set_maxreaders(env, 512);
            LMDB.mdb_env_set_maxdbs(env, 512);
            LMDB.mdb_env_open(env, dir.getPath(), 0, 0644);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            IntBuffer dbPointer = stack.mallocInt(1);
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            LMDB.mdb_dbi_open(txn, "mydb", LMDB.MDB_CREATE|LMDB.MDB_DUPSORT, dbPointer);
            LMDB.mdb_txn_commit(txn);
            db = dbPointer.get(0);
        }

        FileReader input = new FileReader("/home/joshua/Desktop/data.txt");
        BufferedReader reader = new BufferedReader(input);
        String strKey = reader.readLine();
        String strValue = reader.readLine();
        while (null != strKey & null != strValue) {
            try (MemoryStack stack = MemoryStack.stackPush()) {
                PointerBuffer txnPointer = stack.mallocPointer(1);
                LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
                long txn = txnPointer.get(0);
                PointerBuffer cursorPointer = stack.mallocPointer(1);
                LMDB.mdb_cursor_open(txn, db,cursorPointer);
                long cursor = cursorPointer.get(0);
                MDBVal key = MDBVal.callocStack(stack);
                MDBVal value = MDBVal.callocStack(stack);
                key.mv_data(stack.ASCII(strKey, false));
                value.mv_data(stack.ASCII(strValue, false));
                LMDB.mdb_cursor_put(cursor, key, value, 0);
                LMDB.mdb_cursor_close(cursor);
                LMDB.mdb_txn_commit(txn);
            }
            strKey = reader.readLine();
            strValue = reader.readLine();
        }
        LMDB.mdb_dbi_close(env, db);
        LMDB.mdb_env_close(env);
    }
}

[joshua tang]

Hi :

   New findings.

   if these lines in the previous program:
         key.mv_data(stack.ASCII(strKey, false));
         value.mv_data(stack.ASCII(strValue, false));
   changed to:
        key.mv_data(stack.ASCII(strKey, true));
        value.mv_data(stack.ASCII(strValue, true));

  for the attached dataset there are no SIGSEGV error happens anymore.   But when I try to load more data, I sitll get SIGSEGV at some point,  always on mdb_cursor_put method.

[spasi]

I was able to reproduce this. It only happens on the Linux build which is compiled with GCC. Debugging showed that you're hitting this issue, which I believe is caused by GCC vectorizing a loop doing misaligned memory access.

The issue is fixed by:

- Changing optimization level from -O3 to -O2.
- Using -O3 and -fno-strict-aliasing.
- Using -O3 and -fno-tree-vectorize.
- Changing the offending loop from:

for (i=0; i<NUMKEYS(fp); i++)
    mp->mp_ptrs[ i ] = fp->mp_ptrs[ i ] + offset; // fp->mp_ptrs[ i ] is misaligned

to

memcpy(mp->mp_ptrs, fp->mp_ptrs, NUMKEYS(fp) * sizeof(indx_t));
for (i=0; i<NUMKEYS(fp); i++)
    mp->mp_ptrs[ i ] += offset;

The next LWJGL 3.1.6 snapshot will incorporate the last option. Thank you for reporting this!

[joshua tang]

what time will  the next LWJGL 3.1.6 snapshot be ready?

[spasi]

In 30-60'.

[joshua tang]

Seems the issue never happened when using the latest 3.1.6 snapshot.

Great job, thanks guy.