Hello Guest

How do I switch to the latest version of LMDB

  • 12 Replies
  • 8553 Views
How do I switch to the latest version of LMDB
« on: November 28, 2017, 04:21:25 »
Hi Guys:
    when I using lwgjl 3.1.3 lmdb api to write certain data item to the database, sometimes I get SIGSEGV error. when switched to another LMDB java binding (lmdbjni), all works fine.

    the reason I believe is that lmdbjni using a more recent branch of native lmdb library.

   So my question is How can I customized the lwgjl library to use the latest native lmdb library (is it possible that I replace the native code in my local git repository to achieve that?)


Best regards,

*

Offline spasi

  • *****
  • 2261
    • WebHotelier
Re: How do I switch to the latest version of LMDB
« Reply #1 on: November 28, 2017, 09:29:02 »
LWJGL 3.1.3 includes lmdb 0.9.21.

The lmdb build that comes with LWJGL 3.1.5 is based on the latest commit of the master branch, plus a custom patch that improves incremental file growth performance on Windows.

Afaict, lmdbjni is still at 0.9.19.

Please try again with LWJGL 3.1.5 or 3.1.6-SNAPSHOT. If the problem persists, please prepare an MCVE and I'll gladly have a look and maybe help identify what's causing the crash.

Re: How do I switch to the latest version of LMDB
« Reply #2 on: November 29, 2017, 06:47:43 »
Hi :

   Thanks for your quick response. I will switch to 3.15 to see what is the result. If still get the same problem I will give you detailed information about the issue I encountered.

Thanks again.

Re: How do I switch to the latest version of LMDB
« Reply #3 on: November 29, 2017, 10:48:21 »
Hi Guy:

    I have attached the file dumped when the JVM crash. 

    I am writing a DUP_SORT mode database, using mdb_cursor_put. 

    there have been a lot of records successfully written. I found the problematic record that cause the issue, write it to the database again, the issue happens again.
    But when clean all the db, then write the problematic record, it can be successfully written. That is strange.

*

Offline spasi

  • *****
  • 2261
    • WebHotelier
Re: How do I switch to the latest version of LMDB
« Reply #4 on: November 29, 2017, 11:25:46 »
It would help if you could reproduce the crash in a simple program. Share it here and I'll try to debug it.

Re: How do I switch to the latest version of LMDB
« Reply #5 on: November 29, 2017, 17:20:05 »
import org.lwjgl.PointerBuffer;
import org.lwjgl.util.lmdb.LMDB;
import org.lwjgl.system.MemoryStack;
import org.lwjgl.system.MemoryUtil;
import org.lwjgl.util.lmdb.MDBVal;

import java.io.*;
import java.nio.IntBuffer;


public class CrashLmdb {
    static long env;
    static int db;

    public static void main(String[] args) throws IOException {
        try (MemoryStack stack = MemoryStack.stackPush()) {
            File dir = new File("/home/joshua/Data1");
            PointerBuffer envPointer = stack.mallocPointer(1);
            LMDB.mdb_env_create(envPointer);
            env = envPointer.get(0);
            LMDB.mdb_env_set_mapsize(env, 1024 * 1024 * 1024);
            LMDB.mdb_env_set_maxreaders(env, 512);
            LMDB.mdb_env_set_maxdbs(env, 512);
            LMDB.mdb_env_open(env, dir.getPath(), 0, 0644);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            IntBuffer dbPointer = stack.mallocInt(1);
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            LMDB.mdb_dbi_open(txn, "mydb", LMDB.MDB_CREATE|LMDB.MDB_DUPSORT, dbPointer);
            LMDB.mdb_txn_commit(txn);
            db = dbPointer.get(0);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            PointerBuffer cursorPointer = stack.mallocPointer(1);
            long cursor = cursorPointer.get(0);
            LMDB.mdb_cursor_open(txn, db,cursorPointer);
            MDBVal key = MDBVal.callocStack(stack);
            MDBVal value = MDBVal.callocStack(stack);
            key.mv_data(stack.ASCII("AA ABE  AMS  ", false));
            value.mv_data(stack.ASCII("001AA ABE  AMS  KMZ     00100009", false));
            LMDB.mdb_cursor_put(cursor, key, value, 0);
            LMDB.mdb_cursor_close(cursor);
            LMDB.mdb_txn_commit(txn);
        }

    }
}


just run the above code will cause a  JVM crash, please point the db directory to a clean directory. Hope this could help you help me. :)

*

Offline spasi

  • *****
  • 2261
    • WebHotelier
Re: How do I switch to the latest version of LMDB
« Reply #6 on: November 29, 2017, 17:52:28 »
The bug is in these 3 lines:

Code: [Select]
PointerBuffer cursorPointer = stack.mallocPointer(1);
long cursor = cursorPointer.get(0);
LMDB.mdb_cursor_open(txn, db,cursorPointer);

You're dereferencing the cursor pointer before calling mdb_cursor_open. It's trivially fixed by doing:

Code: [Select]
PointerBuffer cursorPointer = stack.mallocPointer(1);
LMDB.mdb_cursor_open(txn, db,cursorPointer);
long cursor = cursorPointer.get(0);

Re: How do I switch to the latest version of LMDB
« Reply #7 on: November 30, 2017, 02:52:58 »
Hi Guy:
 
     I made a mistake in the previous program. you are right about it.  this time I give you the right code, and a data.txt file.  I try to load all data.txt file into db, the last record seems cause the SIGSEGV error.
 
     please run the following code against the attached data.txt file.  Appreciate your help.

import org.lwjgl.PointerBuffer;
import org.lwjgl.util.lmdb.LMDB;
import org.lwjgl.system.MemoryStack;
import org.lwjgl.system.MemoryUtil;
import org.lwjgl.util.lmdb.MDBVal;

import java.io.*;
import java.nio.IntBuffer;


public class CrashLmdb {
    static long env;
    static int db;

    public static void main(String[] args) throws IOException {
        try (MemoryStack stack = MemoryStack.stackPush()) {
            File dir = new File("/home/joshua/Data1");
            PointerBuffer envPointer = stack.mallocPointer(1);
            LMDB.mdb_env_create(envPointer);
            env = envPointer.get(0);
            LMDB.mdb_env_set_mapsize(env, 1024 * 1024 * 1024);
            LMDB.mdb_env_set_maxreaders(env, 512);
            LMDB.mdb_env_set_maxdbs(env, 512);
            LMDB.mdb_env_open(env, dir.getPath(), 0, 0644);
        }

        try (MemoryStack stack = MemoryStack.stackPush()) {
            IntBuffer dbPointer = stack.mallocInt(1);
            PointerBuffer txnPointer = stack.mallocPointer(1);
            LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
            long txn = txnPointer.get(0);
            LMDB.mdb_dbi_open(txn, "mydb", LMDB.MDB_CREATE|LMDB.MDB_DUPSORT, dbPointer);
            LMDB.mdb_txn_commit(txn);
            db = dbPointer.get(0);
        }

        FileReader input = new FileReader("/home/joshua/Desktop/data.txt");
        BufferedReader reader = new BufferedReader(input);
        String strKey = reader.readLine();
        String strValue = reader.readLine();
        while (null != strKey & null != strValue) {
            try (MemoryStack stack = MemoryStack.stackPush()) {
                PointerBuffer txnPointer = stack.mallocPointer(1);
                LMDB.mdb_txn_begin(env, MemoryUtil.NULL, 0, txnPointer);
                long txn = txnPointer.get(0);
                PointerBuffer cursorPointer = stack.mallocPointer(1);
                LMDB.mdb_cursor_open(txn, db,cursorPointer);
                long cursor = cursorPointer.get(0);
                MDBVal key = MDBVal.callocStack(stack);
                MDBVal value = MDBVal.callocStack(stack);
                key.mv_data(stack.ASCII(strKey, false));
                value.mv_data(stack.ASCII(strValue, false));
                LMDB.mdb_cursor_put(cursor, key, value, 0);
                LMDB.mdb_cursor_close(cursor);
                LMDB.mdb_txn_commit(txn);
            }
            strKey = reader.readLine();
            strValue = reader.readLine();
        }
        LMDB.mdb_dbi_close(env, db);
        LMDB.mdb_env_close(env);
    }
}
« Last Edit: November 30, 2017, 03:25:45 by joshua tang »

Re: How do I switch to the latest version of LMDB
« Reply #8 on: November 30, 2017, 06:24:21 »
Hi :

   New findings.

   if these lines in the previous program:
         key.mv_data(stack.ASCII(strKey, false));
         value.mv_data(stack.ASCII(strValue, false));
   changed to:
        key.mv_data(stack.ASCII(strKey, true));
        value.mv_data(stack.ASCII(strValue, true));

  for the attached dataset there are no SIGSEGV error happens anymore.   But when I try to load more data, I sitll get SIGSEGV at some point,  always on mdb_cursor_put method.
« Last Edit: November 30, 2017, 09:14:41 by joshua tang »

*

Offline spasi

  • *****
  • 2261
    • WebHotelier
Re: How do I switch to the latest version of LMDB
« Reply #9 on: November 30, 2017, 10:20:13 »
I was able to reproduce this. It only happens on the Linux build which is compiled with GCC. Debugging showed that you're hitting this issue, which I believe is caused by GCC vectorizing a loop doing misaligned memory access.

The issue is fixed by:

- Changing optimization level from -O3 to -O2.
- Using -O3 and -fno-strict-aliasing.
- Using -O3 and -fno-tree-vectorize.
- Changing the offending loop from:

Quote
for (i=0; i<NUMKEYS(fp); i++)
    mp->mp_ptrs[ i ] = fp->mp_ptrs[ i ] + offset; // fp->mp_ptrs[ i ] is misaligned

to

Quote
memcpy(mp->mp_ptrs, fp->mp_ptrs, NUMKEYS(fp) * sizeof(indx_t));
for (i=0; i<NUMKEYS(fp); i++)
    mp->mp_ptrs[ i ] += offset;

The next LWJGL 3.1.6 snapshot will incorporate the last option. Thank you for reporting this!
« Last Edit: December 01, 2017, 09:31:50 by spasi »

Re: How do I switch to the latest version of LMDB
« Reply #10 on: November 30, 2017, 10:37:23 »
what time will  the next LWJGL 3.1.6 snapshot be ready?

*

Offline spasi

  • *****
  • 2261
    • WebHotelier
Re: How do I switch to the latest version of LMDB
« Reply #11 on: November 30, 2017, 10:39:25 »
In 30-60'.

Re: How do I switch to the latest version of LMDB
« Reply #12 on: December 01, 2017, 03:11:51 »
Seems the issue never happened when using the latest 3.1.6 snapshot.

Great job, thanks guy.