0 Replies Latest reply on Oct 9, 2018 5:31 AM by 3480917

    locktimeout on secondary database inserts

    3480917

      Hello all,

       

      My current setup uses BDBJE 4.1.10. I have 10 threads trying to write a Keyword object to the database.

      The nature of the data is that the secondary keys - stores and items have a lot of duplicates. Meaning, many Keyword objects map to the same store and items entries.

       

      LOCK_N_LOCK_TABLES is 7

       

      @Entity

      public class Keyword implements Serializable {

       

          @PrimaryKey

          private String key;

       

          @SecondaryKey(relate = Relationship.MANY_TO_MANY)

          private List<Long> stores = new ArrayList<Long>();

       

          @SecondaryKey(relate = Relationship.MANY_TO_MANY)

          private List<Long> items = new ArrayList<Long>();

      }

       

      About 10% (out of 100,000) of the secondary database inserts fail with a LockTimeoutException.

       

      Looking at the stacktrace, I understand that the thread 6 waited to obtain the lock for 2000ms before giving up and there are 3 more threads waiting.

       

      Essentially, the operation that the Owner thread (thread 10) is performing is taking more than 2000ms. Thread 6 is trying to insert a record in the secondary database items.

       

      Stacktrace:

       

      com.sleepycat.je.LockTimeoutException: (JE 4.1.10) Lock expired. 
      
      Locker 786527582 -1_fileChannelTaskExecutor-6_ThreadLocker: waited for lock on database=persist#faceted_store#com.xyzq.seo.bdb.entity.Keyword#items 
      
      LockAddr:43409842 node=281964 type=WRITE grant=WAIT_NEW timeoutMillis=2000 startTime=1538759059393 endTime=1538759061393 
      
      Owners: [<LockInfo locker="786774600 -1_fileChannelTaskExecutor-10_ThreadLocker" type="WRITE"/>] 
      
      Waiters: 
      [
      <LockInfo locker="1270863098 -1_fileChannelTaskExecutor-9_ThreadLocker" type="WRITE"/>, 
      <LockInfo locker="1244207474 -1_fileChannelTaskExecutor-4_ThreadLocker" type="WRITE"/>, 
      <LockInfo locker="1391625295 -1_fileChannelTaskExecutor-1_ThreadLocker" type="WRITE"/>
      ] 
      at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:608
       at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:567
       at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:75
       at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:385
       at com.sleepycat.je.txn.LockManager.lock(LockManager.java:272
       at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:134
       at com.sleepycat.je.txn.Locker.lock(Locker.java:453
       at com.sleepycat.je.dbi.CursorImpl.lockDupCountLN(CursorImpl.java:2768
       at com.sleepycat.je.tree.Tree.insertDuplicate(Tree.java:2847
       at com.sleepycat.je.tree.Tree.insert(Tree.java:2488
       at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:1209
       at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:1799
       at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:1756
       at com.sleepycat.je.Cursor.putNotify(Cursor.java:1689
       at com.sleepycat.je.Cursor.putInternal(Cursor.java:1626
       at com.sleepycat.je.SecondaryDatabase.insertKey(SecondaryDatabase.java:984
       at com.sleepycat.je.SecondaryDatabase.updateSecondary(SecondaryDatabase.java:909
       at com.sleepycat.je.SecondaryTrigger.databaseUpdated(SecondaryTrigger.java:41
       at com.sleepycat.je.Database.notifyTriggers(Database.java:2016
       at com.sleepycat.je.Cursor.putNotify(Cursor.java:1702
       at com.sleepycat.je.Cursor.putInternal(Cursor.java:1626
       at com.sleepycat.je.Database.putInternal(Database.java:1186
       at com.sleepycat.je.Database.put(Database.java:1058
       at com.sleepycat.persist.PrimaryIndex.putNoReturn(PrimaryIndex.java:479
       at com.sleepycat.persist.PrimaryIndex.putNoReturn(PrimaryIndex.java:442
       at com.xyzq.bdb.cache.da.impl.BDBDataAccessor.create(BDBDataAccessor.java:77)
      

       

       

      I tried the following

      - Reduced the lock time. Hoping that the winner would release the lock as soon as its done insering records into

      - Catch LockTimeOutException and retry inserting the keyword later. This works but it's manual and takes time. 10% of keywords fail because of a LockTimeOutException. ~ 10k out of 100k.

       

       

      I have a few questions:

      Is there a limit on the number of threads that can write to the DB or does it depend on the data?

      Would it help if I model the entities as pure key value pairs instead of having secondary databases?

       

      @Entity

      Keyword

          @PrimaryKey

          - key

       

      @Entity

      Item

          -id

          -key [reference to Keyword Obj]

       

      @Entity

      Store

          - id

          - key [reference to Keyword obj.]

       

       

       

      Thanks!

      - K