7 Replies Latest reply: Apr 25, 2013 4:42 PM by user11188564 RSS

    BDB Deadlocked when used with TXN_BULK  option

    user11188564
      We have a multi-threaded application with each thread having its own DBHANDLE. We are using the BDSQL interface. We turned on the TXN_BULK option using a pragma -this was to avoid another problem which we encountered. The expectation was that if we child transactions that we can avoid this problem.


      Turning this option "on" resulted in a deadlock.The application has a few threads inserting, deleting & updating tables in the database. Also, there are few threads which query the database. The following is the "bt" of the threads that were waiting on locks. The linenumbers may be a little bit off as we had a few debug statements.

      Thread 1


      0x60000000c05681f0:0 in nanosleep2sys+0x30 ()
      from /usr/lib/hpux32/libc.so.1
      #1 0x60000000c0578f20:0 in __nanosleep2+0xc0 () from /usr/lib/hpux32/libc.so.1
      #2 0x60000000c040fe20:0 in sleep+0x410 () from /usr/lib/hpux32/libc.so.1
      #3 0x60000000d36d9ac0:0 in __os_sleep (env=0x40038730, secs=1, usecs=0)
      at ../src/os/os_yield.c:98
      #4 0x60000000d36d99b0:0 in __os_yield (env=0x40038730, secs=1, usecs=0)
      at ../src/os/os_yield.c:48
      #5 0x60000000d36ac110:0 in __memp_sync_int (env=0x40038730, dbmfp=0x40038b40,
      trickle_max=0, flags=8, wrote_totalp=0x0, interruptedp=0x0)
      at ../src/mp/mp_sync.c:485
      #6 0x60000000d375f9d0:0 in __txn_flush_fe_files (txn=0x400edbe0)
      at ../src/txn/txn_util.c:645
      #7 0x60000000d3726470:0 in __txn_commit (txn=0x400edbe0, flags=0)
      at ../src/txn/txn.c:784
      #8 0x60000000d371f080:0 in __txn_commit_pp (txn=0x400edbe0, flags=0)
      at ../src/txn/txn.c:632
      #9 0x60000000d2d86150:0 in sqlite3BtreeCommitPhaseTwo (p=0x400e0678,
      bCleanup=0) at ../lang/sql/generated/sqlite3.c:38160
      #10 0x60000000d2debd90:0 in vdbeCommit (db=0x400e0028, p=0x40177778)
      at ../lang/sql/generated/sqlite3.c:49931
      #11 0x60000000d2dee9a0:0 in sqlite3VdbeHalt (p=0x40177778)
      at ../lang/sql/generated/sqlite3.c:50321
      ---Type <return> to continue, or q <return> to quit---
      #12 0x60000000d2e08670:0 in sqlite3VdbeExec (p=0x40177778)
      at ../lang/sql/generated/sqlite3.c:56116
      #13 0x60000000d2df39e0:0 in sqlite3Step (p=0x40177778)
      at ../lang/sql/generated/sqlite3.c:51771
      #14 0x60000000d2d1de50:0 in sqlite3_step (pStmt=0x40177778)
      at ../lang/sql/generated/sqlite3.c:51836
      #15 0x400ec80:0 in exec(sqlite3*,std::basic_string<char,std::char_traits<char>,std::allocator<char> >)+0x240 ()
      #16 0x4018440:0 in insert(void*)+0xa70 ()
      #17 0x60000000c013b6e0:0 in __pthread_bound_body+0x190 ()
      from /usr/lib/hpux32/libpthread.so.1

      Thread 2

      #0 0x60000000c020f070:0 in __lwp_cond_wait_sys+0x30 ()
      from /usr/lib/hpux32/libpthread.so.1
      #1 0x60000000c01259c0:0 in <unknown_procedure> + 0xa0 ()
      from /usr/lib/hpux32/libpthread.so.1
      #2 0x60000000c01217c0:0 in <unknown_procedure> + 0x900 ()
      from /usr/lib/hpux32/libpthread.so.1
      #3 0x60000000c011f230:0 in pthread_cond_wait+0xd0 ()
      from /usr/lib/hpux32/libpthread.so.1
      #4 0x60000000d2f3a590:2 in inline __db_pthread_mutex_condwait (timespec=0x0,
      mutexp=0xc619f7d0, mutex=1716, env=0x40038730)
      at ../src/mutex/mut_pthread.c:321
      #5 0x60000000d2f3a370:0 in __db_pthread_mutex_lock (env=0x40038730,
      mutex=1716, timeout=0) at ../src/mutex/mut_pthread.c:416
      #6 0x60000000d33759e0:0 in __lock_get_internal (lt=0x4002d550,
      sh_locker=0xd896de20, flags=0, obj=0x400f2af0, lock_mode=DB_LOCK_WRITE,
      timeout=0, lock=0x77740af0) at ../src/lock/lock.c:983
      #7 0x60000000d33816a0:0 in __lock_get (env=0x40038730, locker=0xd896de20,
      flags=0, obj=0x400f2af0, lock_mode=DB_LOCK_WRITE, lock=0x77740af0)
      at ../src/lock/lock.c:463
      #8 0x60000000d3490f80:0 in __db_lget (dbc=0x400f2a60, action=0, pgno=0,
      mode=DB_LOCK_WRITE, lkflags=0, lockp=0x77740af0)
      at ../src/db/db_meta.c:1255
      #9 0x60000000d34928e0:0 in __db_free (dbc=0x400f2a60, h=0xd5991d44, flags=0)
      ---Type <return> to continue, or q <return> to quit---
      at ../src/db/db_meta.c:334
      #10 0x60000000d34b2290:0 in __db_doff (dbc=0x400f2a60, pgno=0)
      at ../src/db/db_overflow.c:479
      #11 0x60000000d2f8cbd0:0 in __bam_ditem (dbc=0x400f2a60, h=0xd5a2d184,
      indx=188) at ../src/btree/bt_delete.c:138
      #12 0x60000000d2f78e10:0 in __bamc_physdel (dbc=0x400f2a60)
      at ../src/btree/bt_cursor.c:2907
      #13 0x60000000d2f6c5e0:0 in __bamc_close (dbc=0x400f2a60, root_pgno=0,
      rmroot=0x0) at ../src/btree/bt_cursor.c:548
      #14 0x60000000d3415b20:0 in __dbc_close (dbc=0x400f2a60)
      at ../src/db/db_cam.c:121
      #15 0x60000000d3479c10:0 in __dbc_close_pp (dbc=0x400f2a60)
      at ../src/db/db_iface.c:2107
      #16 0x60000000d2da0e20:0 in sqlite3BtreeDelete (pCur=0x40152090)
      at ../lang/sql/generated/sqlite3.c:40325
      #17 0x60000000d2e1b420:0 in sqlite3VdbeExec (p=0x40143cc8)
      at ../lang/sql/generated/sqlite3.c:57818
      #18 0x60000000d2df39e0:0 in sqlite3Step (p=0x40143cc8)
      at ../lang/sql/generated/sqlite3.c:51771
      #19 0x60000000d2d1de50:0 in sqlite3_step (pStmt=0x40143cc8)
      at ../lang/sql/generated/sqlite3.c:51836
      #20 0x400ec80:0 in exec(sqlite3*,std::basic_string<char,std::char_traits<char>,std::allocator<char> >)+0x240 ()
      ---Type <return> to continue, or q <return> to quit---
      #21 0x401b820:0 in update(void*)+0x6d0 ()
      #22 0x60000000c013b6e0:0 in __pthread_bound_body+0x190 ()
      from /usr/lib/hpux32/libpthread.so.1


      The dbstat report is as follows:


      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      Lock conflict matrix:
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      Locks grouped by lockers:
      Locker Mode Count Status ----------------- Object ---------------
      1 dd=11 locks held 2 write locks 0 pid/thread 24281/1 flags 10 priority 100
      1 READ 1 HELD TEST:table00004 handle 2
      1 READ 1 HELD TEST:table00004 handle 0
      2 dd=10 locks held 1 write locks 0 pid/thread 24281/1 flags 10 priority 100
      2 READ 1 HELD TEST:table00004 handle 0
      3 dd= 9 locks held 0 write locks 0 pid/thread 24281/1 flags 0 priority 100
      4 dd= 8 locks held 0 write locks 0 pid/thread 24281/1 flags 0 priority 100
      7 dd= 7 locks held 2 write locks 0 pid/thread 24281/1 flags 10 priority 100
      7 READ 1 HELD TEST:table00004 handle 4
      7 READ 2 HELD TEST:table00004 handle 0
      a dd=-1 locks held 2 write locks 0 pid/thread 24281/1 flags 10 priority 100
      a READ 1 HELD TEST:table00004 handle 6
      a READ 1 HELD TEST:table00004 handle 0
      d dd=-1 locks held 2 write locks 0 pid/thread 24281/1 flags 10 priority 100
      d READ 1 HELD TEST:table00004 handle 8
      d READ 1 HELD TEST:table00004 handle 0
      e dd= 5 locks held 0 write locks 0 pid/thread 24281/4 flags 0 priority 100
      f dd= 3 locks held 0 write locks 0 pid/thread 24281/4 flags 0 priority 100
      80000009 dd= 6 locks held 0 write locks 0 pid/thread 24281/1 flags 8 priority 100
      80000012 dd= 4 locks held 0 write locks 0 pid/thread 24281/2 flags 8 priority 100
      80000019 dd= 2 locks held 0 write locks 0 pid/thread 24281/5 flags 8 priority 100
      80000021 dd= 1 locks held 0 write locks 0 pid/thread 24281/3 flags 8 priority 100
      80000028 dd= 0 locks held 0 write locks 0 pid/thread 24281/4 flags 8 priority 100
      800001de dd= 1 locks held 5 write locks 3 pid/thread 24281/3 flags 1 priority 100
      800001de WRITE 1 WAIT TEST:table00004 page 0
      800001de WRITE 1 HELD TEST:table00004 page 350
      800001de WAS_WRITE 2 HELD TEST:table00004 page 350
      800001de WAS_WRITE 1 HELD TEST:table00004 page 301
      800001de READ 100 HELD TEST:table00004 page 301
      800001de READ 5 HELD TEST:table00004 page 350
      800001e1 dd= 4 locks held 5 write locks 5 pid/thread 24281/2 flags 1 priority 100
      800001e1 WAS_WRITE 1 HELD TEST:table00004 page 351
      800001e1 WAS_WRITE 1 HELD TEST:table00004 page 9
      800001e1 WAS_WRITE 2 HELD TEST:table00004 page 690
      800001e1 WAS_WRITE 4 HELD TEST:table00004 page 0
      800001e1 WAS_WRITE 2 HELD TEST:table00004 page 594
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      Locks grouped by object:
      Locker Mode Count Status ----------------- Object ---------------
      800001e1 WAS_WRITE 2 HELD TEST:table00004 page 594

      a READ 1 HELD TEST:table00004 handle 6

      7 READ 1 HELD TEST:table00004 handle 4

      1 READ 1 HELD TEST:table00004 handle 2

      800001e1 WAS_WRITE 4 HELD TEST:table00004 page 0
      800001de WRITE 1 WAIT TEST:table00004 page 0

      1 READ 1 HELD TEST:table00004 handle 0
      2 READ 1 HELD TEST:table00004 handle 0
      7 READ 2 HELD TEST:table00004 handle 0
      a READ 1 HELD TEST:table00004 handle 0
      d READ 1 HELD TEST:table00004 handle 0

      d READ 1 HELD TEST:table00004 handle 8

      800001e1 WAS_WRITE 1 HELD TEST:table00004 page 9

      800001de READ 100 HELD TEST:table00004 page 301
      800001de WAS_WRITE 1 HELD TEST:table00004 page 301

      800001de READ 5 HELD TEST:table00004 page 350
      800001de WAS_WRITE 2 HELD TEST:table00004 page 350
      800001de WRITE 1 HELD TEST:table00004 page 350

      800001e1 WAS_WRITE 1 HELD TEST:table00004 page 351

      800001e1 WAS_WRITE 2 HELD TEST:table00004 page 690

      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


      Any help will be appreciated.