This discussion is archived
4 Replies Latest reply: Nov 8, 2012 10:03 AM by Bogdan Coman RSS

Looking for help to increase performance on a DB XML database.

971517 Newbie
Currently Being Moderated
I'll try to answer all the questions in the Performance Questionnaire from here.

1) I'm primarily concerned with insertion performance. The best I've seen so far is about 6000 inserts/per second. This is running inside a VMWare VM with 3 GB of RAM. The VM is set up with 2 CPUs each with 2 cores. The host machine has 8GB of RAM with a dual core 2.67 GHZ i7 (2 logical cores per CPU). The best performance I've seen is by running 2 threads of execution. A single thread only gets me about 2500 inserts per/second.
This is all within a very simple, isolate program. I'm trying to determine how to re-architect a more complicated system, but if I can't hope to hit 10k inserts per second with my sample, I don't see how it's possible to expand this out to something more complicated.

2) Versions: BDBXML version 2.5.26 no special patches or config options
3) BDB version 4.8.26, no special patches
4) 2.67 dual core, hyperthreaded intel i7 (4 logical processors)
5) Host: Windows 7 64-bit, Guest: RHEL5 64-bit
6) Underlying disk is a 320GB WesternDigital barricuda (SATA). It's a laptop harddrive, I believe it's only 5400 RPM. Although the VM does not have exclusive access to the drive, it is not the same drive as the Host sytem drive. (i.e. Windows runs off of the C drive, this is the D drive). The has a 60GB slice of this drive.
7) Drive is NTFS formatted for the host. Guest, ext3
8) Host 8gb, guest 3gb (total usage when running tests low, i.e. no swapping by guest or host)
9) not currently using any replication
10) Not using remote filesystem
11) db_cv_mutex=POSIX/pthreads/library/x86_64/gcc-assembly
12) Using the C++ API for DBXML, and the C API for BDB
using gcc/g++ version 4.1.2
13) not using app server or web server
14) flags to 'DB_ENV->open()': | DB_SYSTEM_MEM
          | DB_INIT_MPOOL
          | DB_INIT_LOCK
          | DB_INIT_LOG
          | DB_INIT_TXN
          | DB_RECOVER
          | DB_THREAD
other env flags explicitly set:
DB_LOG_IN_MEMORY 1
DB_LOG_ZERO 1
set_cachesize(env, 1, 0, 1) // 1GB cache in single block
DB_TXN_NOSYNC 1
DB_TXN_WRITE_NOSYNC 1

I am not using a DB_CONFIG file at this time.

15) For the container config:
transactional true
transactionsNotDurable true
containertype wholedoc
indexNodes Off
pagesize 4096


16) In my little test program, I have a single container.
16.1) flags are the same as listed above.
16.2) I've tried with an empty container, and one with documents already inside and haven't noticed much difference at this point. I'm running 1, 2, 3, or 4 threads, each inserting 10k documents in a loop. Each insert is a single transaction.
16.3) Wholedoc (tried both node & wholedoc, I believe wholedoc was slightly faster).
16.4) The best performance I've seen is with a smaller document that is about 500 bytes.
16.5) I'm not currently using any document data.

17)sample document:
<?xml version='1.0' encoding='UTF-8' standalone='no'?>
<Record xmlns='http://someurl.com/test' JID='UUID-f9032e9c-7e9a-4f2c-b40e-621b0e66c47f'>
<DataType>journal</DataType>
<RecordID>f9032e9c-7e9a-4f2c-b40e-621b0e66c47f</RecordID>
<Hostname>test.foo.com</Hostname>
<HostUUID>34c90268-57ba-4d4c-a602-bdb30251ec77</HostUUID>
<Timestamp>2011-11-10T04:09:55-05:00</Timestamp>
<ProcessID>0</ProcessID>
<User name='root'>0</User>
<SecurityLabel>unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</SecurityLabel>
</Record>

18. As mentioned, I'm looked to get at least 10k documents per second for insertion. Updates are much more infrequent, and can run slower. I am not doing any partial updates, or replacing documents. In the actual system, there are minor updates that happen to document metadata, but again, these can be slower.

19. I'm primarily concerned with insertion rate, not query.
20. Xquery samples are not applicable at the moment.
21. I am using transactions, no special flags aside from setting them all to 'not durable'
22. Log files are currently stored on the same disk as the database.
23. I'm not using AUTO_COMMIT
24. I don't believe there are any non-transactional operations
25. best performance from 2 threads doing insertions
26. The primary way I've been testing performance is by using the 'clock_gettime(CLOCK_REALTIME)' calls inside my test program. The test program spawns 1 or more threads, each thread inserts 10k documents. The main thread waits for all the threads to complete, then exits. I'm happy to send the source code for this program if that would be helpful.
27. As mentioned, I'm hoping to get at least 10k inserts per second.
28. db_stat outputs:
28.1 db_stat -c:
93 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
40 Number of lock object partitions
0 Number of current locks
166 Maximum number of locks at any one time
5 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
0 Number of current lockers
35 Maximum number of lockers at any one time
0 Number of current lock objects
95 Maximum number of lock objects at any one time
3 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
565631 Total number of locks requested
542450 Total number of locks released
0 Total number of locks upgraded
29 Total number of locks downgraded
22334 Lock requests not available due to conflicts, for which we waited
23181 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
784KB The size of the lock region
10098 The number of partition locks that required waiting (0%)
866 The maximum number of times any partition lock was waited for (0%)
6 The number of object queue operations that required waiting (0%)
7220 The number of locker allocations that required waiting (2%)
0 The number of region locks that required waiting (0%)
3 Maximum hash bucket length

====================
28.2 db_stat -l:
0x40988 Log magic number
16 Log version number
31KB 256B Log record cache size
0 Log file mode
10Mb Current log file size
0 Records entered into the log
0 Log bytes written
0 Log bytes written since last checkpoint
0 Total log file I/O writes
0 Total log file I/O writes due to overflow
0 Total log file flushes
7 Total log file I/O reads
1 Current log file number
28 Current log file offset
1 On-disk log file number
28 On-disk log file offset
0 Maximum commits in a log flush
0 Minimum commits in a log flush
160KB Log region size
0 The number of region locks that required waiting (0%)

======================
28.3 db_stat -m
1GB Total cache size
1 Number of caches
1 Maximum number of caches
1GB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
1127961 Requested pages found in the cache (99%)
3622 Requested pages not found in the cache
7590 Pages created in the cache
3622 Pages read into the cache
7663 Pages written from the cache to the backing file
0 Clean pages forced from the cache
0 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
11212 Current total page count
11212 Current clean page count
0 Current dirty page count
131071 Number of hash buckets used for page location
4096 Assumed page size used
1142798 Total number of times hash chains searched for a page
1 The longest hash chain searched for a page
1127988 Total number of hash chain entries checked for page
0 The number of hash bucket locks that required waiting (0%)
0 The maximum number of times any hash bucket lock was waited for (0%)
4 The number of region locks that required waiting (0%)
0 The number of buffers frozen
0 The number of buffers thawed
0 The number of frozen buffers freed
11218 The number of page allocations
0 The number of hash buckets examined during allocations
0 The maximum number of hash buckets examined for an allocation
0 The number of pages examined during allocations
0 The max number of pages examined for an allocation
0 Threads waited on page I/O
0 The number of times a sync is interrupted
Pool File: temp.dbxml
4096 Page size
0 Requested pages mapped into the process' address space
1127961 Requested pages found in the cache (99%)
3622 Requested pages not found in the cache
7590 Pages created in the cache
3622 Pages read into the cache
7663 Pages written from the cache to the backing file

=================================
28.4 db_stat -r (n/a, no replication)
28.5 db_stat -t
0/0 No checkpoint LSN
Tue Oct 30 15:05:29 2012 Checkpoint timestamp
0x8001d4d5 Last transaction ID allocated
100 Maximum number of active transactions configured
0 Active transactions
5 Maximum active transactions
120021 Number of transactions begun
0 Number of transactions aborted
120021 Number of transactions committed
0 Snapshot transactions
0 Maximum snapshot transactions
0 Number of transactions restored
48KB Transaction region size
1385 The number of region locks that required waiting (0%)
Active transactions:
  • 1. Re: Looking for help to increase performance on a DB XML database.
    971517 Newbie
    Currently Being Moderated
    Replying with output from iostat & vmstat (including the output exceeded the character count).
    =============================
    output of vm_stat while running 4 threads, inserting 10k documents each. It took just under 18 seconds to complete. I ran vmstat a few times while it was running:
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    3 0 0 896904 218004 1513268 0 0 14 30 261 83 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    5 0 0 889588 218004 1520500 0 0 14 30 261 84 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    2 0 0 882892 218012 1527124 0 0 14 30 261 84 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    4 0 0 896664 218012 1533284 0 0 14 30 261 85 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    5 0 0 890456 218012 1539748 0 0 14 30 261 85 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    2 0 0 884256 218020 1545800 0 0 14 30 261 86 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    4 0 0 878304 218020 1551520 0 0 14 30 261 86 1 1 98 0 0
    $ sudo vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    2 0 0 871980 218028 1558108 0 0 14 30 261 87 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    5 0 0 865780 218028 1563828 0 0 14 30 261 87 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    3 0 0 859332 218028 1570108 0 0 14 30 261 87 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    2 0 0 586756 218028 1572660 0 0 14 30 261 88 1 1 98 0 0
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- system -----cpu------
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    3 2 0 788032 218104 1634624 0 0 14 31 261 88 1 1 98 0 0
    ================================
    sda1 is mount on /boot
    sda2 is mounted on /
    sda3 is swap space
    output for iostat, same scenario, 4 threads inserting 10k documents each:

    $ iostat -x 1
    Linux 2.6.18-308.4.1.el5 (localhost.localdomain) 10/30/2012

    avg-cpu: %user %nice %system %iowait %steal %idle
    27.43 0.00 4.42 1.18 0.00 66.96

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 46.53 0.00 2.97 0.00 396.04 133.33 0.04 14.33 14.33 4.26
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 46.53 0.00 2.97 0.00 396.04 133.33 0.04 14.33 14.33 4.26
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    26.09 0.00 15.94 0.00 0.00 57.97

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    26.95 0.00 29.72 0.00 0.00 43.32

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    29.90 0.00 32.16 0.00 0.00 37.94

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    40.51 0.00 27.85 0.00 0.00 31.65

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    40.50 0.00 26.75 0.50 0.00 32.25

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 3.00 0.00 2.00 0.00 40.00 20.00 0.03 17.00 17.00 3.40
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 3.00 0.00 2.00 0.00 40.00 20.00 0.03 17.00 17.00 3.40
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    30.63 0.00 32.91 0.00 0.00 36.46

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    29.57 0.00 32.83 0.00 0.00 37.59

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    29.65 0.00 32.41 0.00 0.00 37.94

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    46.70 0.00 26.40 0.00 0.00 26.90

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    32.72 0.00 33.25 0.00 0.00 34.04

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 7.00 0.00 57.00 0.00 512.00 8.98 2.25 39.54 0.82 4.70
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 7.00 0.00 57.00 0.00 512.00 8.98 2.25 39.54 0.82 4.70
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    32.08 0.00 31.83 0.00 0.00 36.09

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    33.75 0.00 31.50 0.00 0.00 34.75

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    33.00 0.00 31.99 0.25 0.00 34.76

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 3.00 0.00 2.00 0.00 40.00 20.00 0.05 24.00 24.00 4.80
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 3.00 0.00 2.00 0.00 40.00 20.00 0.05 24.00 24.00 4.80
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    53.62 0.00 21.70 0.00 0.00 24.69

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    33.92 0.00 22.11 0.00 0.00 43.97

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    8.53 0.00 4.44 0.00 0.00 87.03

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    5.58 0.00 2.15 0.00 0.00 92.27

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    0.00 0.00 1.56 12.50 0.00 85.94

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 9.00 0.00 1.00 0.00 80.00 80.00 0.23 86.00 233.00 23.30
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 9.00 0.00 1.00 0.00 80.00 80.00 0.23 86.00 233.00 23.30
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    1.49 0.00 11.90 0.00 0.00 86.61

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.04 182.00 35.00 3.50
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.04 182.00 35.00 3.50
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    0.26 0.00 21.82 0.00 0.00 77.92

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    0.00 0.00 20.48 0.00 0.00 79.52

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    9.49 0.00 13.33 0.00 0.00 77.18

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    20.35 0.00 4.77 0.00 0.00 74.87

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    6.32 0.00 13.22 1.72 0.00 78.74

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 15302.97 0.99 161.39 7.92 34201.98 210.68 65.27 87.75 3.93 63.76
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 15302.97 0.99 161.39 7.92 34201.98 210.68 65.27 87.75 3.93 63.76
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

    avg-cpu: %user %nice %system %iowait %steal %idle
    1.83 0.00 5.49 1.22 0.00 91.46

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 21.00 0.00 95.00 0.00 91336.00 961.43 43.76 1003.00 7.18 68.20
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 21.00 0.00 95.00 0.00 91336.00 961.43 43.76 1003.00 7.18 68.20
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00


    ===================
  • 2. Re: Looking for help to increase performance on a DB XML database.
    Lucas Vogel Newbie
    Currently Being Moderated
    I'm not sure if there's much more that you can do with respect to configuration (though I'm going to try and explore that some more today). Some things to consider though:

    - If your transactions are non-durable, is possible you even need transactions at all? Is that something that you can take out for an additional gain?

    - Chances are pretty decent you're hitting a performance wall in the hardware space, especially if you're using a VM, a slow hard disk, and the same disk for both your database and transaction log. The VM part puts in a smaller dent than the hard disk speed.

    Is it possible to engineer your application to hit your target? For example, have you tried writing to multiple containers? Can you do all your writes to an in-memory container, and then have 1+ background job(s) writing container contents to disk?

    Regards,
    Lucas
  • 3. Re: Looking for help to increase performance on a DB XML database.
    Lucas Vogel Newbie
    Currently Being Moderated
    Also, have you tried using the DB_LOCKDOWN flag to see if that helps any?
  • 4. Re: Looking for help to increase performance on a DB XML database.
    Bogdan Coman Journeyer
    Currently Being Moderated
    Increase the database page size. Make it 32K or 64K instead of 4K.

    You can also configure different mutex interfaces when you build BDB and see which works best for your case.

    Selecting a page size: http://docs.oracle.com/cd/E17076_02/html/programmer_reference/general_am_conf.html#am_conf_pagesize
    Configuring Berkeley DB: http://docs.oracle.com/cd/E17076_02/html/installation/build_unix_conf.html

    Bogdan Coman

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points