14 Replies Latest reply: May 16, 2011 10:05 AM by 802907 RSS

    ns-slapd process is using 99% CPU.

    807573
      I'm using Sun Java Enterprise system 2005Q4 with solaris 10 . I synced my Directory Server with Windows Active Directory. now the problem is when I start the isw (Identity Syncronization for Windows) service prstat shows that ns-slapd process using 99% CPU and if i dont start isw service ns-slapd process use 0.3 CPU.

      My Connectors logs giving ldap error messages like unable to connect jds:389 and unable to connect actDir:389. but my users are getting no problem in authenticating through either Active directory and Java Directory Servers. can someone figure out what happening. Thanks.
        • 1. Re: ns-slapd process is using 99% CPU.
          807573
          There are a few angles to approach this from.

          If you have instrumentation on the system already for things like the number of current connections to the LDAP server and the number of operations per second, I would look those over. If you don't have the DS instrumented, you should mark that as a to-do item for future reference.

          From the ISW standpoint, it's possible that there is a known issue with ISW sending slapd CPU to the ceiling under some circumstance. So you might have a look at the release notes for ISW or maybe search for ISW-related bug reports on that. Maybe someone else here can comment on it, but ISW is not my forte.

          From the LDAP server side again, there are a few obvious things to look for:
          1) A massive influx of connections. The "currentconnections" attribute on the monitor object will show this as well as netstat.
          2) Unindexed searches. If these complete, they will have "notes=U" in the result line. Otherwise, you may need to find whatever searches are not returning and check the filters by eye to see if they are indexed. I'm not sure if 5.2 writes warnings about unindexed searches into the error log but you can check that too.

          If you have slapd CPU maxed out in real time, you can run prstat -L and get a listing of all the LWPs associated with the slapd process. If a few of these are consistently consuming a lot of CPU (say, more than 5%), note the LWP numbers and do a pstack of the slapd process. The stack trace will also show LWP numbers that can be correlated to the prstat output.

          Feel free to paste the prstat and pstack output into the thread and we can see what we make of it.
          • 2. Re: ns-slapd process is using 99% CPU.
            807573
            Sorry for late reply...............actually my problem was solved and it was a network issue. now again i counter the same situation but this time its not network problem............My RAM is 2 GB and only 350 MB is in use. Should i increase the JDS Cache Sizes ?

            here is the prstat and Pstack outputs. Please Help.......

            ------------------------------------------------------------------- PRSTAT --------------------------------------------------------------------------------
            PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
            740 root 209M 189M cpu0 59 0 3:43:27 98% ns-slapd/52
            3421 root 8448K 5032K sleep 59 0 0:00:00 0.2% sshd/1
            3453 root 4848K 4600K cpu1 59 0 0:00:00 0.1% prstat/1
            2687 root 98M 37M sleep 59 0 0:00:08 0.1% ns-httpd/69
            3110 root 152M 73M sleep 59 0 0:00:18 0.0% java/30

            ------------------------------------------------------------------- PSTACK --------------------------------------------------------------------------------

            740: ./ns-slapd -D /opt/DSServers/slapd-myjds -i /opt/DSServers/slapd-fej
            ----------------- lwp# 1 / thread# 1 --------------------
            ffffffff7f0cdd90 pollsys (ffffffff7ffff360, 31, ffffffff7ffff290, 0)
            ffffffff7f067d04 poll (ffffffff7ffff360, 31, 3e8, 0, 0, 0) + 88
            ffffffff7d134bd0 prpoll_with_poll (0, 0, ffffffff7ffff360, 0, ffffffff6f200070, 0) + 4f0
            ffffffff7ea2ff08 slapd_daemon (1006cb820, 0, ffffffff7eb53968, ffffffff7eb4c810, 0, 8) + 228
            0000000100019514 main (100000000, 0, 185, 1001be398, 1c1000, 1001c0a88) + c90
            0000000100010d3c _start (0, 0, 0, 0, 0, 0) + 17c
            ----------------- lwp# 2 / thread# 2 --------------------
            ffffffff7f0cdd90 pollsys (ffffffff71a1fc30, 0, ffffffff71a1fcf0, 0)
            ffffffff7f06cecc pselect (0, ffffffff71a1fc30, ffffffff7f1eb5a8, ffffffff7f1eb5a8, ffffffff71a1fcf0, 0) + 1f0
            ffffffff7f06d23c select (0, 0, 0, 0, 61a800, 30d4) + 6c
            ffffffff7ec95518 DS_Sleep (2710, 0, 1486fc, ffffffff71a1fe88, 1004c0070, 0) + 48
            ffffffff77137fb0 ???????? (1003e46f0, ffffffff7726eff0, ffffffffffffffff, 800, 958, 10027a250)
            ffffffff7d136a7c ptroot (1002601b0, 20000, 0, 0, 20000, ffffffff71a1ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            ----------------- lwp# 3 / thread# 3 --------------------
            ffffffff7f0cdd90 pollsys (ffffffff7181fc20, 0, ffffffff7181fce0, 0)
            ffffffff7f06cecc pselect (0, ffffffff7181fc20, ffffffff7f1eb5a8, ffffffff7f1eb5a8, ffffffff7181fce0, 0) + 1f0
            ffffffff7f06d23c select (0, 0, 0, 0, 0, 0) + 6c
            ffffffff7ec95518 DS_Sleep (186a0, 1400, 1590, ffffffff7edfa128, 1, 0) + 48
            ffffffff77138c9c ???????? (1003e46f0, 0, ffffffff7727f910, 4000, ffff86e9, 1003e943c)
            ffffffff7d136a7c ptroot (100260250, 20000, 0, 0, 20000, ffffffff7181ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            ----------------- lwp# 4 / thread# 4 --------------------
            ffffffff7f0cd030 lwp_park (0, ffffffff7171fb70, 0)
            ffffffff7f0c6854 cond_wait_queue (100606a98, 100244750, ffffffff7171fb70, 0, 0, 0) + 28
            ffffffff7f0c6d08 cond_wait_common (100606a98, 100244750, ffffffff7171fb70, 100244754, 16, 0) + 2d8
            ffffffff7f0c6e94 condtimedwait (100606a98, 100244750, ffffffff7171fda0, 3, ffffffff7a0e2030, 0) + 34
            ffffffff7f0c6f30 cond_timedwait (100606a98, 100244750, ffffffff7171fda0, c350, 0, 11b35c) + 14
            ffffffff7f0c6f70 pthread_cond_timedwait (100606a98, 100244750, ffffffff7171fda0, 4bd9224e, 4bd9224e, 2e427fb0) + c
            ffffffff7d12e460 PR_WaitCondVar (100606a90, bd7ae, 2e427fb0, 0, 100606a98, 3b9aca00) + 118
            ffffffff771384cc ???????? (1003e9378, 0, 4000, 0, ffffffff7726eff0, 2c00)
            ffffffff7d136a7c ptroot (1002602f0, 20000, 0, 0, 20000, ffffffff7171ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            ----------------- lwp# 5 / thread# 5 --------------------
            ffffffff7f0cdd90 pollsys (ffffffff7161fc20, 0, ffffffff7161fce0, 0)
            ffffffff7f06cecc pselect (0, ffffffff7161fc20, ffffffff7f1eb5a8, ffffffff7f1eb5a8, ffffffff7161fce0, 0) + 1f0
            ffffffff7f06d23c select (0, 0, 0, 0, 0, 0) + 6c
            ffffffff7ec95518 DS_Sleep (186a0, 2000, ffffffff7161fe88, ffffffff74b00000, 0, 100210370) + 48
            ffffffff77138e50 ???????? (1003e46f0, 800, bf8, 800, 958, 13630c)
            ffffffff7d136a7c ptroot (100260390, 20000, 0, 0, 20000, ffffffff7161ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            ----------------- lwp# 8 / thread# 8 --------------------
            ffffffff7f0cd030 lwp_park (0, ffffffff7130fb70, 0)
            ffffffff7f0c6854 cond_wait_queue (10060feb8, 1002908b8, ffffffff7130fb70, 0, 0, 0) + 28
            ffffffff7f0c6d08 cond_wait_common (10060feb8, 1002908b8, ffffffff7130fb70, 1002908bc, 16, 0) + 2d8
            ffffffff7f0c6e94 condtimedwait (10060feb8, 1002908b8, ffffffff7130fda0, 3, ffffffff71901800, 0) + 34
            ffffffff7f0c6f30 cond_timedwait (10060feb8, 1002908b8, ffffffff7130fda0, c350, 0, 11b35c) + 14
            ffffffff7f0c6f70 pthread_cond_timedwait (10060feb8, 1002908b8, ffffffff7130fda0, 4bd92243, 14, 0) + c
            ffffffff7d12ea48 PR_Wait (100260570, 0, d374d, 3b9ac800, 1002908b0, 8) + 138
            ffffffff768b2150 __1cXAsynchronousSaintLoggerDrun6M_v_ (100ba0070, ffffffff76a5b3c8, 100cdd670, ffffffff768b9430, 0, 0) + d8
            ffffffff7d136a7c ptroot (100260570, 10000, 0, 0, 10000, ffffffff7130ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            ----------------- lwp# 9 / thread# 9 --------------------
            ffffffff7f0cdd90 pollsys (ffffffff7120f6b0, 1, ffffffff7120f5e0, 0)
            ffffffff7f067d04 poll (ffffffff7120f6b0, 1, 1, f4240, 1, 3d09) + 88
            ffffffff7d134bd0 prpoll_with_poll (0, 0, ffffffff7120f6b0, 0, ffffffff7120f980, 4) + 4f0
            ffffffff768bf598 __1cJTCPSocketNpollForEvents6MkpnKPRPollDesc_kI_I_ (100cd97e8, ffffffff7120fa68, 1002a3c70, 6, 64, 296932c1) + 158
            ffffffff768bdf28 __1cJTCPSocketHconnect6MpkckHkI_I_ (100cd97e8, 100c80220, ffffffff76a5be80, 3938700, ffffffff768be0b0, fffffc00) + 2d0
            ffffffff768b8f34 __1cbBSubcomponentSaintConnectionEopen6MkpnRSubcomponentSCSIF_kikpknNUnicodeString_kH8888_I_ (100cdbbf0, ffffffff76a5be80, 3938400, ffffffff7120fcf0, ffffffff76a5c080, ffffffff768bdc58) + 4d0
            ffffffff768ae278 saint_OpenConnection (0, 1001f0b08, 2710, 1001f03b8, 0, ffffffff) + 258
            ffffffff768ade90 wpsync_openConnection (76a58000, 76a58000, ffffffff76a58000, ffffffff76a58230, ffffffff76a58200, fffffc00) + 144
            ffffffff768a8760 connectionManagerMain (106708158, 1, 7a120, 106708158, 10670a900, ffffffff76a54070) + cc
            ffffffff7d136a7c ptroot (100260610, 10000, 0, 0, 10000, ffffffff7120ff48) + d4
            ffffffff7f0ccf90 lwpstart (0, 0, 0, 0, 0, 0)
            • 3. Re: ns-slapd process is using 99% CPU.
              807573
              Unfortunately, you did not use the -L option on your prstat, so we cannot see which LWPs might be consuming your CPU. You also did not include the entirety of the pstack. Do you want to try again?

              Also, did you check for unindexed searches during the high CPU time period?
              • 4. Re: ns-slapd process is using 99% CPU.
                857612
                Hi,

                I am also facing similar issue. "ns-slapd" process taking high CPU utilization between 40-60% and some times it goes to 90-100%.

                Please help me to investigate on the cause

                pasted here the prstat output

                # prstat -a

                PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
                24161 root 356M 337M cpu1 59 0 141:57:00 69% ns-slapd/41

                2177 root 2354M 2253M sleep 59 0 91:52:36 0.1% java/94
                9472 patrol 11M 8352K sleep 59 0 0:37:52 0.0% bgscollect/1
                19304 root 3624K 3544K cpu21 19 0 0:00:00 0.0% prstat/1
                1291 root 137M 46M sleep 59 0 12:27:49 0.0% java/52

                # prstat -L

                PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
                24161 root 356M 336M cpu22 10 0 4:32:42 2.5% ns-slapd/36
                24161 root 356M 336M cpu13 10 0 4:35:09 2.5% ns-slapd/34
                24161 root 356M 336M cpu23 10 0 4:35:14 2.5% ns-slapd/35
                24161 root 356M 336M cpu15 10 0 4:33:54 2.5% ns-slapd/23
                24161 root 356M 336M run 0 0 4:34:38 2.5% ns-slapd/18
                24161 root 356M 336M cpu6 0 0 4:34:10 2.5% ns-slapd/21
                24161 root 356M 336M cpu29 0 0 4:37:13 2.5% ns-slapd/37
                24161 root 356M 336M cpu31 0 0 4:35:48 2.5% ns-slapd/33
                24161 root 356M 336M cpu1 10 0 4:35:49 2.5% ns-slapd/19
                24161 root 356M 336M cpu9 0 0 4:33:12 2.5% ns-slapd/13
                24161 root 356M 336M cpu30 10 0 4:34:56 2.5% ns-slapd/41
                24161 root 356M 336M cpu5 10 0 4:32:57 2.5% ns-slapd/15
                24161 root 356M 336M cpu19 10 0 4:34:37 2.5% ns-slapd/12
                24161 root 356M 336M cpu28 10 0 4:35:48 2.5% ns-slapd/40
                24161 root 356M 336M cpu17 0 0 4:35:05 2.5% ns-slapd/31
                24161 root 356M 336M cpu12 10 0 4:33:13 2.5% ns-slapd/30
                24161 root 356M 336M cpu27 0 0 4:37:13 2.5% ns-slapd/20
                24161 root 356M 336M cpu11 10 0 4:35:03 2.5% ns-slapd/17
                24161 root 356M 336M cpu0 10 0 4:33:22 2.5% ns-slapd/16
                24161 root 356M 336M cpu26 10 0 4:33:14 2.5% ns-slapd/38
                24161 root 356M 336M cpu3 10 0 4:33:37 2.5% ns-slapd/32
                24161 root 356M 336M cpu24 10 0 4:38:54 2.5% ns-slapd/29
                24161 root 356M 336M cpu21 10 0 4:32:35 2.5% ns-slapd/28
                24161 root 356M 336M cpu2 0 0 4:33:46 2.5% ns-slapd/26
                24161 root 356M 336M cpu16 0 0 4:33:26 2.5% ns-slapd/24
                24161 root 356M 336M cpu10 0 0 4:31:44 2.5% ns-slapd/39
                24161 root 356M 336M cpu7 10 0 4:35:50 2.5% ns-slapd/25
                24161 root 356M 336M cpu4 0 0 4:34:00 2.5% ns-slapd/14
                24161 root 356M 336M cpu14 10 0 4:33:50 2.4% ns-slapd/27
                24161 root 356M 336M cpu8 0 0 4:33:22 2.4% ns-slapd/22

                24161 root 356M 336M sleep 59 0 3:54:30 0.2% ns-slapd/1

                Edited by: 854609 on May 17, 2011 6:32 PM

                Edited by: 854609 on May 17, 2011 6:32 PM
                • 5. Re: ns-slapd process is using 99% CPU.
                  802907
                  I don't think you got all the LWPs from ns-slapd, but you got enough for it to be clear that the cpu utilization appears to be spread fairly evenly across most of your worker threads. The key information that's missing here is what those LWPs were doing at the time. That's what the pstack would tell us.

                  So for future reference, you can specify the pid of the ns-slapd process in the prstat. A pipeline like this works on Solaris if there is exactly one ns-slapd process running:

                  pgrep ns-slapd | xargs -I {} prstat -L -p {} 1 1 > prstat.`date +%H%M%S`;pgrep ns-slapd | xargs pstack > pstack.`date +%H%M%S`

                  You can run this a few times while the problem is occurring. The timestamps embedded in the file names should allow you to correlate the stack trace with the associated prstat output.
                  • 6. Re: ns-slapd process is using 99% CPU.
                    857612
                    Hi,

                    here are the outputs of two files.


                    # cat prstat.173107
                    PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
                    24161 root 397M 377M cpu31 0 0 17:16:51 2.9% ns-slapd/35
                    24161 root 397M 377M cpu23 0 0 17:07:07 2.9% ns-slapd/26
                    24161 root 397M 377M cpu20 0 0 17:18:05 2.9% ns-slapd/29
                    24161 root 397M 377M cpu10 0 0 17:12:12 2.9% ns-slapd/33
                    24161 root 397M 377M cpu13 0 0 17:08:09 2.9% ns-slapd/32
                    24161 root 397M 377M cpu25 0 0 17:10:09 2.9% ns-slapd/28
                    24161 root 397M 377M cpu9 0 0 17:08:57 2.9% ns-slapd/25
                    24161 root 397M 377M cpu12 0 0 17:08:59 2.9% ns-slapd/36
                    24161 root 397M 377M cpu29 0 0 17:13:21 2.9% ns-slapd/24
                    24161 root 397M 377M cpu6 0 0 17:15:52 2.9% ns-slapd/27
                    24161 root 397M 377M cpu28 0 0 17:12:21 2.9% ns-slapd/22
                    24161 root 397M 377M cpu14 0 0 17:10:39 2.9% ns-slapd/16
                    24161 root 397M 377M cpu11 0 0 17:19:06 2.9% ns-slapd/37
                    24161 root 397M 377M cpu19 0 0 17:12:50 2.9% ns-slapd/38
                    24161 root 397M 377M cpu4 0 0 17:09:15 2.9% ns-slapd/14
                    Total: 1 processes, 41 lwps, load averages: 21.89, 10.31, 8.31

                    # cat pstack.173107
                    24161: ./ns-slapd -D /var/Sun/mps/slapd-ldap-imfs-consum6 -i /var/Sun/mps/sla
                    ----------------- lwp# 1 / thread# 1 --------------------
                    ffffffff7f1d35b4 sosetsockopt (1002a3970, ffffffff7ffff2b8, ffffffff7d63311c, 100, ffffffffffffff78, ffffffff71e00080) + 8
                    ffffffff7ec33160 configure_pr_socket (ffffffff7ffff4e8, 1, ffffffff7ffff478, 70, ffffffff7ed4c810, 1) + e8
                    ffffffff7ec322e8 ???????? (10043ae10, 100561618, 0, 1002a03a0, 1002a03a0, ffffffff71e00080)
                    ffffffff7ec3008c slapd_daemon (100698200, 100561618, ffffffff7ed53968, ffffffff7ed4c810, 0, 8) + 3ac
                    0000000100019514 main (100000000, 0, 185, 1001be398, 1c1000, 1001c0a88) + c90
                    0000000100010d3c _start (0, 0, 0, 0, 0, 0) + 17c
                    ----------------- lwp# 2 / thread# 2 --------------------
                    ffffffff7f1d3c7c pollsys (ffffffff77b1fc30, 0, ffffffff77b1fcf0, 0)
                    ffffffff7f172e20 pselect (0, ffffffff77b1fc30, ffffffff7f2f45d0, ffffffff7f2f45d0, ffffffff77b1fcf0, 0) + 1f0
                    ffffffff7f1731c4 select (0, 0, 0, 0, ffffffff77b1fdb0, 30d4) + a0
                    ffffffff7ee95518 DS_Sleep (2710, 0, 1486fc, ffffffff77b1fe88, 1004a0250, 0) + 48
                    ffffffff77c37fb0 ???????? (1003e7568, ffffffff77d6eff0, ffffffffffffffff, 800, 958, 100276470)
                    ffffffff7d636a7c ptroot (1002601b0, 20000, 0, 0, 20000, ffffffff77b1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 3 / thread# 3 --------------------
                    ffffffff7f1d3c7c pollsys (ffffffff7591fc20, 0, ffffffff7591fce0, 0)
                    ffffffff7f172e20 pselect (0, ffffffff7591fc20, ffffffff7f2f45d0, ffffffff7f2f45d0, ffffffff7591fce0, 0) + 1f0
                    ffffffff7f1731c4 select (0, 0, 0, 0, ffffffff7591fda0, 0) + a0
                    ffffffff7ee95518 DS_Sleep (186a0, 1400, 1590, ffffffff7effa128, 1, 0) + 48
                    ffffffff77c38c9c ???????? (1003e7568, 0, ffffffff77d7f910, 4000, ffff86e9, 1003e2bf4)
                    ffffffff7d636a7c ptroot (100260250, 20000, 0, 0, 20000, ffffffff7591ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 4 / thread# 4 --------------------
                    ffffffff7f1d2f1c lwp_park (0, ffffffff73b1fb70, 0)
                    ffffffff7f1cc748 cond_wait_queue (10041aeb8, 100244750, ffffffff73b1fb70, 0, 0, 0) + 28
                    ffffffff7f1ccbac cond_wait_common (10041aeb8, 100244750, ffffffff73b1fb70, 100244754, 16, 0) + 2d4
                    ffffffff7f1ccd38 condtimedwait (10041aeb8, 100244750, ffffffff73b1fda0, 3, 4db933dc, 0) + 34
                    ffffffff7f1ccdd4 cond_timedwait (10041aeb8, 100244750, ffffffff73b1fda0, 0, 10041aeb8, 3b9aca00) + 14
                    ffffffff7f1cce14 pthread_cond_timedwait (10041aeb8, 100244750, ffffffff73b1fda0, 4db933dc, 4db933dc, 12cd2410) + c
                    ffffffff7d62e460 PR_WaitCondVar (10041aeb0, 4d02a, 12cd2410, 0, 10041aeb8, 3b9aca00) + 118
                    ffffffff77c384cc ???????? (1003e2b30, 0, 4000, 0, ffffffff77d6eff0, 2c00)
                    ffffffff7d636a7c ptroot (1002602f0, 20000, 0, 0, 20000, ffffffff73b1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 5 / thread# 5 --------------------
                    ffffffff7f1d3c7c pollsys (ffffffff73a1fc20, 0, ffffffff73a1fce0, 0)
                    ffffffff7f172e20 pselect (0, ffffffff73a1fc20, ffffffff7f2f45d0, ffffffff7f2f45d0, ffffffff73a1fce0, 0) + 1f0
                    ffffffff7f1731c4 select (0, 0, 0, 0, ffffffff73a1fda0, 0) + a0
                    ffffffff7ee95518 DS_Sleep (186a0, 2000, ffffffff73a1fe88, ffffffff78b00000, 0, 100210530) + 48
                    ffffffff77c38e50 ???????? (1003e7568, 800, bf8, 800, 958, 13630c)
                    ffffffff7d636a7c ptroot (100260390, 20000, 0, 0, 20000, ffffffff73a1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 6 / thread# 6 --------------------
                    ffffffff7f1d3c7c pollsys (ffffffff7391f9a0, 0, ffffffff7391fa60, 0)
                    ffffffff7f172e20 pselect (0, ffffffff7391f9a0, ffffffff7f2f45d0, ffffffff7f2f45d0, ffffffff7391fa60, 0) + 1f0
                    ffffffff7f1731c4 select (0, 0, 0, 0, ffffffff7391fb20, 0) + a0
                    ffffffff7ee95518 DS_Sleep (30d40, 0, 2400, 5400, 54b0, 0) + 48
                    ffffffff77c3cb1c ???????? (1003e7568, 1003e2b30, 1d4c, 0, 400, 0)
                    ffffffff7d636a7c ptroot (100260430, 20000, 0, 0, 20000, ffffffff7391ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 7 / thread# 7 --------------------
                    ffffffff7f1d3c7c pollsys (ffffffff7381fc50, 0, ffffffff7381fd10, 0)
                    ffffffff7f172e20 pselect (0, ffffffff7381fc50, ffffffff7f2f45d0, ffffffff7f2f45d0, ffffffff7381fd10, 0) + 1f0
                    ffffffff7f1731c4 select (0, 0, 0, 0, ffffffff7381fdd0, 0) + a0
                    ffffffff7ee95518 DS_Sleep (186a0, 1002c7590, 1007176a8, 4, 20000, ffffffff7381ff48) + 48
                    ffffffff77c37dfc ???????? (1003e7568, 0, ffffffffffffffff, 800, 958, 100276650)
                    ffffffff7d636a7c ptroot (1002604d0, 20000, 0, 0, 20000, ffffffff7381ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 8 / thread# 8 --------------------
                    ffffffff7f1d2f1c lwp_park (0, 0, 0)
                    ffffffff7f1cc748 cond_wait_queue (100255198, 1004dfd90, 0, 0, ffffffff7f2f9d08, 0) + 28
                    ffffffff7f1ccca8 cond_wait (100255198, 1004dfd90, 0, 800, 400, 4c0) + 10
                    ffffffff7f1ccce4 pthread_cond_wait (100255198, 1004dfd90, ffffffffffffffff, 1009e0070, 0, 1009e0070) + 8
                    ffffffff7d62e3b4 PR_WaitCondVar (100255190, ffffffff, 0, 800, 400, 4c0) + 6c
                    ffffffff7ee89ff4 slapi_wait_condvar (100255190, 0, 1, 1009e0070, 0, 1009e0070) + 58
                    ffffffff77e04a64 ???????? (0, 10752c, ffffffffffffffff, 800, 400, 4c0)
                    ffffffff7d636a7c ptroot (100260570, 20000, 0, 0, 20000, ffffffff7371ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 10 / thread# 10 --------------------
                    ffffffff7f1d2f1c lwp_park (0, ffffffff7351fba0, 0)
                    ffffffff7f1cc748 cond_wait_queue (1002554d8, 1004dcb80, ffffffff7351fba0, 0, 0, 0) + 28
                    ffffffff7f1ccbac cond_wait_common (1002554d8, 1004dcb80, ffffffff7351fba0, 1004dcb84, 16, 0) + 2d4
                    ffffffff7f1ccd38 condtimedwait (1002554d8, 1004dcb80, ffffffff7351fdd0, 3, 4db933e4, 0) + 34
                    ffffffff7f1ccdd4 cond_timedwait (1002554d8, 1004dcb80, ffffffff7351fdd0, 0, 1002554d8, 3b9aca00) + 14
                    ffffffff7f1cce14 pthread_cond_timedwait (1002554d8, 1004dcb80, ffffffff7351fdd0, 4db933e4, 4db933e4, 2bd8b370) + c
                    ffffffff7d62e460 PR_WaitCondVar (1002554d0, b3986, 2bd8b370, 0, 1002554d8, 3b9aca00) + 118
                    ffffffff7eea2ba4 ???????? (a, 0, ffffffff7effb918, 1003492b0, f4240, 1)
                    ffffffff7d636a7c ptroot (1002606b0, 20000, 0, 0, 20000, ffffffff7351ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 12 / thread# 12 --------------------
                    ffffffff7f1d3c80 __pollsys (ffffffff7341e020, 1, ffffffff7341df50, 0, 0, 26c) + 8
                    ffffffff7f16dc58 poll (ffffffff7341e020, 1, 1388, 0, 0, 0) + 88
                    ffffffff7d634bd0 prpoll_with_poll (0, 0, ffffffff7341e020, 0, ffffffff7341e2f8, 4) + 4f0
                    ffffffff7ec310b8 ???????? (1002a33a0, 2, 7a120, 1, 11b7cc, 1)
                    ffffffff7ec316dc pr_secure_write_function (1002a33a0, 11395f3c8, ee, 0, 0, ffffffff7ec3bb04) + 5c
                    ffffffff7e030458 ber_flush (1008d1500, 11395f190, 1, 11395f190, 0, 1025d8) + 150
                    ffffffff7ec3bb28 ldap_getBytes_and_flush (ffffffff72008a08, 11395f190, 77, 0, 0, ffffffff7ec3bb04) + 24
                    ffffffff7ee8f894 ???????? (1139594f8, ffffffff72008a08, 11395f7d0, 11395f190, 2, ffffffff7ec3b9d8)
                    ffffffff7ee8f644 send_ldap_search_entry_ext (1139594f8, 115092bc0, 3, 0, 0, ffffffff7f00b878) + 774
                    ffffffff7ee8e5a0 send_ldap_search_entry (1139594f8, 115092bc0, 0, 0, 0, 0) + 24
                    ffffffff7eeaf3d8 ???????? (1139594f8, 1003c1828, 1, ffffffff7341eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (1139594f8, 1, ffffffff7341eaf4, 0, 1154b2488, ffffffff7341e980)
                    ffffffff7eeaede8 op_shared_search (1139594f8, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (1139594f8, 1, 1000, 4000, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (1139594f8, 1139594f8, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72008a08, 11395f7d0, 2, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72008a08, 11395f7d0, 2, 100357e18, 10033b948, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72008b30)
                    ffffffff7d636a7c ptroot (100260750, 20000, 0, 0, 20000, ffffffff7341ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 13 / thread# 13 --------------------
                    ffffffff7eeb85d4 ???????? (100310428, 53f31a1796, ffffffff7331d2e0, 14fcc685e58, 53f31a15fe8, 14fcc6857fa)
                    ffffffff7eeb8644 PL_HashTableLookup_const (100310428, ffffffff7331d2e0, 0, 0, 0, ffffffff7f2ec000) + 18
                    ffffffff7eeacc48 ???????? (106e86f00, ffffffff7331d4b0, ffffffff7331d4c0, ffffffff7331d4bc, 0, ffffffff7efe9d08)
                    ffffffff7ee715f8 ???????? (0, ffffffff7331d698, ffffffff7331d598, ffffffff7331d590, 0, 1001bc5e0)
                    ffffffff7ee71a00 ???????? (1128ab1b8, ffffffff7331d698, ffffffff7ee71568, 0, 1001bc5e0, 0)
                    ffffffff7ee73138 ???????? (ffffffff7331dc58, ffffffff7331d698, ffffffff7efe9d08, 0, ffffffff7331d8e8, 1001bc5e0)
                    ffffffff7ee7363c slapi_dn_strict_normalize (ffffffff7331dc58, 114482258, 0, ffffffff7331deb8, ffffffffffffffff, 4000) + 3c
                    ffffffff7ee7366c slapi_dn_strict_normalize_case (ffffffff7331dc58, 1, ffffffff7eff9fa0, 0, 63, 11448228a) + 4
                    ffffffff7ee70940 slapi_sdn_norm_dn (1126164a0, 114482258, 0, ffffffff7331deb8, ffffffffffffffff, 4000) + 30
                    ffffffff7ee709e4 slapi_sdn_get_ndn (1126164a0, 1a49, 0, ffffffff7331e7f4, 0, ffffffff77d6eff0) + 34
                    ffffffff77c241f4 ???????? (1006883b8, 107fb3670, 0, ffffffff7331deb8, ffffffffffffffff, 4000)
                    ffffffff77c25d70 id2entry (1003c1828, 1a49, 0, ffffffff7331e7f4, 0, ffffffff77d6eff0) + 420
                    ffffffff77c5e478 ldbm_back_next_search_entry_ext (105377558, 1144808f8, 0, 115a948d0, 22, 4db933f9) + 300
                    ffffffff7eeaf370 ???????? (105377558, 1003c1828, 1, ffffffff7331eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (105377558, 1, ffffffff7331eaf4, 0, 1144808f8, ffffffff7331e980)
                    ffffffff7eeaede8 op_shared_search (105377558, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (105377558, 114f707f0, 114109480, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (105377558, 105377558, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72007c28, 10537f7d0, 93043, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72007c28, 10537f7d0, 2, 100357b78, 10033f2c8, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72005bc8)
                    ffffffff7d636a7c ptroot (1002607f0, 20000, 0, 0, 20000, ffffffff7331ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 14 / thread# 14 --------------------
                    ffffffff7f1d3c80 __pollsys (ffffffff71d1e020, 1, ffffffff71d1df50, 0, 0, 26c) + 8
                    ffffffff7f16dc58 poll (ffffffff71d1e020, 1, 1388, 0, 0, 0) + 88
                    ffffffff7d634bd0 prpoll_with_poll (0, 0, ffffffff71d1e020, 0, ffffffff71d1e2f8, 4) + 4f0
                    ffffffff7ec310b8 ???????? (1002a2440, 2, 7a120, 1, 11b7cc, 1)
                    ffffffff7ec316dc pr_secure_write_function (1002a2440, 10f9bfbe8, ee, 0, 0, ffffffff7ec3bb04) + 5c
                    ffffffff7e030458 ber_flush (1008d0bd0, 10f9bf9b0, 1, 10f9bf9b0, 0, 1025d8) + 150
                    ffffffff7ec3bb28 ldap_getBytes_and_flush (ffffffff72008d80, 10f9bf9b0, 7a, 0, 0, ffffffff7ec3bb04) + 24
                    ffffffff7ee8f894 ???????? (10f9b8408, ffffffff72008d80, 10f9b6698, 10f9bf9b0, 2, ffffffff7ec3b9d8)
                    ffffffff7ee8f644 send_ldap_search_entry_ext (10f9b8408, 115e13100, 3, 0, 0, ffffffff7f00b878) + 774
                    ffffffff7ee8e5a0 send_ldap_search_entry (10f9b8408, 115e13100, 0, 0, 0, 0) + 24
                    ffffffff7eeaf3d8 ???????? (10f9b8408, 1003c1828, 1, ffffffff71d1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (10f9b8408, 1, ffffffff71d1eaf4, 0, 114494de0, ffffffff71d1e980)
                    ffffffff7eeaede8 op_shared_search (10f9b8408, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (10f9b8408, 1, 1000, 4000, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (10f9b8408, 10f9b8408, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72008d80, 10f9b6698, 2, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72008d80, 10f9b6698, 2, 100357ec0, 10033b948, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72009598)
                    ffffffff7d636a7c ptroot (100260890, 20000, 0, 0, 20000, ffffffff71d1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 15 / thread# 15 --------------------
                    ffffffff7f407c9c shi_allocSmall2 (0, ffffffff7f510f80, 18, 1062d0020, ffe0, 1062d03e8) + d4
                    ffffffff7f407710 MemAllocPtr (100200020, 18, 0, 0, 5400, 5688) + 68
                    ffffffff7ee54d38 slapi_ch_malloc (18, a08, ffffffff7f01b1d8, 2000, ffffffff7f01d1d8, 0) + e0
                    ffffffff7ee6dc94 ???????? (0, 1, 0, ffffffff71c1dca4, b40, b10)
                    ffffffff7ee572d4 ???????? (112f929b0, 110952a70, 0, 0, ffffffff7efe9d08, 0)
                    ffffffff7ee5899c slapi_str2entry (11240a330, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                    ffffffff77c25d20 id2entry (1003c1828, 1a62, 0, ffffffff71c1e7f4, 0, ffffffff77d6eff0) + 3d0
                    ffffffff77c5e478 ldbm_back_next_search_entry_ext (1124053c0, 114940508, 0, 115b83430, 3b, 4db933f9) + 300
                    ffffffff7eeaf370 ???????? (1124053c0, 1003c1828, 1, ffffffff71c1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (1124053c0, 1, ffffffff71c1eaf4, 0, 114940508, ffffffff71c1e980)
                    ffffffff7eeaede8 op_shared_search (1124053c0, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (1124053c0, 112f80070, 1062fff00, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (1124053c0, 1124053c0, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72004de8, 11240f7d0, 23775, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72004de8, 11240f7d0, 2, 100354ea0, 10033b9c0, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff720080c8)
                    ffffffff7d636a7c ptroot (100260930, 20000, 0, 0, 20000, ffffffff71c1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 16 / thread# 16 --------------------
                    ffffffff7f13b6c0 strlen (70, 100642920, 0, 800, 2, b10) + a0
                    ffffffff7ee6cb4c slapi_attr_init (110e76830, 1140bc480, 0, 110e767d0, 110e767e0, 115c70d50) + 3c
                    ffffffff7eeb89c4 attrlist_find_or_create (115c70d50, 1140bc480, ffffffff71b1dc88, 800, 2, b10) + a8
                    ffffffff7ee57360 ???????? (0, 115c70d20, 0, 1, ffffffff7efe9d08, 0)
                    ffffffff7ee5899c slapi_str2entry (1140bb6f8, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                    ffffffff77c25d20 id2entry (1003c1828, 1a49, 0, ffffffff71b1e7f4, 0, ffffffff77d6eff0) + 3d0
                    ffffffff77c5e478 ldbm_back_next_search_entry_ext (1140bfd98, 115e71b58, 0, 115a948d0, 22, 4db933f9) + 300
                    ffffffff7eeaf370 ???????? (1140bfd98, 1003c1828, 1, ffffffff71b1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (1140bfd98, 1, ffffffff71b1eaf4, 0, 115e71b58, ffffffff71b1e980)
                    ffffffff7eeaede8 op_shared_search (1140bfd98, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (1140bfd98, 1151301f0, 108bee450, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (1140bfd98, 1140bfd98, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72007098, 1140b9b70, c6710, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72007098, 1140b9b70, 2, 100355cd8, 10033f1d8, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72006630)
                    ffffffff7d636a7c ptroot (1002609d0, 20000, 0, 0, 20000, ffffffff71b1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 17 / thread# 17 --------------------
                    ffffffff7ee728c4 ???????? (114685d68, ffffffff71a1da10, 0, ffffffff71a1deb8, 0, 2)
                    ffffffff7ee72f80 ???????? (ffffffff71a1dc58, 0, ffffffff7efe9d08, ffffffff7ee71568, 0, 1001bc5e0)
                    ffffffff7ee7363c slapi_dn_strict_normalize (ffffffff71a1dc58, 114683918, 0, ffffffff71a1deb8, ffffffffffffffff, 4000) + 3c
                    ffffffff7ee7366c slapi_dn_strict_normalize_case (ffffffff71a1dc58, 1, ffffffff7eff9fa0, 0, 63, 114683949) + 4
                    ffffffff7ee70940 slapi_sdn_norm_dn (113a95e10, 114683918, 0, ffffffff71a1deb8, ffffffffffffffff, 4000) + 30
                    ffffffff7ee709e4 slapi_sdn_get_ndn (113a95e10, 1a46, 0, ffffffff71a1e7f4, 0, ffffffff77d6eff0) + 34
                    ffffffff77c241f4 ???????? (1006883b8, 115471bd0, 0, ffffffff71a1deb8, ffffffffffffffff, 4000)
                    ffffffff77c25d70 id2entry (1003c1828, 1a46, 0, ffffffff71a1e7f4, 0, ffffffff77d6eff0) + 420
                    ffffffff77c5e478 ldbm_back_next_search_entry_ext (10fae4368, 114684558, 0, 112826a30, 1f, 4db933f9) + 300
                    ffffffff7eeaf370 ???????? (10fae4368, 1003c1828, 1, ffffffff71a1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                    ffffffff7eeaf83c ???????? (10fae4368, 1, ffffffff71a1eaf4, 0, 114684558, ffffffff71a1e980)
                    ffffffff7eeaede8 op_shared_search (10fae4368, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                    ffffffff7ec2efcc search_core_pb (10fae4368, 115340070, 106fa0a90, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                    ffffffff7ec3a710 dispatch_operation_core_pb (10fae4368, 10fae4368, c00, ffffffff7ed4c810, 0, d10) + cc
                    ffffffff7ec26420 ???????? (ffffffff72006d20, 10faed7c8, 82e6f, 63, 2, 200000)
                    ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72006d20, 10faed7c8, 2, 100355c30, 10033d248, 0) + fc
                    ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72008568)
                    ffffffff7d636a7c ptroot (100260a70, 20000, 0, 0, 20000, ffffffff71a1ff48) + d4
                    ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                    ----------------- lwp# 18 / thread# 18 --------------------
                    • 7. Re: ns-slapd process is using 99% CPU.
                      857612
                      ----------------- lwp# 18 / thread# 18 --------------------
                      ffffffff7d1006f4 memcpy (ffffffff7191d680, 1071860c0, 5, 10c02fa48, ff0000, 80808080) + 1ac
                      ffffffff7ee73240 ???????? (ffffffff7191dc58, 1698, ffffffff7efe9d08, 1, ffffffff7191d8e8, 1001bc5e0)
                      ffffffff7ee7363c slapi_dn_strict_normalize (ffffffff7191dc58, 114c11458, 0, ffffffff7191deb8, ffffffffffffffff, 4000) + 3c
                      ffffffff7ee7366c slapi_dn_strict_normalize_case (ffffffff7191dc58, 1, ffffffff7eff9fa0, 0, 63, 114c1148a) + 4
                      ffffffff7ee70940 slapi_sdn_norm_dn (106b42f40, 114c11458, 0, ffffffff7191deb8, ffffffffffffffff, 4000) + 30
                      ffffffff7ee709e4 slapi_sdn_get_ndn (106b42f40, 1a49, 0, ffffffff7191e7f4, 0, ffffffff77d6eff0) + 34
                      ffffffff77c241f4 ???????? (1006883b8, 113fa3a90, 0, ffffffff7191deb8, ffffffffffffffff, 4000)
                      ffffffff77c25d70 id2entry (1003c1828, 1a49, 0, ffffffff7191e7f4, 0, ffffffff77d6eff0) + 420
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10d4cd570, 114c10fc0, 0, 115a948d0, 22, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10d4cd570, 1003c1828, 1, ffffffff7191eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10d4cd570, 1, ffffffff7191eaf4, 0, 114c10fc0, ffffffff7191e980)
                      ffffffff7eeaede8 op_shared_search (10d4cd570, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10d4cd570, 115bd03f0, 1071818f0, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10d4cd570, 10d4cd570, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72008318, 10d4cf7d0, 242a7, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72008318, 10d4cf7d0, 2, 100357cc8, 10033b9c0, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff720081f0)
                      ffffffff7d636a7c ptroot (100260b10, 20000, 0, 0, 20000, ffffffff7191ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 19 / thread# 19 --------------------
                      ffffffff7ee51164 slapi_is_loglevel_set (10d3002b0, ffffffff783a3b60, 13d1b8, ffffffff782471d0, 0, 800) + c
                      ffffffff782471dc ???????? (1008ccbd8, 10d3002b0, 10b490130, 2, 0, ffffffff783a1770)
                      ffffffff78245124 acl_access_allowed (1008cd980, 110951ea0, 1149807b0, 2, 1006cedd0, 1008ccbd8) + 874
                      ffffffff78253478 acl_access_allowed_main (1083f6490, 110951ea0, ffffffff7181dbb0, 116000690, 2, 0) + e8
                      ffffffff7eeb5e68 plugin_call_acl_plugin (1083f6490, 110951ea0, ffffffff7181dbb0, 116000690, 2, 0) + a4
                      ffffffff7ee7a390 ???????? (1083f6490, 110951ea0, 1149807b0, 116000690, ffffffff7f003ff8, 0)
                      ffffffff7ee79c2c slapi_vattr_filter_test_ext (1083f6490, 110951ea0, 116000670, 1, 0, a0) + 20c
                      ffffffff7ee7a4e8 ???????? (1083f6490, 110951ea0, 0, 1001c5390, 1, 0)
                      ffffffff7ee7a0a4 slapi_vattr_filter_test_ext (1083f6490, 110951ea0, 1160005f0, 1, 0, 4db933f9) + 684
                      ffffffff7ee799e0 slapi_vattr_filter_test (1083f6490, 110951ea0, 1160005f0, 1, 26f8, 115e246e0) + 14
                      ffffffff77c5e8f4 ldbm_back_next_search_entry_ext (1083f6490, 115e246e0, 0, 112860910, 22, 4db933f9) + 77c
                      ffffffff7eeaf370 ???????? (1083f6490, 1003c1828, 1, ffffffff7181eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (1083f6490, 1, ffffffff7181eaf4, 0, 115e246e0, ffffffff7181e980)
                      ffffffff7eeaede8 op_shared_search (1083f6490, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (1083f6490, 116000570, 1149820c0, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (1083f6490, 1083f6490, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72006190, 1083ff7d0, 9f51b, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72006190, 1083ff7d0, 2, 100354bc8, 10033f1d8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72007538)
                      ffffffff7d636a7c ptroot (100260bb0, 20000, 0, 0, 20000, ffffffff7181ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 20 / thread# 20 --------------------
                      ffffffff7eed3120 ???????? (10c6e43f0, 10f743f00, 0, 10c6e4480, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (10472b298, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a49, 0, ffffffff7171e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10472c080, 114a23870, 0, 115a948d0, 22, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10472c080, 1003c1828, 1, ffffffff7171eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10472c080, 1, ffffffff7171eaf4, 0, 114a23870, ffffffff7171e980)
                      ffffffff7eeaede8 op_shared_search (10472c080, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10472c080, 115d301f0, 10d2bb710, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10472c080, 10472c080, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff720079d8, 104727750, a047b, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff720079d8, 104727750, 2, 100355e28, 10033f2c8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72006f70)
                      ffffffff7d636a7c ptroot (100260c50, 20000, 0, 0, 20000, ffffffff7171ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 21 / thread# 21 --------------------
                      ffffffff7f2f1d24 ???????? (14, ffffffff7eee7bc0, ffffffff7161d6b0, 3, 75, 76)
                      ffffffff7f1b88dc sscanf (111d15f23, ffffffff7eee7bc0, ffffffff7161d6d2, ffffffff7161dca4, ffffffff7161dca0, 0) + 1c
                      ffffffff7eeb78c4 csn_init_by_string (ffffffff7161d790, 111d15f13, 75, 76, 75, 76) + 94
                      ffffffff7ee55a3c ???????? (76, ffffffff7161dcf0, ffffffff7161dcf8, ffffffff7161dca4, ffffffff7161dca0, 0)
                      ffffffff7ee57040 ???????? (112d97aa0, 1141a0e70, 0, 0, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (111d15408, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a49, 0, ffffffff7161e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (111d19aa8, 113dc0150, 0, 115a948d0, 22, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (111d19aa8, 1003c1828, 1, ffffffff7161eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (111d19aa8, 1, ffffffff7161eaf4, 0, 113dc0150, ffffffff7161e980)
                      ffffffff7eeaede8 op_shared_search (111d19aa8, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (111d19aa8, 113aa0770, 10602c3b0, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (111d19aa8, 111d19aa8, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72005160, 111d1f7d0, 23c87, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72005160, 111d1f7d0, 2, 100354e68, 10033f6b0, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff720087b8)
                      ffffffff7d636a7c ptroot (100260cf0, 20000, 0, 0, 20000, ffffffff7161ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 22 / thread# 22 --------------------
                      ffffffff7f1d2f1c lwp_park (0, 0, 0)
                      ffffffff7d62e0cc PR_Lock (1004dd090, 1a5c, 0, ffffffff7151e7f4, 0, ffffffff77d6eff0) + 4
                      ffffffff77c24230 ???????? (1006883b8, 10f9110f0, 0, ffffffff7151deb8, ffffffffffffffff, 4000)
                      ffffffff77c25d70 id2entry (1003c1828, 1a5c, 0, ffffffff7151e7f4, 0, ffffffff77d6eff0) + 420
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10fd9fd98, 107f94328, 0, 112823f10, 35, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10fd9fd98, 1003c1828, 1, ffffffff7151eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10fd9fd98, 1, ffffffff7151eaf4, 0, 107f94328, ffffffff7151e980)
                      ffffffff7eeaede8 op_shared_search (10fd9fd98, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10fd9fd98, 1108702f0, 107ecbad0, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10fd9fd98, 10fd9fd98, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72007538, 10fd99b10, 7901f, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72007538, 10fd99b10, 2, 100355e98, 10033b9c0, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72004a70)
                      ffffffff7d636a7c ptroot (100260d90, 20000, 0, 0, 20000, ffffffff7151ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 23 / thread# 23 --------------------
                      ffffffff7ee57468 ???????? (0, 114304600, 0, 8, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (1064c6d68, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a49, 0, ffffffff7141e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (109c0fd98, 1131d5400, 0, 115a948d0, 22, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (109c0fd98, 1003c1828, 1, ffffffff7141eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (109c0fd98, 1, ffffffff7141eaf4, 0, 1131d5400, ffffffff7141e980)
                      ffffffff7eeaede8 op_shared_search (109c0fd98, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (109c0fd98, 1118003f0, 109ee5740, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (109c0fd98, 109c0fd98, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72003478, 109c06f90, a9170, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72003478, 109c06f90, 2, 1003569c0, 10033f1d8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72005288)
                      ffffffff7d636a7c ptroot (100260e30, 20000, 0, 0, 20000, ffffffff7141ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 24 / thread# 24 --------------------
                      ffffffff7f1b4818 number (ffffffff7131d424, 0, 1001c5181, 34, ffffffff7131d12e, 10) + 798
                      ffffffff7f1b3ea8 __doscan_u (403, 0, 0, 1001c5282, ffffffff7131d4f0, 0) + 9d4
                      ffffffff7f1b9e24 vsscanf (14, ffffffff7eee7bb0, ffffffff7131d6b0, 3, 75, 76) + 58
                      ffffffff7f1b88dc sscanf (10614c9f9, ffffffff7eee7bb0, ffffffff7131d6d6, ffffffff7131dca4, ffffffff7131dca0, 0) + 1c
                      ffffffff7eeb7894 csn_init_by_string (ffffffff7131d790, 10614c9f1, 75, 76, 75, 76) + 64
                      ffffffff7ee55a3c ???????? (76, ffffffff7131dcf0, ffffffff7131dcf8, ffffffff7131dca4, ffffffff7131dca0, 0)
                      ffffffff7ee57040 ???????? (113f2d540, 114f15710, 0, 0, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (10614b980, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a62, 0, ffffffff7131e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10614d160, 114fc01c0, 0, 115b83430, 3b, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10614d160, 1003c1828, 1, ffffffff7131eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10614d160, 1, ffffffff7131eaf4, 0, 114fc01c0, ffffffff7131e980)
                      ffffffff7eeaede8 op_shared_search (10614d160, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10614d160, 1127b0070, 115a38510, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10614d160, 10614d160, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72005600, 10614f7d0, 856ef, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72005600, 10614f7d0, 2, 100354df8, 10033f6b0, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72004820)
                      ffffffff7d636a7c ptroot (100260ed0, 20000, 0, 0, 20000, ffffffff7131ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 25 / thread# 25 --------------------
                      ffffffff7ee6f9d8 slapi_dn_ignore_case (104f91490, 104f96468, 0, ffffffff7121deb8, ffffffffffffffff, 4000) + 30
                      ffffffff7ee73680 slapi_dn_strict_normalize_case (ffffffff7121dc58, 1, ffffffff7eff9fa0, 0, 63, 104f96499) + 18
                      ffffffff7ee70940 slapi_sdn_norm_dn (115780d20, 104f96468, 0, ffffffff7121deb8, ffffffffffffffff, 4000) + 30
                      ffffffff7ee709e4 slapi_sdn_get_ndn (115780d20, 1a62, 0, ffffffff7121e7f4, 0, ffffffff77d6eff0) + 34
                      ffffffff77c241f4 ???????? (1006883b8, 114c81030, 0, ffffffff7121deb8, ffffffffffffffff, 4000)
                      ffffffff77c25d70 id2entry (1003c1828, 1a62, 0, ffffffff7121e7f4, 0, ffffffff77d6eff0) + 420
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (1156ffd98, 104f9d468, 0, 115b83430, 3b, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (1156ffd98, 1003c1828, 1, ffffffff7121eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (1156ffd98, 1, ffffffff7121eaf4, 0, 104f9d468, ffffffff7121e980)
                      ffffffff7eeaede8 op_shared_search (1156ffd98, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (1156ffd98, 115b40370, 104f6c720, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (1156ffd98, 1156ffd98, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72005600, 1156f75d8, 856ee, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72005600, 1156f75d8, 2, 100354df8, 10033f1d8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72005728)
                      ffffffff7d636a7c ptroot (100260f70, 20000, 0, 0, 20000, ffffffff7121ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 26 / thread# 26 --------------------
                      ffffffff7f1d3c7c pollsys (ffffffff7111f310, 1, ffffffff7111f250, 0)
                      ffffffff7f16dc58 poll (ffffffff7111f310, 1, 1388, 0, 0, 0) + 88
                      ffffffff7d62f76c pt_poll_now (ffffffff7111f3d0, 0, 0, 3e711251, 20, 9) + 1fc
                      ffffffff7d631c14 pt_Recv (10041fe70, 107a061e8, 91, 0, 7a120, 5) + 174
                      ffffffff7ea1f794 ssl_DefRecv (107a05f88, 107a061e8, 5, 0, 7a120, ffffffff7ea27c50) + 24
                      ffffffff7ea19950 ssl3_GatherData (107a05f88, 107a06198, 0, 4805, 107a061d8, 4800) + 90
                      ffffffff7ea19abc ssl3_GatherCompleteHandshake (107a05f88, 0, 1, 301, 7a120, ffffffff7ea27c50) + 18
                      ffffffff7ea1b1f4 ssl_GatherRecord1stHandshake (107a05f88, 60, de, 0, ffffffff7ea1b1d8, 10ea92c40) + 1c
                      ffffffff7ea20ff0 ssl_Do1stHandshake (107a05f88, 0, 1, 0, 7a120, ffffffff7ea27c50) + d8
                      ffffffff7ea22474 ssl_SecureRecv (107a05f88, 1064cdfe8, 1, 0, 0, 1) + 1a4
                      ffffffff7ea27cb0 ssl_Recv (1002a4ae0, 1064cdfe8, 1, 0, 7a120, ffffffff7ea27c50) + 60
                      ffffffff7ec31b14 ???????? (1002a4ae0, 1064cdfe8, 1, 1, 7a120, 1)
                      ffffffff7e02fe14 ber_filbuf (1008e08c0, 1, 1064cdfed, 0, 10033b948, 0) + b4
                      ffffffff7e02ff34 BerRead (1008e08c0, ffffffff7111fc0f, 1, 800, 958, ffffffff720087b8) + 88
                      ffffffff7e0307cc get_tag (1008e08c0, 1095cf9b0, 2, 100358198, 10033b948, 0) + c
                      ffffffff7e0309fc ber_get_next (1008e08c0, ffffffff7111fd78, 1095cf9b0, 800, 958, ffffffff720087b8) + 44
                      ffffffff7ec2663c ldap_frontend_main_using_core_api (ffffffff72009b60, 1095cf7d0, 2, 100358198, 10033b948, 0) + c
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff720087b8)
                      ffffffff7d636a7c ptroot (100261010, 20000, 0, 0, 20000, ffffffff7111ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 27 / thread# 27 --------------------
                      ffffffff7f1d3c80 __pollsys (ffffffff7101e020, 1, ffffffff7101df50, 0, 0, 26c) + 8
                      ffffffff7f16dc58 poll (ffffffff7101e020, 1, 1388, 0, 0, 0) + 88
                      ffffffff7d634bd0 prpoll_with_poll (0, 0, ffffffff7101e020, 0, ffffffff7101e2f8, 4) + 4f0
                      ffffffff7ec310b8 ???????? (1002a3e50, 2, 7a120, 1, 11b7cc, 1)
                      ffffffff7ec316dc pr_secure_write_function (1002a3e50, 11252b160, 3d, 0, 0, ffffffff7ec3bb04) + 5c
                      ffffffff7e030458 ber_flush (1008d1810, 11252af28, 1, 11252af28, 0, 1025d8) + 150
                      ffffffff7ec3bb28 ldap_getBytes_and_flush (ffffffff720088e0, 11252af28, 76, 1098600f0, 0, ffffffff7ec3bb04) + 24
                      ffffffff7ee8f894 ???????? (11252b568, ffffffff720088e0, 11252d7c8, 11252af28, 2, ffffffff7ec3b9d8)
                      ffffffff7ee8f644 send_ldap_search_entry_ext (11252b568, 114463f00, 0, 1098600f0, 0, ffffffff7f00b878) + 774
                      ffffffff7ee8e5a0 send_ldap_search_entry (11252b568, 114463f00, 0, 1098600f0, 0, 0) + 24
                      ffffffff7eeaf3d8 ???????? (11252b568, 1003c1828, 1, ffffffff7101eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (11252b568, 1, ffffffff7101eaf4, 0, 10754e8c0, ffffffff7101e980)
                      ffffffff7eeaede8 op_shared_search (11252b568, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (11252b568, 1098600f0, 111606250, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (11252b568, 11252b568, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff720088e0, 11252d7c8, 2, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff720088e0, 11252d7c8, 2, 100357de0, 10033b948, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72009348)
                      ffffffff7d636a7c ptroot (1002610b0, 20000, 0, 0, 20000, ffffffff7101ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 28 / thread# 28 --------------------
                      ffffffff7e02cc80 ldap_utf8next (0, 2, 0, 5, 74, 66) + 10
                      ffffffff79a08f74 value_cmp (2, ffffffff70f1d560, 2, 5, 3, a) + 104
                      ffffffff79a03e50 string_filter_ava (113125110, 11250bcd8, 2, a3, 0, ffffffff79a06294) + 4c
                      ffffffff7ee88c98 plugin_call_syntax_filter_ava_sv (115cb14c0, a3, 113125108, 0, 0, 5400) + 1c0
                      ffffffff7ee78ae0 ???????? (0, 113e18dc0, 115cb14c0, 113125108, a3, 4000)
                      ffffffff7eeb9948 ???????? (113e18dc0, 1131250f0, 5400, 104d0af40, ffffffffffffffff, 4000)
                      ffffffff7ee79bfc slapi_vattr_filter_test_ext (11250fd98, 113e18dc0, 1131250f0, 1, 0, a0) + 1dc
                      ffffffff7ee7a4e8 ???????? (11250fd98, 113e18dc0, 0, 1001c5390, 1, 0)
                      ffffffff7ee7a0a4 slapi_vattr_filter_test_ext (11250fd98, 113e18dc0, 113125170, 1, 0, 4db933f9) + 684
                      ffffffff7ee799e0 slapi_vattr_filter_test (11250fd98, 113e18dc0, 113125170, 1, 26f8, 11461d238) + 14
                      ffffffff77c5e8f4 ldbm_back_next_search_entry_ext (11250fd98, 11461d238, 0, 115b83f10, 3b, 4db933f9) + 77c
                      ffffffff7eeaf370 ???????? (11250fd98, 1003c1828, 1, ffffffff70f1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (11250fd98, 1, ffffffff70f1eaf4, 0, 11461d238, ffffffff70f1e980)
                      ffffffff7eeaede8 op_shared_search (11250fd98, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (11250fd98, 113125270, 104d0d320, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (11250fd98, 11250fd98, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72006e48, 11250b000, 15bd0, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72006e48, 11250b000, 2, 100355c68, 10033f2c8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72007b00)
                      ffffffff7d636a7c ptroot (100261150, 20000, 0, 0, 20000, ffffffff70f1ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 29 / thread# 29 --------------------
                      ffffffff7ee57314 ???????? (0, 114674bb0, 0, f, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (115dbaf58, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a62, 0, ffffffff70e1e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (1073b9560, 115d50658, 0, 115b83430, 3b, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (1073b9560, 1003c1828, 1, ffffffff70e1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (1073b9560, 1, ffffffff70e1eaf4, 0, 115d50658, ffffffff70e1e980)
                      ffffffff7eeaede8 op_shared_search (1073b9560, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (1073b9560, 113e90670, 1073e4c90, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (1073b9560, 1073b9560, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72003478, 1073bf7d0, a9171, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72003478, 1073bf7d0, 2, 1003569c0, 10033d248, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72008d80)
                      ffffffff7d636a7c ptroot (1002611f0, 20000, 0, 0, 20000, ffffffff70e1ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 30 / thread# 30 --------------------
                      ffffffff7d1006f0 memcpy (ffffffff70d1d680, 1115fbf70, a, 113fecdf0, ff00, 80808080) + 1a8
                      ffffffff7ee73240 ???????? (ffffffff70d1dc58, 1698, ffffffff7efe9d08, 2, ffffffff70d1d8e8, 1001bc5e0)
                      ffffffff7ee7363c slapi_dn_strict_normalize (ffffffff70d1dc58, 1129b0230, 0, ffffffff70d1deb8, ffffffffffffffff, 4000) + 3c
                      ffffffff7ee7366c slapi_dn_strict_normalize_case (ffffffff70d1dc58, 1, ffffffff7eff9fa0, 0, 63, 1129b0261) + 4
                      ffffffff7ee70940 slapi_sdn_norm_dn (114e53020, 1129b0230, 0, ffffffff70d1deb8, ffffffffffffffff, 4000) + 30
                      ffffffff7ee709e4 slapi_sdn_get_ndn (114e53020, 1a62, 0, ffffffff70d1e7f4, 0, ffffffff77d6eff0) + 34
                      ffffffff77c241f4 ???????? (1006883b8, 1149a2a70, 0, ffffffff70d1deb8, ffffffffffffffff, 4000)
                      ffffffff77c25d70 id2entry (1003c1828, 1a62, 0, ffffffff70d1e7f4, 0, ffffffff77d6eff0) + 420
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (114d787c8, 1129b3988, 0, 115b83430, 3b, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (114d787c8, 1003c1828, 1, ffffffff70d1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (114d787c8, 1, ffffffff70d1eaf4, 0, 1129b3988, ffffffff70d1e980)
                      ffffffff7eeaede8 op_shared_search (114d787c8, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (114d787c8, 10bda04f0, 1105a5610, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (114d787c8, 114d787c8, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff720044a8, 114d721f0, cc9, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff720044a8, 114d721f0, 2, 100356678, 10033e6c0, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72006e48)
                      ffffffff7d636a7c ptroot (100261290, 20000, 0, 0, 20000, ffffffff70d1ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 31 / thread# 31 --------------------
                      ffffffff7f4081a4 MemFreePtr (115848950, 0, 1125850f0, 10faf2598, 115b83f10, 1) + 3c
                      ffffffff7ee555b4 slapi_ch_free (10faf2918, 1a62, 0, ffffffff70c1e7f4, 0, ffffffff77d6eff0) + 2c
                      ffffffff7ee6ddc4 slapi_value_free (10faf2918, 0, 1125850f0, 10faf2598, 115b83f10, 1) + 28
                      ffffffff7eeab060 valuearrayfast_done (1107e2f58, 1a62, 0, ffffffff70c1e7f4, 0, ffffffff77d6eff0) + 34
                      ffffffff7ee6ccbc attr_done (1107e2f50, 0, 1125850f0, 1125850f0, 115b83f10, 1) + 20
                      ffffffff7ee6cc84 slapi_attr_free (ffffffff70c1dc58, 1a62, 0, ffffffff70c1e7f4, 0, ffffffff77d6eff0) + 24
                      ffffffff7eeb87a4 attrlist_free (1107e32c0, 0, 1125850f0, 1125850f0, 115b83f10, 1) + 18
                      ffffffff7ee59404 slapi_entry_free (1125850f0, 1a62, 0, ffffffff70c1e7f4, 0, ffffffff77d6eff0) + 64
                      ffffffff77c2ddb4 backentry_free (1128215d0, 1128215d0, 115b83f10, ffffffff70c1deb8, 115b83f10, 1) + 18
                      ffffffff77c25d84 id2entry (1003c1828, 1a62, 0, ffffffff70c1e7f4, 0, ffffffff77d6eff0) + 434
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10faf1ba8, 105092958, 0, 115b83430, 3b, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10faf1ba8, 1003c1828, 1, ffffffff70c1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10faf1ba8, 1, ffffffff70c1eaf4, 0, 105092958, ffffffff70c1e980)
                      ffffffff7eeaede8 op_shared_search (10faf1ba8, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10faf1ba8, 10de20970, 10e0369d0, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10faf1ba8, 10faf1ba8, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72006bf8, 10faf2fa8, 87197, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72006bf8, 10faf2fa8, 2, 100355bf8, 10033b948, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72008568)
                      ffffffff7d636a7c ptroot (100261330, 20000, 0, 0, 20000, ffffffff70c1ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      ----------------- lwp# 32 / thread# 32 --------------------
                      ffffffff7f1b36e4 __doscan_u (84, 0, ffffffff70b1d330, 1001c5282, ffffffff70b1d4f0, 0) + 210
                      ffffffff7f1b9e24 vsscanf (14, ffffffff7eee7ba8, ffffffff70b1d6b0, 3, 75, 76) + 58
                      ffffffff7f1b88dc sscanf (106d13ca4, ffffffff7eee7ba8, ffffffff70b1d6d8, ffffffff70b1dca4, ffffffff70b1dca0, 0) + 1c
                      ffffffff7eeb787c csn_init_by_string (ffffffff70b1d790, 106d13ca4, 75, 76, 75, 76) + 4c
                      ffffffff7ee55a3c ???????? (76, ffffffff70b1dcf0, ffffffff70b1dcf8, ffffffff70b1dca4, ffffffff70b1dca0, 0)
                      ffffffff7ee57040 ???????? (111fc2910, 110c92bc0, 0, 0, ffffffff7efe9d08, 0)
                      ffffffff7ee5899c slapi_str2entry (106d13110, 5400, 5638, 1001c5390, ffffffffffffffff, 4000) + 94
                      ffffffff77c25d20 id2entry (1003c1828, 1a49, 0, ffffffff70b1e7f4, 0, ffffffff77d6eff0) + 3d0
                      ffffffff77c5e478 ldbm_back_next_search_entry_ext (10fa18080, 115690dc8, 0, 115a948d0, 22, 4db933f9) + 300
                      ffffffff7eeaf370 ???????? (10fa18080, 1003c1828, 1, ffffffff70b1eaf4, ffffffffffffffff, ffffffff7efe9d08)
                      ffffffff7eeaf83c ???????? (10fa18080, 1, ffffffff70b1eaf4, 0, 115690dc8, ffffffff70b1e980)
                      ffffffff7eeaede8 op_shared_search (10fa18080, 0, 1006cbad0, 0, ffffffffffffffff, ffffffff7efe9d08) + 8c0
                      ffffffff7ec2efcc search_core_pb (10fa18080, 115650170, 109ee1d60, 0, ffffffff7ed4c810, ffffffff7ed56088) + 6c4
                      ffffffff7ec3a710 dispatch_operation_core_pb (10fa18080, 10fa18080, c00, ffffffff7ed4c810, 0, d10) + cc
                      ffffffff7ec26420 ???????? (ffffffff72008ea8, 10fa1d7c8, 3c492, 63, 2, 200000)
                      ffffffff7ec2672c ldap_frontend_main_using_core_api (ffffffff72008ea8, 10fa1d7c8, 2, 100357ef8, 10033f2c8, 0) + fc
                      ffffffff7ec27764 ???????? (220, 0, ffffffffffffffff, 800, 958, ffffffff72006bf8)
                      ffffffff7d636a7c ptroot (1002613d0, 20000, 0, 0, 20000, ffffffff70b1ff48) + d4
                      ffffffff7f1d2e7c lwpstart (0, 0, 0, 0, 0, 0)
                      • 8. Re: ns-slapd process is using 99% CPU.
                        John Prince
                        Hi Chris,

                        How to you use this command, if we multiple ldap running on the same system.

                        pgrep ns-slapd | xargs -I {} prstat -L -p {} 1 1 > prstat.`date +%H%M%S`;pgrep ns-slapd | xargs pstack > pstack.`date +%H%M%S`


                        Thanks, John
                        • 9. Re: ns-slapd process is using 99% CPU.
                          802907
                          The simplest way is to do ps -ef | grep slapd and just get the list of pids. Then identify the pid of the DS instance you are troubleshooting and substitute it into the prstat and pstack commands directly, instead of using xargs the way I did.
                          • 10. Re: ns-slapd process is using 99% CPU.
                            802907
                            From the stack trace, it looks like most of your busy threads (~3% CPU each) are servicing search queries. If you want a more detailed analysis of the stack trace, you can open a ticket with Support. BTW, you seem to have a lot of CPUs. DS does not utilize a large number of CPUs very well. IIRC, the "sweet spot" is somewhere around 12-16.

                            I do think it's safe to say that there is something about your workload that is causing the spikes. In order to characterize the workload, I suggest you create a slapd health monitor log. I use something like this:
                            #!/usr/bin/perl
                            use Net::LDAP;
                            
                            $BIND_DN="cn=dmanager";
                            $BIND_PW="password";
                            $HEADER="timestmp  cpu_util  virtsize  cconns  readwt  rqbklg  pndops  txlgct\n";
                            
                            # Monitor slapd constants
                            $ATTRIBUTES = [ 'currentconnections','readwaiters','opsinitiated','opscompleted','request-que-backlog'];
                            $DEBUG=0;
                            $PATH="/export/home/chris/monitor_logs/slapd_monitor";
                            $HOSTNAME=`hostname`;
                            $TS=`date +%H:%M:%S`;
                            $DATE=`date +%Y%m%d`;
                            $txlogpath="/data/messaging/slapd-solx86/db";
                            
                            chomp($TS);
                            chomp($DATE);
                            chomp($HOSTNAME);
                            
                            $OUTPUT_FILE="$PATH.$HOSTNAME.$DATE";
                            
                            if ( ! -f $OUTPUT_FILE ) {
                                 open(OUTPUT,">$OUTPUT_FILE") or die "ERROR: cannot create $PATH.$DATE: $!\n";
                                 print OUTPUT $HEADER;
                            } 
                            else {
                                 open(OUTPUT,">>$OUTPUT_FILE") or die "ERROR: cannot append to $PATH.$DATE: $!\n";
                            }
                            
                            printf OUTPUT ("%8s  ",$TS);
                            
                            $psinfo = `ps -eo pcpu,vsz,args|grep 'ns-slapd'| grep -v grep`;
                            if ($psinfo =~ /\s?(.+?)\s+(\d+?)\s+/) {
                                    $pcpu = $1;
                                    $vsz = $2 / 1024**2;
                                    printf OUTPUT ("%7.1f%%  %6.3fGB  ",$pcpu,$vsz);
                            }
                            else {
                                    $pcpu = 'NO_DAT';
                                    $vsz = 'NO_DAT';
                                    printf OUTPUT ("%8s  %8s  ",$pcpu,$vsz);
                            }
                            
                            $ldap = Net::LDAP->new ($HOSTNAME);
                            
                            if (! $ldap) {
                                    undef($slapd_up);
                                    $conns = 'NO_DAT';
                                    $rwait = 'NO_DAT';
                                    $backlog = 'NO_DAT';        
                                    $pending = 'NO_DAT';
                                    $txlognum = 'NO_DAT';
                                    printf OUTPUT ("%6s  %6s  %6s  %6s  %6s\n",$conns,$rwait,$backlog,$pending,$txlognum);
                            }     
                            else {
                                    $mesg = $ldap->bind ( "$BIND_DN",
                                         password => "$BIND_PW",
                                         version => 3
                                    );
                            
                                    my $result = $ldap->search ( base    => "cn=monitor",
                                         scope   => "sub",
                                           attrs   => [ 'currentconnections','readwaiters','opsinitiated','opscompleted','request-que-backlog' ],
                                         filter  => "(cn=monitor)"
                                         );
                            
                                    if ($result->count) {
                                         $entry =  $result->entry(0);
                            
                                            if (($entry->exists('opsinitiated')) && ($entry->exists('opscompleted'))) {
                                                    $opsinitiated = $entry->get_value('opsinitiated');
                                                    $opscompleted = $entry->get_value('opscompleted');
                                                    $pending = ($opsinitiated - $opscompleted);
                                            }
                                            else {
                                                 $pending = 'NO_DAT';
                                            }
                                            if ($entry->exists('currentconnections')) {
                                                    $conns = $entry->get_value('currentconnections');
                                            }
                                            else {
                                                    $conns = 'NO_DAT';
                                            }
                                            if ($entry->exists('readwaiters')) {
                                                    $rwait = $entry->get_value('readwaiters');
                                            }
                                            else {
                                                    $rwait = 'NO_DAT';
                                            }
                                            if ($entry->exists('request-que-backlog')) {
                                                    $backlog = $entry->get_value('request-que-backlog');
                                            }
                                            else {
                                                    $backlog = 'NO_DAT';
                                            }
                                    }
                                    else {
                                            $pending = 'NO_DAT';
                                            $conns = 'NO_DAT';
                                            $rwait = 'NO_DAT';
                                            $backlog = 'NO_DAT';
                                    }
                                    $try = `ls $txlogpath\/log.* \| wc -l`;
                                    if ($try =~ /(\d+)/) {
                                            $txlognum = $1;
                                    }
                                    else {
                                            $txlognum = 'NO_DAT';
                                    }
                                    printf OUTPUT ("%6d  %6d  %6d  %6d  %6d\n",$conns,$rwait,$backlog,$pending,$txlognum);
                            }
                            
                            close(OUTPUT);
                            Note on the psinfo line there is a grep for ns-slapd to get the pid from ps. If you have more than one instance of slapd running, you will need to grep out the particular instance of slapd you want to monitor. The script uses Net::LDAP with perl. It's pretty easy to install. You will need to change the variables at the top of the script to suit your environment. Also, if you are running on a non-default ldap port, you will need to change the line that makes the connection.

                            If you run out of cron every minute, you should see output like:
                            timestmp  cpu_util  virtsize  srchtime  cconns  readwt  rqbklg  pndops  txlgct
                            00:00:04      0.1%   0.243GB     291ms       2       0       0       1       1
                            00:01:01      0.1%   0.243GB      88ms       5       0       0       1       1
                            00:02:01      0.1%   0.243GB      87ms       4       0       0       1       1
                            00:03:01      0.1%   0.243GB      87ms       4       0       0       1       1
                            00:04:01      0.1%   0.243GB      87ms       4       0       0       1       1
                            So you will want this logging for a day or two while your issue is happening (I use this or something like it on all my installations, regardless.)

                            You will start to see your CPU spikes in the log. Once you have identified several periods of the highest CPU and several periods of the lowest CPU, you can use the access log to characterize the workload and start correlating differing types of load with your high and low CPU intervals. I use the stats output of lgrep http://wikis.sun.com/display/lgrep/lgrep for this, since it lets me drill down into the access log pretty easily.
                            • 11. Re: ns-slapd process is using 99% CPU.
                              857612
                              From my monitoring tool :

                              My CPU spikes are random. For every 10 min (1,11,21,31......) the CPU hit 70-80% and rest of the time it fall between 10-20%.

                              Do you still want me to run your script ?
                              • 12. Re: ns-slapd process is using 99% CPU.
                                802907
                                Ok, if you have a log that gives a minute-by minute cpu utilization for the slapd process, identify:
                                - three 1 minute intervals of max cpu
                                - three 1 minute intervals of average cpu
                                - three 1 minute intervals of min cpu

                                Then you should be able to go straight to the access log and determine if there is a type of load that's happening when your cpu is high but not when it's low. Once you have a hypothesis of what type of load may be associated with the high cpu, you can check the access log for any interval where that type of load was high, and verify that during that interval in your health monitor log there was an associated increase in cpu utilization.
                                • 13. Re: ns-slapd process is using 99% CPU.
                                  857612
                                  Since this is private data ....cleared

                                  Edited by: 854609 on May 17, 2011 6:31 PM
                                  • 14. Re: ns-slapd process is using 99% CPU.
                                    802907
                                    The logconv output is helpful in as much as we can see there are some long etimes but we don't know what kinds of operations at what times and we don't know how they correlate to your CPU spikes. I outlined a process earlier in the thread for drilling down into the access log and correlating your CPU spikes with a workload characterization. The tool I usually use for this is lgrep: http://wikis.sun.com/display/lgrep/lgrep