1 2 Previous Next 29 Replies Latest reply on Jul 1, 2009 6:15 AM by 807573 Go to original post
      • 15. Re: SunDS 6.3 "crashes" when do ldapmodify
        807573
        when it crashes, where would we look for the core file
        http://kbase.redhat.com/faq/docs/DOC-4897
        You had mentioned that while in gdb, the SunDS would be paused. If that's the
        case, and if we then run the ldapmodify, I don't understand how that would work?
        In the initial post, you had mentioned DS "hung", so I was explaining how to attach to a running process and obtain it's real time stack. Now that you know for sure it is crashing, you'll need to
        - ensure core dumps are being put into a partition with plenty of space (as stated in above doc).
        - Perform your change using ldapmodify. Keep a copy of your LDIF file so the support person can use it later

        Now use gdb to open the core file
        - cd into the ns-slapd location
        - run: gdb /path/to/corefile ns-slapd
        - run the commands i specified earlier to get the stack trace
        - log a case with Sun.

        Good luck

        Edited by: etst123 on Apr 16, 2009 2:10 PM
        • 16. Re: SunDS 6.3 "crashes" when do ldapmodify
          jimcpl
          Hi,

          Ok, thanks. We'll give that a try.

          Jim
          • 17. Re: SunDS 6.3 "crashes" when do ldapmodify
            jimcpl
            Hi,

            We're still trying to get a core, but meanwhile, we found something interesting.

            We were testing with one "bad" group ("bad" => ldapmodify uniquemember causes crash), and we changed the ldif to a replace instead of add, and that worked. The interesting part (to me at least) is that we could then do ldapmodify to the same group with add in the ldif, and doing that to that "bad" group didn't cause crash anymore.

            So, it seems like doing a replace (a) works and (b) eliminates whatever the problem was with that 'bad' group.

            I don't think that that helps much, but figured that I'd mention it.

            Jim

            P.S. Applying that patch to bring it to 6.3.1 didn't fix the problem either.
            • 18. Re: SunDS 6.3 "crashes" when do ldapmodify
              jimcpl
              Hi,

              Ok, we did an "strace -f" on ns-slapd, and we are seeing SIGSEGV, i.e., SunDS is segfaulting. We also got a core, and I ran gdp==> where, and these are what it is showing:

              in csnset_size_sh
              in slapi_attrlist_size_sh
              in slapi_entry_size_sh
              in slapi_entry_size_sh
              in slapi_entry_size_sh
              in cache_replace
              in ldbm_back_modify_try
              in ldbm_back_modify
              in op_shared_modify
              in modify_core_pb
              in dispatch_operation_core_pb
              in process_ldap_operation_using_core_api
              in ldap_frontend_main_using_core_api
              in generic_handle_read_data
              in generic_workerthread_main
              in ptroot
              in start_thread
              in clone


              I won't type it all, but here's some of the output from "thread apply all bt":

              Thread 42 (Process 4037):
              in __kernel_vsyscall
              in poll
              in prpoll_with_poll
              in slapd_daemon
              in main


              Jim
              • 19. Re: SunDS 6.3 "crashes" when do ldapmodify
                807573
                Wonderful news that you are able to provide a stack trace. Matches a known bug. Bad news is that it's not fixed in 6.3.1, so upgrading won't help. Please log a support case and mention that you're running into this bug.

                6710342 Memory corruption causes crash during cache update
                • 20. Re: SunDS 6.3 "crashes" when do ldapmodify
                  jimcpl
                  Hi,

                  THANKS for that info!

                  I'll pass it along to my colleague, and hopefully he can track the patch down. We do have some Sun folks on our program, so I'll ping them also.

                  Thanks again,
                  Jim
                  • 21. Re: SunDS 6.3 "crashes" when do ldapmodify
                    jimcpl
                    Hi,

                    BTW, sorry it took so long to get that info to you. Before getting the core and running gdb, I wasn't sure how much info I'd need to get from the "where", so I was kind of hesitant about even trying it.

                    Once I ran gdb ==> where, it looked like transcribing the stacktrace info "by hand" woudn't be so terrible, so that's what I ended up doing. I'm glad it was enough for you to look at.


                    Also, BTW, I had a heck of a time getting the core. I had followed that Redhat article and ran the "ulimit -c unlimited", and we still weren't getting a core file anywhere on the system. I also tried the "echo "/tmp/core" > /procs/..." (and then I created /tmp/core and chmod'ed /tmp/core to 777) to try to set the location for the core file, but that didn't do anything either.

                    I think what finally got a core file to be produced was I put an "strace -f" on the ns-slapd in start-slapd.

                    The strace showed we were getting a segfault when ns-slapd died, and, only then did we start seeing core files appearing in /tmp.

                    It is possible that it would only write the core file if strace was running?

                    I'd never heard that before. I had had problems awhile ago, when I was tracking a problem with Apache, and couldn't get that to core initially, but then I did the "ulimit -c unlimited" and set the location, etc., and I was then able to get the core from Apache.

                    In this case, with ns-slapd, I was almost ready to give up on getting a core :), but thankfully, it finally worked, but I'm still curious about this.

                    Thanks again,
                    Jim
                    • 22. Re: SunDS 6.3 "crashes" when do ldapmodify
                      jimcpl
                      Hi,

                      We're having problems finding info on 6710342.

                      Do you know if there is a PATCH corresponding to the above BUG?

                      Thanks,
                      Jim
                      • 23. Re: SunDS 6.3 "crashes" when do ldapmodify
                        807573
                        Unless you have coreadm configured, you will probably find the core file under the logs directory.
                        • 24. Re: SunDS 6.3 "crashes" when do ldapmodify
                          jimcpl
                          Hi,

                          BTW, just before the SIGSEGV in the strace output, we're seeing:

                          madvise ................ EBADF (Bad file descriptor)
                          madvise ................ EBADF (Bad file descriptor)
                          time(null)
                          futex(xxxx,FUTEX_CMP_REQUEUE,....) = 1
                          <...futex resumed>
                          pread64(28, <unfinished...>
                          futex(xxxxx, FUTEX_WAKE, 1 <unfinished...>
                          <...pread64 resumed>
                          <...futext resumed> = 0
                          SIGSEGV at...

                          It seems like ns-slapd is trying to get some memory, but is not getting any, then it segfaults??

                          Just guessing...

                          Jim
                          • 25. Re: SunDS 6.3 "crashes" when do ldapmodify
                            807573
                            We're having problems finding info on 6710342.
                            Do you know if there is a PATCH corresponding to the above BUG?
                            This bug (and some others which are similar in nature) is not fixed in 6.3.1 which is the latest release as of now. This is why you won't see a downloadable patch. You'll need a hotfix and for this you need to log a support case.
                            • 26. Re: SunDS 6.3 "crashes" when do ldapmodify
                              jimcpl
                              Hi,

                              We finally "found" our support contract #, and the other fellow is opening a case with Sun support.

                              Thanks, for your help!

                              Jim
                              • 27. Re: SunDS 6.3 "crashes" when do ldapmodify
                                jimcpl
                                Hi,

                                Just an update. We have had a case with Sun on this problem, and, although they are aware of that bug #, they don't seem have a fix at this point, and nothing is scheduled.

                                Jim
                                • 28. Re: SunDS 6.3 "crashes" when do ldapmodify
                                  807573
                                  Hi,

                                  running:
                                  Sun Microsystems, Inc.
                                  Sun-Java(tm)-System-Directory/6.3 B2008.0311.0058 64-bit
                                  ns-slapd : 6.3 B2008.0311.0058 DSEE-63-RTM_FIXBranch (SunOS tagada 5.9 Generic_118558-27 sun4u sparc SUNW,Sun-Fire-V250) NAT
                                  Slapd Library : 6.3 B2008.0311.0058 DSEE-63-RTM_FIXBranch (SunOS tagada 5.9 Generic_118558-27 sun4u sparc SUNW,Sun-Fire-V250)
                                  Front-End Library : 6.3 B2008.0311.0058 DSEE-63-RTM_FIXBranch (SunOS tagada 5.9 Generic_118558-27 sun4u sparc SUNW,Sun-Fire-V250)
                                  System: usparcv9-SUNW,Sun-Fire-V210-solaris5.10_s10s_u7wos_08

                                  today we had a crash of the directory server V6.3. No core dump, however. Running Solaris 10 on Sparc. What coreadm command do I need to use to have the core dump being properly written to a file?

                                  Is this bug already fixed?

                                  /rolf
                                  • 29. Re: SunDS 6.3 "crashes" when do ldapmodify
                                    807573
                                    If you are not able to look at the core dump and search based on the stack trace of the crashing thread, we won't know what bug is involved here. Read these to get started on coreadm:

                                    [http://docs.sun.com/app/docs/doc/816-5166/coreadm-1m?a=view|http://docs.sun.com/app/docs/doc/816-5166/coreadm-1m?a=view]
                                    [http://developers.sun.com/solaris/articles/manage_core_dump.html|http://developers.sun.com/solaris/articles/manage_core_dump.html]
                                    1 2 Previous Next