5 Replies Latest reply: May 30, 2012 8:20 PM by Ivanigorovich-Oracle RSS

    Linux dbx can't initialize thread debugging library

    919271
      Hello,

      I am encountering an issue trying to debug programs compiled with SS 12.3 in a Linux environment.

      The target environment is a customized Wind River Linux distribution that we deploy. I understand that Oracle only officially supports RHEL and OEL - I was just hoping someone who is an expert could possibly chime in with some clues to help us make this work.

      Some relevant background info:

      glibc-2.8-1_WR4.3a_274.0.0.0.0.x86_32
      libstdc++-4.3a_274-1_WR4.3a_274.0.0.0.0.x86_32
      Linux msclab66 2.6.27-pne #1 SMP Wed May 9 17:49:29 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux

      Note - we don't deploy any 64-bit libraries in our environment. These are all 32-bit libraries and programs.

      dbx is unable to initalize the 'thread debugging library'.

      Yes, there are some issues with name demangling, but I'm not as concerened about that right now. I'm not as worried about debugging libstdc++ code as I am having the ability to debug multithreaded programs properly. As it stands, dbx is pretty much useless for us because it can't do anything thread-related.

      Another note - We have a RHEL 5.7 system that is fully capable of debugging this the same program /w threading support working correctly.

      Any bit of advice here are greately appreciated.

      msclab66# /opt/oracle/solarisstudio12.3/bin/dbx -xexec32 /opt/SANTone/msc/10.01.00.00/bin/CsFm
      For information about new features see `help changes'
      To remove this message, put `dbxenv suppress_startup_message 7.9' in your .dbxrc
      Reading CsFm
      Reading ld-linux.so.2
      Reading libCstd.so.1
      Reading librt.so.1
      Reading libpthread.so.0
      Reading libnsl.so.1
      Reading libdl.so.2
      Reading libttclient.so
      dbx: internal warning: cannot demangle '__14COdbcSQLTables'
      Reading libCrun.so.1
      Reading libm.so.6
      Reading libc.so.6
      Reading libstdc++-libc6.2-2.so.3
      dbx: internal warning: cannot demangle '__14__si_type_infoPCcRC16__user_type_info'
      dbx: internal warning: cannot demangle '__13bad_exception'
      <snipped the rest of the libstdc++ warnings for brevity>
      Reading libgcc_s.so.1
      dbx: warning: could not initialize thread debugging library -- program is not linked with libthread
      dbx: warning: thread related commands will not be available
      dbx: warning: see `help lwp', `help lwps' and `help where'
      (dbx)

      Here is the output of ldd on the CsFm executable. It's linked with libpthread.so.


      msclab66# ldd /opt/SANTone/msc/10.01.00.00/bin/CsFm
      linux-gate.so.1 => (0x55555000)
      libCstd.so.1 => /usr/lib/libCstd.so.1 (0x55557000)
      librt.so.1 => /lib/librt.so.1 (0x451c8000)
      libpthread.so.0 => /lib/libpthread.so.0 (0x44ea5000)
      libnsl.so.1 => /lib/libnsl.so.1 (0x450d5000)
      libdl.so.2 => /lib/libdl.so.2 (0x44e9f000)
      libttclient.so => /opt/SANTone/msc/active/TimesTen/lib/libttclient.so (0x556c1000)
      libCrun.so.1 => /usr/lib/libCrun.so.1 (0x55705000)
      libm.so.6 => /lib/libm.so.6 (0x44ebf000)
      libc.so.6 => /lib/libc.so.6 (0x44d5e000)
      /lib/ld-linux.so.2 (0x44d40000)
      libstdc++-libc6.2-2.so.3 => /usr/lib/libstdc++-libc6.2-2.so.3 (0x55716000)
      libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x451b9000)
        • 1. Re: Linux dbx can't initialize thread debugging library
          Ivanigorovich-Oracle
          Dbx relies on a system-supplied library, libthread_db.so with an interface specified in /usr/include/thread_db.h.
          The error message: "program is not linked with libthread" corresponds to the TD_NOLIBTHREAD returned by
          the call to td_init().

          libthread_db.so provides a stable API for debuggers to access libpthread data structures.
          Both are supplied by the system.

          It's hard to tell why this might be happening. One common case is as follows:
          libthread_db.so and libpthread.so have to be "in sync". We've seen, even on Solaris systems, cases
          where for example a patch updates libpthread but libthread_db.so gets neglected.

          We may be able to tell more if we get some logging information from dbx as follows:

          Create a file called "C.dbx" in the directory where you run dbx with content as follows:
          options -file L.dbx
          LogTDB # call tdb_log(1) fo (thread_db)
          LogDB # echo calls from ps_plog
          LogPO # proc_service

          Reproduce the problem and post the resulting L.dbx.
          • 2. Re: Linux dbx can't initialize thread debugging library
            919271
            #cat C.dbx
            options -file L.dbx
            LogTDB # call tdb_log(1) fo (thread_db)
            LogDB # echo calls from ps_plog
            LogPO # proc_service

            # /opt/oracle/solarisstudio12.3/bin/dbx -xexec32 ./CsFm
            <output snipped - same as before>

            # cat L.dbx
            LogPO ps_pentry(8c923a4)
            LogPO ps_pglobal_lookup(8c923a4, "PS_OBJ_LDSO", "_dl_init")
            LogPO ps_pglobal_lookup() could not find symbol
            LogPO ps_pentry(8c923a4)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x4f88664c, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x4f886660, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x4f88664c, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x4f88664c, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x4f886660, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x4f88664c, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x4f886660, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x4f886940, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7f043e8, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7f04680, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7f04910, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7f04bb0, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d9a000, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d9a298, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d9a540, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d9a7d8, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d9aa60, 20)
            LogPO ps_pdread(0x0x8c923a4, 0x4f886298, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d45000, 20)
            LogPO ps_pdread(0x0x8c923a4, 0xf7d45340, 20)
            LogPO ps_pglobal_lookup(8c923a4, "libpthread.so.0", "nptl_version")
            LogPO ps_pglobal_lookup() could not find symbol
            LogPO ps_pdread(0x0x8c923a4, 0x863f194, 280)
            LogPO ps_pdread(0x0x8c923a4, 0x4f88664c, 20)
            • 3. Re: Linux dbx can't initialize thread debugging library
              919271
              Hmm, that's interesting (Reading the L.dbx output).


              Here is a RHEL 5 box:

              c3linuxdev.genband.com:/lib$ nm libpthread.so.0 | grep "nptl_version"
              0000000000941b5d r nptl_version

              On the Wind River system:

              # nm libpthread.so.0
              nm: libpthread.so.0: no symbols

              # nm libpthread-2.8.so
              nm: libpthread-2.8.so: no symbols

              # ls -lh libpthread-2.8.so
              -rwxr-xr-x 1 root root 85K May 27 2010 libpthread-2.8.so

              I'm going to have to get with our WRL developers and ask them what's up.

              Thanks for the info so far, at least we've found something.
              • 4. Re: Linux dbx can't initialize thread debugging library
                Ivanigorovich-Oracle
                Try "nm -D". maybe 'nptl_version' migrated to the dynamic symbol table.
                • 5. Re: Linux dbx can't initialize thread debugging library
                  Ivanigorovich-Oracle
                  One thing I neglected to mention.

                  In the absence of working threads commands you can still get a lot of traction
                  with the lwp and lwps commands and use l@ instead of t@.