Possible to get data from a partly optimized/stripped core file?

961722

Hello,

This may not be possible, but I figured it was worth asking about.

I've got a C/C++ GUI application compiled with Solaris Studio 12.3 that is experiencing an infrequent crash when compiled for production and running on production boxes.  This is on Solaris 10 for x86 running in 64-bit mode.  Most of the app is in libraries which are statically linked.

I working on trying to replicate the issue in a development environment, but have not had luck yet. In any case, it would be interesting to know what kind of data can be gleaned postmortem from the core file I've got access to.

The application is actually a small "main.c" file which is complied and linked in debug mode with "-g" and no optimization, but this thin wrapper calls into the main logic in statically linked libraries which are optimized and not built in debug mode.  (See the call stack below.)

From the core file :

1) For functions in the call stack that have names, can I get the value of one of the parameters?  I ask because several such functions take pointers to structs with data that should be very useful.

2) For functions in the call stack that appear as ??????, is it possible to determine at least what .o or .a file they came from?  This could help narrow things down.

Some basic Googling indicates that either of the above may not be trivial or even possible.  But I'm wondering if the fact that we've got a "main.c" debuggable wrapper might somehow help.

As a related question, pstack produces sensible output, but dbx shows the error: "dbx: internal error: could not iterate over load objects -- link-maps are not initialized".  Is there some flag I need to supply to dbx?

Thank you for any help,

David

Background info:

I've been unable to replicate on non-production deployments, but the machines do differ a bit.   Eventually I will be able to borrow a production box to deploy an instrumented binary, but for now all I've got is a core file and access to source.

The core was generated with gcore while the app was displaying a popup from it's SIGABRT cleanup handler.   The production build scripts do some binary stripping, but I'm not yet sure where it is getting done.

Here is the (slightly cleaned up) output of pstack for the core file:

fffffd7ffeb3244a nanosleep (fffffd7fffdfd4b0, 0)

0000000000514485 ZWidget_ModalEventLoop () + 65

00000000004f74a9 ZWidget_ShowPopup () + 4a9

000000000049d2ab ???????? ()

fffffd7ffeb2dd16 __sighndlr () + 6

fffffd7ffeb225e2 call_user_handler () + 252

fffffd7ffeb2280e sigacthandler (6, 0, fffffd7fffdfd640) + ee

--- called from signal handler with signal 6 (SIGABRT) ---

fffffd7ffeb3351a _lwp_kill () + a

fffffd7ffead81b9 raise () + 19

fffffd7ffeab6b4e abort () + 5e

000000000052c3bc ZUtil_Query () + 3c

000000000059b66e ZUtil_QueryString () + 3e

00000000004a1e2a ???????? ()

00000000004a0879 ???????? ()

000000000058b303 ???????? ()

000000000052d517 ZUtil_Set () + 767

00000000004f4805 ZUtil_DBSet () + 35

00000000005094b5 ZWidget_ProcessCallback () + 465

0000000000516814 ???????? ()

fffffd7fff242424 XtCallCallbackList () + 114

fffffd7ffef84d2e ActivateCommon () + 126

fffffd7ffef84b72 Activate () + 1e

fffffd7fff244efa HandleActions () + 14a

fffffd7fff24b1b7 HandleComplexState () + 177

fffffd7fff243a9e _XtTranslateEvent () + 4e

fffffd7fff24382a XtDispatchEventToWidget () + 2ea

fffffd7fff2430ee _XtDefaultDispatcher () + 15e

fffffd7fff242db6 XtDispatchEvent () + 106

00000000005142df ZWidget_ProcessEvent () + ff

0000000000514099 ZWidget_ProcessEvents () + 19

00000000005ac67a ZEventLoop_ProcessEvents () + 5a

00000000005ac528 ZEventLoop_Execute () + 48

000000000049d133 Main () + c93

000000000049bdf9 main () + 9

000000000049bc7b ???????? ()

Janitor-Oracle

Thanks for reporting this problem.

>1) For functions in the call stack that have names, can I get the value of one of the parameters?  I ask because several such functions take pointers to structs with data that should be very useful.

Use compiler option -preserve_argvalues={none|simple|complete} to preserve incoming argument values. Note that this feature was introduced in Oracle Solaris Studio 12.4.

You may also be interested in a new option in Oracle Solaris Studio 12.4 which provides much finer-grained control over debug information, which allows you to choose how much information is provided and to reduce the amount of disk space needed for the executable. .

>2) For functions in the call stack that appear as ??????, is it possible to determine at least what .o or .a file they came from?  This could help narrow things down.

The following 2 commands may help:

where -l

# Include library name with function name.

whereis -a <addr-of-?????> # Print location of an address expression

>As a related question, pstack produces sensible output, but dbx shows the error: "dbx: internal error: could not iterate over load objects -- link-maps are not initialized".  Is there some flag I need to supply to dbx?

This may be caused by corefile mismatch. See dbx online help: "help core mismatch" for suggestions.

Hope this helps.