SungHoon_Kim

Collecting PKGAPP and how to get the stack trace

Blog Post created by SungHoon_Kim Employee on Feb 25, 2016

The use case is, the PS is crashing on linux.

User has pkgapp script and is trying to collect a packaged core file.

 

The pkgapp script usage can by found simply by running the pkgapp script itself without any parameters.

(This is based on pkgapp 3.4, not the latest)

pkgapp

* ------------------------------------------------------------------------------

* Oracle Corporation CSD pkgapp 3.4 Linux                           [02/25/2016]

* ------------------------------------------------------------------------------

pkgapp 3.4, an Oracle Corporation data gathering utility.

Usage:

  pkgapp [options] -p <active pid> -a <full path to process binary> [-c <core file>] [-s path to write tar file]

 

 

Required parameters:

-p <pid of a active process>

    Note: pid should of the same type/path to match -a & -c that produced the core

-a <full path to to and including the process binary> (ns-slapd, imapd, httpd etc.)

 

 

Optional parameters:

-c <core file>

-i (Include a previously generated core file with the final tar.gz)

-j FUNCTION DISABLED (Javacore; process a java core)

    NOTE jstack/jinfo etc. are not available on Linux platforms

-r (Remove all temp files)

-t (Remove all history files)

-q (Quiet)

-d (Debug)

-J FUNCTION DISABLED <JstackPath; path to the jstack (jdk) commands>

        defaults to /usr/jdk/instances/...

    NOTE jstack/jinfo etc. are not available on Linux platforms

-l <ldd Path; path to a known good ldd/list of the core or process>

-s <Storage; path to store the final tar file in>

 

 

usage:  pkgapp -p <pid of the running app> -a <path to process binary>

 

 

Examples: these are examples mixing various parameters

 

 

The example paths are examples only, not to be used in your environment.

-e Locate the correct path for the -a value on your system before using.

 

 

Directory Server

pkgapp -ir -p 9965 -c ./core.14740 -a /var/mps/ds52p4/bin/slapd/server/64/ns-slapd

 

 

Messaging Server

pkgapp -p 9965 -c ./core.3496 -a /opt/SUNWmsgsr/lib/stored

 

 

Web Server

pkgapp -p 9965 -c ./core.1092 -a /space/iws70/lib/httpd -s /var/crash

 

 

Calendar Server

pkgapp -r -p 9965 -c ./core -a /opt/SUNWics5/cal/lib/cshttpd

 

 

Cacao (java)

pkgapp -j -i -r -p 9965 -c ./core -a /opt/dsee63/jre/bin/java

 

 

Sendmail

pkgapp -i -p 9965 -c 512 -a /usr/lib/sendmail

 

 

Mysqld

pkgapp -i -r -p 9965 -c ./core -a /support/mysql-5.0.41/bin/mysqld

 

But before running the script, you must ensure that you have switched to the user account that runs the service and that you have sourced the environment variables.

 

So you need to switch to smuser (su - smuser) and then source the environment variable(. ./ca_ps_env.ksh).

Then you need to run the policy server if it is not already running. (This is to give sufficient information to the pkgapp script to possibly collect all the library files that are referenced in the core file)

 

Following is the environment information.

core file filepath is /apps/core-smpolicysrv-2615-500-11-1451996208

smpolicysrv file path is /apps/CA/siteminder/bin/smpolicysrv

currently running smpolicysrv pid is 20541

pkgapp filepath is /apps/binary/pkgapp/pkgapp

 

When creating packaged core file, you need to have access to various system resources(/var/log/messages) so smuser may not be sufficient to collect all that information but you can still run pkgapp as smuser.

So, switch to either root (su -) or smuser (su - smuser).

Source the environment variable (. /apps/CA/siteminder/ca_ps_env.ksh)

 

Then run pkgapp as below:

cd /apps

/apps/binary/pkgapp/pkgapp -p 20541 -a /apps/CA/siteminder/bin/smpolicysrv -c /apps/core-smpolicysrv-2615-500-11-1451996208 -i -s /apps

 

pkgapp result

[smuser@redhatlinux apps]$ /apps/binary/pkgapp/pkgapp -p 20541 -a /apps/CA/siteminder/bin/smpolicysrv -c /apps/core-smpolicysrv-2615-500-11-1451996208 -i -s /apps

* ------------------------------------------------------------------------------

* Oracle Corporation CSD pkgapp 3.4 Linux                           [02/25/2016]

* ------------------------------------------------------------------------------

* Checking install epoch/delta           [good, 59 seconds]

* Checking pkgapp path                   [using /apps/binary/pkgapp]

* Checking for locale en_US.utf8         [success]

* Checking usage history                 [not recently run]

* OS Release [Linux]                     [2.6.32-504.el6.x86_64]

* Platform/Type                          [x86_64 x86_64]

* Checking [-p] pid                      [using pid 20541]

* Checking [-a] path                     [using /apps/CA/siteminder/bin/smpolicysrv]

* Process root                           [/apps/CA/siteminder/bin/smpolicysrv]

* Checking [-c] core                     [using core /apps/core-smpolicysrv-2615-500-11-1451996208]

* Databin parameter [-s] checks          [success]

* Databin found                          [/apps]

* Databin writable check                 [success]

* Databin used/created is                [/apps/pkgapp-022516-05]

* Creating temp area                     [/tmp/pkgapp.23249/]

* Process binary                         [smpolicysrv]

* smpolicysrv binary bit version         [32]

* core-smpolicysrv-2615-500-11-1451996208 bit version [32]

* Checking path [-a] to binary name      [success, path includes binary name]

* Checking path [-a] exists              [success]

* Locating smpolicysrv                   [success]

* Checking located smpolicysrv is 32 bits. [success]

* Binary located                         [/apps/CA/siteminder/bin/smpolicysrv]

* Adding binary to pkgapp.pldd           [success]

* Grabbing libs from /proc               [success]

* Grabbing gdb backtrace                 Missing separate debuginfo for /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/00/9c8f6ad2d8577fef41a38f1ac9e3407be3c0e6

 

 

warning: .dynamic section for "/apps/CA/siteminder/lib/libsmobjldap.so" is not at the expected address (wrong library or version mismatch?)

Missing separate debuginfo for

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/09/8872f5308ed76cfe3e97cfaea495b50cfaac38

Missing separate debuginfo for /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/00/9c8f6ad2d8577fef41a38f1ac9e3407be3c0e6.debug

[success]

* Grabbing [-i] core/gcore               [success]

* Grabbing core/gcore cksum              [success]

* Javatools [-j] not available           [skipped]

* Grabbing /var/log/messages             [success]

* Grabbing uname -a                      [success]

* Grabbing date/time                     [success]

* Grabbing rpm -qa                       [success]

* Grabbing /etc/redhat-release           [success]

* Grabbing ulimit -a                     [success]

* Grabbing libs                          [success]

* Making lib paths app/                  [success]

* Making lib paths libs/                 [success]                                                                                                                                              * Linking libraries29 of 129              [success]

* Libraries linked                       [129 ttl]

*

* Using hostid for naming .tar.gz        [007f0101]

* Writing file                           [pkgapp-007f0101-{hostname}-160225-123614.tar.gz]

*

* Done gathering files

* Writing dbxrc & opencore.sh files      [success]

* Writing validate-libs.sh file          [success]

* Writing manifest-160225-123614.log     [success]

* Writing pkgapp-args-160225-123614      [success]

* Creating final tarfile                 tar: os-info/messages/messages: Cannot open: Permission denied

tar: Exiting with failure status due to previous errors

[success]

* Compressing tarfile                    [success]

* End of runtime logging

* Saving history info                    [success]

* Saving runtime log                     [success]

* Saving Library list                    [success]

* Saving Library list README             [success]

* Removing [-r] temp area/files          [left alone]

*

* Operations Complete

* Total runtime                          [11 minutes]

* ------------------------------------------------------------------------------

* Upload the following file(s) to http://supportfiles.sun.com Cores Directory at

Oracle

 

 

1) File(s) located in directory /apps/pkgapp-022516-05

 

 

                [ pkgapp-007f0101-{hostname}-160225-123614.tar.gz ]

 

 

 

 

 

 

                                Thank you.

                                Customer Service Delivery

                                Global Systems Support

 

 

 

 

NOTES:

1) You can check for updates to this script here:

        BigAdmin - http://www.sun.com/bigadmin/scripts/indexSjs.html

        Blog - http://blogs.sun.com/Dirtracer

 

 

2) GDD information located here:

        Rel Notes - http://docs.sun.com/app/docs/doc/820-0437

        Docs - http://www.sun.com/service/gdd/index.xml

        Video - http://mediacast.sun.com/search?query=gdd

        Video - http://mediacast.sun.com/search?query=dirtracer

 

 

3) Please send all Bugs and RFE's to the following address:

        Subject "pkgapp bug/rfe" - pkgapp-feedback_ww@oracle.com

 

 

4) See the following reference files.

        /var/tmp/pkgapp-history/runtime-160225-123614.log

        /var/tmp/pkgapp-history/history-160225-123614.log

        /var/tmp/pkgapp-history/library.list-160225-123614

        /var/tmp/pkgapp-history/library.list-160225-123614.README

 

Now, let's say you received this file (/apps/pkgapp-022516-05/pkgapp-007f0101-{hostname}-160225-123614.tar.gz), what next?

(The assumption is that you do not have access to the machine that generated the core)

 

You need to extract this(tar zxvf pkgapp-007f0101-{hostname}-160225-123614.tar.gz) on a machine where you have the gdb installed.

 

If the machine that generated this core and the packaged core file had *gdb* installed, you will be able to find the backtrace information in "<pkgapp-007f0101-{hostname}-123614.tar.gz>/proctool-info/pkgapp.backtrace-xxxx-yyyyy"

 

But in most cases, the production servers will not have gdb installed so it could be missing that file.

If you find it, it will look something like the attachment (pkgapp.backtrace-xxxx-yyyyy).

Note: The attachment's extension is zip but it is actually just a text file. zip extension was appended automatically.

 

 

Now, since you did not have the above file, you need to get it yourself.

 

Extracted files (/apps/pkgapp-022516-05)

[smuser@redhatlinux pkgapp-022516-05]$ ls

app  core-lib-data  opencore.sh  os-info  pkgapp-007f0101-{hostname}-160225-123614.tar.gz  pkgapp-info  proctool-info  runtime-160225-123614.log  validate-libs.sh

 

Run the opencore.sh file (./opencore.sh)

opencore.sh

[smuser@redhatlinux pkgapp-022516-05]$ ./opencore.sh

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/bin/smpolicysrv...expanding to full symbols...(no debugging symbols found)...done.

[New Thread 2699]

[New Thread 2765]

[New Thread 2701]

[New Thread 2804]

[New Thread 2763]

[New Thread 2806]

[New Thread 2807]

[New Thread 2825]

[New Thread 2764]

[New Thread 2798]

[New Thread 2799]

[New Thread 2805]

[New Thread 2808]

[New Thread 2824]

[New Thread 2826]

[New Thread 2827]

[New Thread 2828]

[New Thread 2829]

[New Thread 2830]

[New Thread 2831]

[New Thread 2832]

[New Thread 2675]

[New Thread 2833]

[New Thread 2698]

[New Thread 2837]

[New Thread 2639]

[New Thread 2841]

[New Thread 2645]

[New Thread 2757]

[New Thread 2876]

[New Thread 2637]

[New Thread 2671]

[New Thread 2877]

[New Thread 2669]

[New Thread 2615]

[New Thread 2656]

[New Thread 2622]

[New Thread 2727]

[New Thread 2654]

[New Thread 2624]

[New Thread 2710]

[New Thread 2632]

[New Thread 2625]

[New Thread 2696]

[New Thread 2627]

[New Thread 2704]

[New Thread 2673]

[New Thread 2628]

[New Thread 2686]

[New Thread 2665]

[New Thread 2650]

[New Thread 2695]

[New Thread 2737]

[New Thread 2758]

[New Thread 2702]

[New Thread 2746]

[New Thread 2721]

[New Thread 2668]

[New Thread 2690]

[New Thread 2685]

[New Thread 2749]

[New Thread 2682]

[New Thread 2743]

[New Thread 2726]

[New Thread 2693]

[New Thread 2640]

[New Thread 2672]

[New Thread 2692]

[New Thread 2646]

[New Thread 2694]

[New Thread 2687]

[New Thread 2642]

[New Thread 2661]

[New Thread 2741]

[New Thread 2742]

[New Thread 2739]

[New Thread 2750]

[New Thread 2633]

[New Thread 2657]

[New Thread 2630]

[New Thread 2738]

[New Thread 2644]

[New Thread 2718]

[New Thread 2755]

[New Thread 2715]

[New Thread 2744]

[New Thread 2730]

[New Thread 2647]

[New Thread 2722]

[New Thread 2634]

[New Thread 2712]

[New Thread 2659]

[New Thread 2708]

[New Thread 2649]

[New Thread 2751]

[New Thread 2653]

[New Thread 2667]

[New Thread 2759]

[New Thread 2679]

[New Thread 2638]

[New Thread 2691]

[New Thread 2641]

[New Thread 2724]

[New Thread 2631]

[New Thread 2717]

[New Thread 2662]

[New Thread 2733]

[New Thread 2723]

[New Thread 2683]

[New Thread 2648]

[New Thread 2753]

[New Thread 2700]

[New Thread 2756]

[New Thread 2719]

[New Thread 2752]

[New Thread 2664]

[New Thread 2689]

[New Thread 2677]

[New Thread 2666]

[New Thread 2658]

[New Thread 2728]

[New Thread 2735]

[New Thread 2684]

[New Thread 2635]

[New Thread 2651]

[New Thread 2697]

[New Thread 2703]

[New Thread 2725]

[New Thread 2714]

[New Thread 2740]

[New Thread 2732]

[New Thread 2731]

[New Thread 2734]

[New Thread 2747]

[New Thread 2713]

[New Thread 2655]

[New Thread 2729]

[New Thread 2670]

[New Thread 2761]

[New Thread 2663]

[New Thread 2674]

[New Thread 2716]

[New Thread 2705]

[New Thread 2680]

[New Thread 2736]

[New Thread 2643]

[New Thread 2660]

[New Thread 2711]

[New Thread 2720]

[New Thread 2629]

[New Thread 2760]

[New Thread 2676]

[New Thread 2762]

[New Thread 2709]

[New Thread 2626]

[New Thread 2748]

[New Thread 2754]

[New Thread 2688]

[New Thread 2636]

[New Thread 2652]

[New Thread 2623]

[New Thread 2834]

Missing separate debuginfo for /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/00/9c8f6ad2d8577fef41a38f1ac9e3407be3c0e6

Missing separate debuginfo for

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/09/8872f5308ed76cfe3e97cfaea495b50cfaac38

Reading symbols from /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so...Missing separate debuginfo for /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/00/9c8f6ad2d8577fef41a38f1ac9e3407be3c0e6.debug

expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Reading symbols from /lib/libpthread.so.0...expanding to full symbols...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib/libpthread.so.0

Reading symbols from /lib/libdl.so.2...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libdl.so.2

Reading symbols from /lib/librt.so.1...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/librt.so.1

Reading symbols from /usr/lib/libstdc++.so.6...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libstdc++.so.6

Reading symbols from /lib/libm.so.6...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libm.so.6

Reading symbols from /lib/libgcc_s.so.1...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libgcc_s.so.1

Reading symbols from /lib/libc.so.6...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libc.so.6

Reading symbols from /lib/ld-linux.so.2...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/ld-linux.so.2

Reading symbols from /lib/libnsl.so.1...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libnsl.so.1

Reading symbols from /lib/libnss_files.so.2...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_files.so.2

Reading symbols from /lib/libnss_dns.so.2...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libnss_dns.so.2

Reading symbols from /lib/libresolv.so.2...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /lib/libresolv.so.2

Core was generated by `smpolicysrv'.

Program terminated with signal 11, Segmentation fault.

#0  0x03654b06 in __strlen_sse2_bsf () from /lib/libc.so.6

Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6.i686 libgcc-4.4.7-11.el6.i686 libstdc++-4.4.7-11.el6.i686

 

You will get brief information about which process generated the core file and why.

And you will also see lots of messages (or no messages) about symbols not found.

 

What you need to ensure is, the gdb will reference the binaries/libraries that came with the packaged core file.

At the current folder(/apps/pkgapp-022516-05) you will find the "core-lib-data" folder.

Under that folder there is "libs" folder which has the collection of files from the machine where the core file was generated.

Those are the ones that you have to tell gdb to load, not your local system files.

 

Run the following commands within gdb.

set solib-absolute-prefix /apps/pkgapp-022516-05/core-lib-data/libs

What this does is to tell gbd to prefix "/apps/pkgapp-022516-05/core-lib-data/libs" to any file reference.

So, if it wanted to look for /lib/ld-linux.so.2 then it will actually be /apps/pkgapp-022516-05/core-lib-data/libs/lib/ld-linux.so.2

Or, you can also  run "set sysroot /apps/pkgapp-022516-05/core-lib-data/libs" gdb will look for files from there.

set solib-absolute-prefix /apps/pkgapp-022516-05/core-lib-data/libs

(gdb) set solib-absolute-prefix /apps/pkgapp-022516-05/core-lib-data/libs

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmartheap_smp.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmartheap_smp.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libGCL.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libGCL.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmcommonutil.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmcommonutil.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmi18n.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmi18n.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicudata.so.49...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicudata.so.49

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicui18n.so.49...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicui18n.so.49

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuio.so.49...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuio.so.49

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuuc.so.49...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuuc.so.49

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauth.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauth.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreports.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreports.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmplatform.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmplatform.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmds.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmds.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmadmobj.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmadmobj.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaz.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaz.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauthcert.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauthcert.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjvmsupport.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjvmsupport.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmvariable.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmvariable.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmconapi.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmconapi.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmerrlog.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmerrlog.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmradius.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmradius.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libldap60.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libldap60.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssldap60.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssldap60.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssl3.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssl3.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libprldap60.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libprldap60.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnss3.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnss3.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnspr4.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnspr4.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnssutil3.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnssutil3.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplds4.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplds4.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so...Missing separate debuginfo for /apps/pkgapp-022516-05/core-lib-data/libs/apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/00/9c8f6ad2d8577fef41a38f1ac9e3407be3c0e6.debug

expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmmonapips.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmmonapips.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPS.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPS.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSAudit.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSAudit.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXLogger.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXLogger.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmXlate.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmXlate.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-c.so.28...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-c.so.28

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-depdom.so.28...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-depdom.so.28

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmazuser.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmazuser.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmpolicyapi45.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmpolicyapi45.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedconfig.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedconfig.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libaceclnt.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libaceclnt.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmShutdownManager.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmShutdownManager.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsoftokn3.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsoftokn3.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplc4.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplc4.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libz.so.1...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libz.so.1

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/lib/ld-linux.so.2...Missing separate debuginfo for /apps/pkgapp-022516-05/core-lib-data/libs/lib/ld-linux.so.2

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/cf/64bd33f05b1081feb1b9a01e9ae82ed098bd80.debug

expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/lib/ld-linux.so.2

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsqlite3.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsqlite3.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmidentity.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmidentity.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentfunccomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentfunccomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentconmgrcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentconmgrcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmservercomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmservercomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisprotectedcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisprotectedcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmloginlogoutcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmloginlogoutcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisauthorizedcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisauthorizedcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtunnelcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtunnelcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjavaapicomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjavaapicomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdirectorycomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdirectorycomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmodbccomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmodbccomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmimscomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmimscomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libtxmcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libtxmcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedservercomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedservercomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdlpcomponent.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdlpcomponent.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmgda.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmgda.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfipsmode.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfipsmode.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcapki.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcapki.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_crypto.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_crypto.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_ssl.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_ssl.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib//libcapki_thread_posix.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib//libcapki_thread_posix.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libcryptocme.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libcryptocme.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_base.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_base.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_asym.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_asym.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecdrbg.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecdrbg.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc_accel_fips.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc_accel_fips.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_error_info.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_error_info.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/lib/libnss_files.so.2...Missing separate debuginfo for /apps/pkgapp-022516-05/core-lib-data/libs/lib/libnss_files.so.2

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/1b/e2ab751dc66899f491d8050600112b35136a8a.debug

expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/lib/libnss_files.so.2

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransact.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransact.so

warning: .dynamic section for "/apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so" is not at the expected address (wrong library or version mismatch?)

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapps.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapps.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldapims.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldapims.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjims.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjims.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsds.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsds.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsutil.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsutil.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreportstextlog.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreportstextlog.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSEval.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSEval.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmprovider.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmprovider.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSLDAP.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSLDAP.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjadapter.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjadapter.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactadapter.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactadapter.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfmdeploy.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfmdeploy.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactems2.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactems2.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libmigration.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libmigration.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSoaObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSoaObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFssObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFssObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libIdMObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libIdMObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libEPMObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libEPMObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmRLS.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmRLS.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdsadapter.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdsadapter.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libCDSObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libCDSObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFedObjects.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFedObjects.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libJVMSupportAdapter.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libJVMSupportAdapter.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmCounters.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmCounters.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbc.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbc.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbcinst.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbcinst.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libNSicu27.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libNSicu27.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaps.so...expanding to full symbols...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaps.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libAPSMail.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libAPSMail.so

Reading symbols from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libbtunicode.so...expanding to full symbols...(no debugging symbols found)...done.

Loaded symbols for /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libbtunicode.so

(gdb)

 

Now, when you run the "where" command to get the back trace, if it resolves all the symbols then you should not see any "??".

Following is where it actually shows unresolved ones.

where

(gdb) where

#0  0x03654b06 in ?? ()

#1  0xc91132b7 in ?? ()

#2  0x0806ca6f in CpaStream& CpaStream::append<char const*>(char const*) ()

#3  0x0806ca46 in CpaStream::operator<<(char const*) ()

#4  0x0806c990 in CpaArgList& CpaArgList::add<char const*>(char const* const&) ()

#5  0x0806c191 in char const* CSmErrLogMessage::FormatMessage<char const*, char const*>(unsigned int, char const* const&, char const* const&) ()

#6  0x03acd10b in SmBindLDAP () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#7  0x03acf5ba in SmImproveConnection(int) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#8  0x03ad0513 in SmSearchLDAP () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#9  0x03a17bb8 in AgentKey_Search () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#10 0x00a42367 in CSmObjProvider::Fetch(CSmObjBase&) const () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

#11 0x00a8c903 in CSmObjStore::Fetch(CSmObjBase&, unsigned int, void const*, CString const&) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

#12 0x08070a0d in CSm_Auth_Message::DoManagement() ()

#13 0x080668ae in CSm_Auth_Message::ProcessAgentMessage() ()

#14 0x080bc6f8 in CSm_Auth_Message::ProcessMessage() ()

#15 0x08125a15 in CSmPolicyServer::vOnRequest(CClientSession const*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) ()

#16 0x00e1632d in CServer::ProcessRequest(CClientSession*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) ()

   from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#17 0x00df0790 in CAgentMessageHandler::DoWork(unsigned char*, unsigned char*, int) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#18 0x00de855f in ThreadPool::Run(bool) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#19 0x00e96ca4 in ThreadPoolBase::ThreadProc(void*) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#20 0x00173f05 in BtThreadBase(ThreadArgs*) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmcommonutil.so

#21 0x00836b39 in ?? ()

#22 0x090640c0 in ?? ()

#23 0x036b6c1e in ?? ()

 

Although gdb was told to lookup the /apps/pkgapp-022516-05/core-lib-data/libs folder, problem is it is not finding some files in that folder structure.

 

You can get a list of libraries and see which one is missing.

 

info sharedlibrary

(gdb) info sharedlibrary

From        To          Syms Read   Shared Object Library

0x004256f0  0x004313c6  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmartheap_smp.so

0x00dc61e0  0x00f383c0  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

0x0085ff94  0x008666ac  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libGCL.so

0x00156558  0x0018c3b0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmcommonutil.so

0x007d564c  0x0081278c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmi18n.so

0x00f9c1fc  0x00f9c1fc  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicudata.so.49

0x0021a6c0  0x0035d7e4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicui18n.so.49

0x004af414  0x004b5654  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuio.so.49

0x004f0b4c  0x005b3ff0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libicuuc.so.49

0x009c8408  0x00ab3388  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

0x0064b594  0x00722524  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauth.so

0x003b3b80  0x003c257c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreports.so

0x003d7890  0x003e96ec  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmplatform.so

0x00886360  0x008dc94c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmds.so

0x0044fb0c  0x0047ab2c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmadmobj.so

0x00b27184  0x00b7df24  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaz.so

0x00bba300  0x00cadf60  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmauthcert.so

0x003f8784  0x00400bac  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjvmsupport.so

0x0075f628  0x0076fb70  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmvariable.so

0x00789f54  0x0079a200  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmconapi.so

0x020cb158  0x020df43c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmerrlog.so

0x008fc4d0  0x00912a4c  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmradius.so

0x00cdd204  0x00cfd2e0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libldap60.so

0x004052f4  0x0040717c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssldap60.so

0x00d06718  0x00d276b4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libssl3.so

0x0040d650  0x0040f23c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libprldap60.so

0x08269ae4  0x083330a0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnss3.so

0x02130dbc  0x0214ffd4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnspr4.so

0x00493e24  0x0049e278  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libnssutil3.so

0x00411bb0  0x0041298c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplds4.so

0x022324b0  0x02966c68  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/jdk1.7.0_80/jre/lib/i386/server/libjvm.so

0x00419298  0x0041e564  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmmonapips.so

                       No          /lib/libpthread.so.0

                        No          /lib/libdl.so.2

                        No          /lib/librt.so.1

0x07339f40  0x07534cac  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPS.so

0x02e7eae8  0x02eb3690  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSAudit.so

0x084accec  0x084b9f64  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXLogger.so

0x031f3abc  0x0320ddb4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmXlate.so

0x03337194  0x034c4344  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-c.so.28

0x03039728  0x03063560  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libxerces-depdom.so.28

0x040fafac  0x0413b3b4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmazuser.so

0x02ee4a44  0x02f7bd20  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmpolicyapi45.so

0x0092efc4  0x009353e4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedconfig.so

0x048b87a0  0x048f6c94  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libaceclnt.so

0x0084ed48  0x00850f24  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmShutdownManager.so

                       No          /usr/lib/libstdc++.so.6

                        No          /lib/libm.so.6

                        No          /lib/libgcc_s.so.1

                        No          /lib/libc.so.6

0x06726f98  0x0674dafc  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsoftokn3.so

0x00853f6c  0x008555e8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libplc4.so

0x02fbf6d4  0x02fcce60  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libz.so.1

0x00942830  0x0095a47f  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/lib/ld-linux.so.2

                       No          /lib/libnsl.so.1

0x054eed10  0x0553eeb4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsqlite3.so

0x004aab48  0x004ab304  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmidentity.so

0x00918bf8  0x00919084  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentfunccomponent.so

---Type <return> to continue, or q <return> to quit---

0x0091bbc4  0x0091c050  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmagentconmgrcomponent.so

0x0092002c  0x0092064c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmservercomponent.so

0x00924ef0  0x009254d8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisprotectedcomponent.so

0x0093b6f8  0x0093bd18  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmloginlogoutcomponent.so

0x0093ffe8  0x00940604  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmisauthorizedcomponent.so

0x00d2bf5c  0x00d2c578  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtunnelcomponent.so

0x00d2f2d0  0x00d2f8f0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmjavaapicomponent.so

0x088ea184  0x088ea7a0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdirectorycomponent.so

0x03f6a048  0x03f6a664  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmodbccomponent.so

0x02fef5c8  0x02fefbe8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapcomponent.so

0x05460540  0x05460b60  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmimscomponent.so

0x05814fe8  0x05815604  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libtxmcomponent.so

0x0782de98  0x0782e324  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmfedservercomponent.so

0x046faf10  0x046fb52c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdlpcomponent.so

0x083f1b38  0x083f1fd0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmgda.so

0x02ff2b04  0x02ff2f28  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfipsmode.so

0x06ce2540  0x06db9e94  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcapki.so

0x03093340  0x031892a4  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_crypto.so

0x06ba13c0  0x06c020d4  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib/libcaopenssl_ssl.so

0x05fd3d40  0x05fd5694  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI/Linux/x86/32/lib//libcapki_thread_posix.so

0x038e87f0  0x03918fe8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libcryptocme.so

0x07e12a30  0x07e3df18  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_base.so

0x0376cf80  0x0379a3cc  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_asym.so

0x037a46d0  0x0380001c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc.so

0x03829e10  0x0383eb28  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecdrbg.so

0x03920510  0x03920658  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_ecc_accel_fips.so

0x02ff57cc  0x02ff591c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/CAPKI//Linux/x86/32/lib/libccme_error_info.so

0x02ff9aa0  0x03001e98  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/lib/libnss_files.so.2

0x05e44d9c  0x05e4f358  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransact.so

0x03a0abc4  0x03ae1f28  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

0x058c8374  0x058eb744  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmldapps.so

0x03d3721c  0x03e4f780  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldapims.so

0x041c0e78  0x04509ed4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjims.so

0x03b2a9fc  0x03b6095c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsds.so

0x03bb82b4  0x03bf90d0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libimsutil.so

                        No          /lib/libnss_dns.so.2

                        No          /lib/libresolv.so.2

0x038adb60  0x038b636c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmreportstextlog.so

0x0474eb40  0x0483efa4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSEval.so

0x0300c4f0  0x03010810  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmprovider.so

0x03c30ff8  0x03c5c930  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libXPSLDAP.so

0x07c27d9c  0x07c99afc  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmObjects.so

0x03ff2dd0  0x040854a8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjadapter.so

0x03c81c90  0x03c98fbc  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactadapter.so

0x07dff4a8  0x07dff5a8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libfmdeploy.so

0x05580638  0x05645688  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmtransactems2.so

0x03cbfc30  0x03ce21ec  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libmigration.so

0x038c0624  0x038c4da0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSoaObjects.so

0x066d68ac  0x066fa644  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFssObjects.so

0x038d0bac  0x038d7b68  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libIdMObjects.so

0x08d5269c  0x08d6b9a8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libEPMObjects.so

0x03cf6d38  0x03cfb644  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmRLS.so

0x03ea0bbc  0x03eb94e8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmdsadapter.so

0x038e0e54  0x038e236c  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libCDSObjects.so

0x056cb370  0x05777ce8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libFedObjects.so

0x08bf07b8  0x08bf0924  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libJVMSupportAdapter.so

0x031c17cc  0x031c2cd0  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libSmCounters.so

0x0592ac40  0x059f78f4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbc.so

0x0460aa40  0x0467fac4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libodbcinst.so

---Type <return> to continue, or q <return> to quit---

0x9b32f3e0  0x9b3d8bb4  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/odbc/lib/libNSicu27.so

0x05a3f760  0x05b521ac  Yes         /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmaps.so

0x03ec7e20  0x03ede3f8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libAPSMail.so

0x05c385cc  0x05cc12a8  Yes (*)     /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libbtunicode.so

                       No          /apps/CA/siteminder/lib/libsmauthhtml.so

                        No          /apps/CA/siteminder/lib/libsmdsldap.so

                        No          /apps/CA/siteminder/lib/libsmdsplugin_Sun.so

(*): Shared library is missing debugging information.

 

You can see what files are not collected by the pkgapp script.

Those files must be collected from the machine that generated the core.

You need to copy them to the /apps/pkgapp-022516-05/core-lib/data/libs/xxxxxxx folder to match the structure.

For example, /apps/CA/siteminder/lib/libsmdsplugin_Sun.so was missing.

You must copy this to "/apps/pkgapp-022516-05/core-lib/data/libs/apps/CA/siteminder/lib/" folder.

 

Quit the gdb(quit) and run opencore.sh again.

Run "set solib-absolute-prefix /apps/pkgapp-022516-05/core-lib-data/libs" or "set sysroot /apps/pkgapp-022516-05/core-lib-data/libs" again.

Then run the "info sharedlibrary" to confirm all library files are available.

 

Then run the "where" again.

where

(gdb) where

#0  0x03654b06 in __strlen_sse2_bsf () from /lib/libc.so.6

#1  0x076e58f3 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator+=(char const*) () from /usr/lib/libstdc++.so.6

#2  0x0806ca6f in CpaStream& CpaStream::append<char const*>(char const*) ()

#3  0x0806ca46 in CpaStream::operator<<(char const*) ()

#4  0x0806c990 in CpaArgList& CpaArgList::add<char const*>(char const* const&) ()

#5  0x0806c191 in char const* CSmErrLogMessage::FormatMessage<char const*, char const*>(unsigned int, char const* const&, char const* const&) ()

#6  0x03acd10b in SmBindLDAP () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#7  0x03acf5ba in SmImproveConnection(int) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#8  0x03ad0513 in SmSearchLDAP () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#9  0x03a17bb8 in AgentKey_Search () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobjldap.so

#10 0x00a42367 in CSmObjProvider::Fetch(CSmObjBase&) const () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

#11 0x00a8c903 in CSmObjStore::Fetch(CSmObjBase&, unsigned int, void const*, CString const&) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmobj.so

#12 0x08070a0d in CSm_Auth_Message::DoManagement() ()

#13 0x080668ae in CSm_Auth_Message::ProcessAgentMessage() ()

#14 0x080bc6f8 in CSm_Auth_Message::ProcessMessage() ()

#15 0x08125a15 in CSmPolicyServer::vOnRequest(CClientSession const*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) ()

#16 0x00e1632d in CServer::ProcessRequest(CClientSession*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) ()

   from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#17 0x00df0790 in CAgentMessageHandler::DoWork(unsigned char*, unsigned char*, int) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#18 0x00de855f in ThreadPool::Run(bool) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#19 0x00e96ca4 in ThreadPoolBase::ThreadProc(void*) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmutilities.so

#20 0x00173f05 in BtThreadBase(ThreadArgs*) () from /apps/pkgapp-022516-05/core-lib-data/libs/apps/CA/siteminder/lib/libsmcommonutil.so

#21 0x00836b39 in start_thread () from /lib/libpthread.so.0

#22 0x036b6c1e in clone () from /lib/libc.so.6

(gdb)

 

This back trace is what need to be reviewed.

Running "bt" also displays the thread that caused the crash. (It will actually display the current *selected* thread, by default thread #1 (not LWP #1) is selected and is the thread that caused the crash *in general*) But "where" should be the one to run as that will display the thread that caused the crash.

If you have debug symbol files, running "where full" or "bt full" would give you more information.

 

Another useful information would be to know what other threads were doing at that moment.

Following command will give you the list of all threads.

The crashing thread would usually be at the very bottom.

info threads

(gdb) info threads

  162 Thread 0x958f0b70 (LWP 2834)  0x00927430 in __kernel_vsyscall ()

  161 Thread 0xf755db70 (LWP 2623)  0x00927430 in __kernel_vsyscall ()

  160 Thread 0xe5340b70 (LWP 2652)  0x00927430 in __kernel_vsyscall ()

  159 Thread 0xef350b70 (LWP 2636)  0x00927430 in __kernel_vsyscall ()

  158 Thread 0xcff1eb70 (LWP 2688)  0x00927430 in __kernel_vsyscall ()

  157 Thread 0xa88dfb70 (LWP 2754)  0x00927430 in __kernel_vsyscall ()

  156 Thread 0xac4e5b70 (LWP 2748)  0x00927430 in __kernel_vsyscall ()

  155 Thread 0xf575ab70 (LWP 2626)  0x00927430 in __kernel_vsyscall ()

  154 Thread 0xc410bb70 (LWP 2709)  0x00927430 in __kernel_vsyscall ()

  153 Thread 0xa38d7b70 (LWP 2762)  0x00927430 in __kernel_vsyscall ()

  152 Thread 0xd6328b70 (LWP 2676)  0x00927430 in __kernel_vsyscall ()

  151 Thread 0xa4cd9b70 (LWP 2760)  0x00927430 in __kernel_vsyscall ()

  150 Thread 0xf3957b70 (LWP 2629)  0x00927430 in __kernel_vsyscall ()

  149 Thread 0xbd300b70 (LWP 2720)  0x00927430 in __kernel_vsyscall ()

  148 Thread 0xc2d09b70 (LWP 2711)  0x00927430 in __kernel_vsyscall ()

  147 Thread 0xe0338b70 (LWP 2660)  0x00927430 in __kernel_vsyscall ()

  146 Thread 0xead49b70 (LWP 2643)  0x00927430 in __kernel_vsyscall ()

  145 Thread 0xb32f0b70 (LWP 2736)  0x00927430 in __kernel_vsyscall ()

  144 Thread 0xd4525b70 (LWP 2680)  0x00927430 in __kernel_vsyscall ()

  143 Thread 0xc550db70 (LWP 2705)  0x00927430 in __kernel_vsyscall ()

  142 Thread 0xbfb04b70 (LWP 2716)  0x00927430 in __kernel_vsyscall ()

  141 Thread 0xd772ab70 (LWP 2674)  0x00927430 in __kernel_vsyscall ()

  140 Thread 0xde535b70 (LWP 2663)  0x00927430 in __kernel_vsyscall ()

  139 Thread 0xa42d8b70 (LWP 2761)  0x00927430 in __kernel_vsyscall ()

  138 Thread 0xd9f2eb70 (LWP 2670)  0x00927430 in __kernel_vsyscall ()

  137 Thread 0xb78f7b70 (LWP 2729)  0x00927430 in __kernel_vsyscall ()

  136 Thread 0xe353db70 (LWP 2655)  0x00927430 in __kernel_vsyscall ()

  135 Thread 0xc1907b70 (LWP 2713)  0x00927430 in __kernel_vsyscall ()

  134 Thread 0xacee6b70 (LWP 2747)  0x00927430 in __kernel_vsyscall ()

  133 Thread 0xb46f2b70 (LWP 2734)  0x00927430 in __kernel_vsyscall ()

  132 Thread 0xb64f5b70 (LWP 2731)  0x00927430 in __kernel_vsyscall ()

  131 Thread 0xb5af4b70 (LWP 2732)  0x00927430 in __kernel_vsyscall ()

  130 Thread 0xb0aecb70 (LWP 2740)  0x00927430 in __kernel_vsyscall ()

  129 Thread 0xc0f06b70 (LWP 2714)  0x00927430 in __kernel_vsyscall ()

  128 Thread 0xba0fbb70 (LWP 2725)  0x00927430 in __kernel_vsyscall ()

  127 Thread 0xc690fb70 (LWP 2703)  0x00927430 in __kernel_vsyscall ()

  126 Thread 0xca515b70 (LWP 2697)  0x00927430 in __kernel_vsyscall ()

  125 Thread 0xe5d41b70 (LWP 2651)  0x00927430 in __kernel_vsyscall ()

  124 Thread 0xefd51b70 (LWP 2635)  0x00927430 in __kernel_vsyscall ()

  123 Thread 0xd2722b70 (LWP 2684)  0x00927430 in __kernel_vsyscall ()

  122 Thread 0xb3cf1b70 (LWP 2735)  0x00927430 in __kernel_vsyscall ()

  121 Thread 0xb82f8b70 (LWP 2728)  0x00927430 in __kernel_vsyscall ()

  120 Thread 0xe173ab70 (LWP 2658)  0x00927430 in __kernel_vsyscall ()

  119 Thread 0xdc732b70 (LWP 2666)  0x00927430 in __kernel_vsyscall ()

  118 Thread 0xd5927b70 (LWP 2677)  0x00927430 in __kernel_vsyscall ()

  117 Thread 0xcf51db70 (LWP 2689)  0x00927430 in __kernel_vsyscall ()

  116 Thread 0xddb34b70 (LWP 2664)  0x00927430 in __kernel_vsyscall ()

  115 Thread 0xa9ce1b70 (LWP 2752)  0x00927430 in __kernel_vsyscall ()

  114 Thread 0xbdd01b70 (LWP 2719)  0x00927430 in __kernel_vsyscall ()

  113 Thread 0xa74ddb70 (LWP 2756)  0x00927430 in __kernel_vsyscall ()

  112 Thread 0xc8712b70 (LWP 2700)  0x00927430 in __kernel_vsyscall ()

  111 Thread 0xa92e0b70 (LWP 2753)  0x00927430 in __kernel_vsyscall ()

  110 Thread 0xe7b44b70 (LWP 2648)  0x00927430 in __kernel_vsyscall ()

  109 Thread 0xd3123b70 (LWP 2683)  0x00927430 in __kernel_vsyscall ()

  108 Thread 0xbb4fdb70 (LWP 2723)  0x00927430 in __kernel_vsyscall ()

  107 Thread 0xb50f3b70 (LWP 2733)  0x00927430 in __kernel_vsyscall ()

  106 Thread 0xdef36b70 (LWP 2662)  0x00927430 in __kernel_vsyscall ()

  105 Thread 0xbf103b70 (LWP 2717)  0x00927430 in __kernel_vsyscall ()

  104 Thread 0xf2555b70 (LWP 2631)  0x00927430 in __kernel_vsyscall ()

---Type <return> to continue, or q <return> to quit---

  103 Thread 0xbaafcb70 (LWP 2724)  0x00927430 in __kernel_vsyscall ()

  102 Thread 0xec14bb70 (LWP 2641)  0x00927430 in __kernel_vsyscall ()

  101 Thread 0xce11bb70 (LWP 2691)  0x00927430 in __kernel_vsyscall ()

  100 Thread 0xedf4eb70 (LWP 2638)  0x00927430 in __kernel_vsyscall ()

  99 Thread 0xd4f26b70 (LWP 2679)  0x00927430 in __kernel_vsyscall ()

  98 Thread 0xa56dab70 (LWP 2759)  0x00927430 in __kernel_vsyscall ()

  97 Thread 0xdbd31b70 (LWP 2667)  0x00927430 in __kernel_vsyscall ()

  96 Thread 0xe493fb70 (LWP 2653)  0x00927430 in __kernel_vsyscall ()

  95 Thread 0xaa6e2b70 (LWP 2751)  0x00927430 in __kernel_vsyscall ()

  94 Thread 0xe7143b70 (LWP 2649)  0x00927430 in __kernel_vsyscall ()

  93 Thread 0xc4b0cb70 (LWP 2708)  0x00927430 in __kernel_vsyscall ()

  92 Thread 0xe0d39b70 (LWP 2659)  0x00927430 in __kernel_vsyscall ()

  91 Thread 0xc2308b70 (LWP 2712)  0x00927430 in __kernel_vsyscall ()

  90 Thread 0xf0752b70 (LWP 2634)  0x00927430 in __kernel_vsyscall ()

  89 Thread 0xbbefeb70 (LWP 2722)  0x00927430 in __kernel_vsyscall ()

  88 Thread 0xe8545b70 (LWP 2647)  0x00927430 in __kernel_vsyscall ()

  87 Thread 0xb6ef6b70 (LWP 2730)  0x00927430 in __kernel_vsyscall ()

  86 Thread 0xae2e8b70 (LWP 2744)  0x00927430 in __kernel_vsyscall ()

  85 Thread 0xc0505b70 (LWP 2715)  0x00927430 in __kernel_vsyscall ()

  84 Thread 0xa7edeb70 (LWP 2755)  0x00927430 in __kernel_vsyscall ()

  83 Thread 0xbe702b70 (LWP 2718)  0x00927430 in __kernel_vsyscall ()

  82 Thread 0xea348b70 (LWP 2644)  0x00927430 in __kernel_vsyscall ()

  81 Thread 0xb1eeeb70 (LWP 2738)  0x00927430 in __kernel_vsyscall ()

  80 Thread 0xf2f56b70 (LWP 2630)  0x00927430 in __kernel_vsyscall ()

  79 Thread 0xe213bb70 (LWP 2657)  0x00927430 in __kernel_vsyscall ()

  78 Thread 0xf1153b70 (LWP 2633)  0x00927430 in __kernel_vsyscall ()

  77 Thread 0xab0e3b70 (LWP 2750)  0x00927430 in __kernel_vsyscall ()

  76 Thread 0xb14edb70 (LWP 2739)  0x00927430 in __kernel_vsyscall ()

  75 Thread 0xaf6eab70 (LWP 2742)  0x00927430 in __kernel_vsyscall ()

  74 Thread 0xb00ebb70 (LWP 2741)  0x00927430 in __kernel_vsyscall ()

  73 Thread 0xdf937b70 (LWP 2661)  0x00927430 in __kernel_vsyscall ()

  72 Thread 0xeb74ab70 (LWP 2642)  0x00927430 in __kernel_vsyscall ()

  71 Thread 0xd091fb70 (LWP 2687)  0x00927430 in __kernel_vsyscall ()

  70 Thread 0xcc318b70 (LWP 2694)  0x00927430 in __kernel_vsyscall ()

  69 Thread 0xe8f46b70 (LWP 2646)  0x00927430 in __kernel_vsyscall ()

  68 Thread 0xcd71ab70 (LWP 2692)  0x00927430 in __kernel_vsyscall ()

  67 Thread 0xd8b2cb70 (LWP 2672)  0x00927430 in __kernel_vsyscall ()

  66 Thread 0xecb4cb70 (LWP 2640)  0x00927430 in __kernel_vsyscall ()

  65 Thread 0xccd19b70 (LWP 2693)  0x00927430 in __kernel_vsyscall ()

  64 Thread 0xb96fab70 (LWP 2726)  0x00927430 in __kernel_vsyscall ()

  63 Thread 0xaece9b70 (LWP 2743)  0x00927430 in __kernel_vsyscall ()

  62 Thread 0xd3b24b70 (LWP 2682)  0x00927430 in __kernel_vsyscall ()

  61 Thread 0xabae4b70 (LWP 2749)  0x00927430 in __kernel_vsyscall ()

  60 Thread 0xd1d21b70 (LWP 2685)  0x00927430 in __kernel_vsyscall ()

  59 Thread 0xceb1cb70 (LWP 2690)  0x00927430 in __kernel_vsyscall ()

  58 Thread 0xdb330b70 (LWP 2668)  0x00927430 in __kernel_vsyscall ()

  57 Thread 0xbc8ffb70 (LWP 2721)  0x00927430 in __kernel_vsyscall ()

  56 Thread 0xad8e7b70 (LWP 2746)  0x00927430 in __kernel_vsyscall ()

  55 Thread 0xc7310b70 (LWP 2702)  0x00927430 in __kernel_vsyscall ()

  54 Thread 0xa60dbb70 (LWP 2758)  0x00927430 in __kernel_vsyscall ()

  53 Thread 0xb28efb70 (LWP 2737)  0x00927430 in __kernel_vsyscall ()

  52 Thread 0xcb917b70 (LWP 2695)  0x00927430 in __kernel_vsyscall ()

  51 Thread 0xe6742b70 (LWP 2650)  0x00927430 in __kernel_vsyscall ()

  50 Thread 0xdd133b70 (LWP 2665)  0x00927430 in __kernel_vsyscall ()

  49 Thread 0xd1320b70 (LWP 2686)  0x00927430 in __kernel_vsyscall ()

  48 Thread 0xf4358b70 (LWP 2628)  0x00927430 in __kernel_vsyscall ()

  47 Thread 0xd812bb70 (LWP 2673)  0x00927430 in __kernel_vsyscall ()

  46 Thread 0xc5f0eb70 (LWP 2704)  0x00927430 in __kernel_vsyscall ()

  45 Thread 0xf4d59b70 (LWP 2627)  0x00927430 in __kernel_vsyscall ()

---Type <return> to continue, or q <return> to quit---

  44 Thread 0xcaf16b70 (LWP 2696)  0x00927430 in __kernel_vsyscall ()

  43 Thread 0xf615bb70 (LWP 2625)  0x00927430 in __kernel_vsyscall ()

  42 Thread 0xf1b54b70 (LWP 2632)  0x00927430 in __kernel_vsyscall ()

  41 Thread 0xc370ab70 (LWP 2710)  0x00927430 in __kernel_vsyscall ()

  40 Thread 0xf6b5cb70 (LWP 2624)  0x00927430 in __kernel_vsyscall ()

  39 Thread 0xe3f3eb70 (LWP 2654)  0x00927430 in __kernel_vsyscall ()

  38 Thread 0xb8cf9b70 (LWP 2727)  0x00927430 in __kernel_vsyscall ()

  37 Thread 0x5310b70 (LWP 2622)  0x00927430 in __kernel_vsyscall ()

  36 Thread 0xe2b3cb70 (LWP 2656)  0x00927430 in __kernel_vsyscall ()

  35 Thread 0xf775f6e0 (LWP 2615)  0x00927430 in __kernel_vsyscall ()

  34 Thread 0xda92fb70 (LWP 2669)  0x00927430 in __kernel_vsyscall ()

  33 Thread 0x930ecb70 (LWP 2877)  0x00927430 in __kernel_vsyscall ()

  32 Thread 0xd952db70 (LWP 2671)  0x00927430 in __kernel_vsyscall ()

  31 Thread 0xee94fb70 (LWP 2637)  0x00927430 in __kernel_vsyscall ()

  30 Thread 0x93aedb70 (LWP 2876)  0x00927430 in __kernel_vsyscall ()

  29 Thread 0xa6adcb70 (LWP 2757)  0x00927430 in __kernel_vsyscall ()

  28 Thread 0xe9947b70 (LWP 2645)  0x00927430 in __kernel_vsyscall ()

  27 Thread 0x944eeb70 (LWP 2841)  0x00927430 in __kernel_vsyscall ()

  26 Thread 0xed54db70 (LWP 2639)  0x00927430 in __kernel_vsyscall ()

  25 Thread 0x94eefb70 (LWP 2837)  0x00927430 in __kernel_vsyscall ()

  24 Thread 0xc9b14b70 (LWP 2698)  0x00927430 in __kernel_vsyscall ()

  23 Thread 0x962f1b70 (LWP 2833)  0x00927430 in __kernel_vsyscall ()

  22 Thread 0xd6d29b70 (LWP 2675)  0x00927430 in __kernel_vsyscall ()

  21 Thread 0x96cf2b70 (LWP 2832)  0x00927430 in __kernel_vsyscall ()

  20 Thread 0x976f3b70 (LWP 2831)  0x00927430 in __kernel_vsyscall ()

  19 Thread 0x980f4b70 (LWP 2830)  0x00927430 in __kernel_vsyscall ()

  18 Thread 0x98af5b70 (LWP 2829)  0x00927430 in __kernel_vsyscall ()

  17 Thread 0x994f6b70 (LWP 2828)  0x00927430 in __kernel_vsyscall ()

  16 Thread 0x99ef7b70 (LWP 2827)  0x00927430 in __kernel_vsyscall ()

  15 Thread 0x9a8f8b70 (LWP 2826)  0x00927430 in __kernel_vsyscall ()

  14 Thread 0x9caccb70 (LWP 2824)  0x00927430 in __kernel_vsyscall ()

  13 Thread 0x9d4cdb70 (LWP 2808)  0x00927430 in __kernel_vsyscall ()

  12 Thread 0x9f2d0b70 (LWP 2805)  0x00927430 in __kernel_vsyscall ()

  11 Thread 0xa06d2b70 (LWP 2799)  0x00927430 in __kernel_vsyscall ()

  10 Thread 0xa10d3b70 (LWP 2798)  0x00927430 in __kernel_vsyscall ()

  9 Thread 0xa24d5b70 (LWP 2764)  0x00927430 in __kernel_vsyscall ()

  8 Thread 0x9b2f9b70 (LWP 2825)  0x00927430 in __kernel_vsyscall ()

  7 Thread 0x9deceb70 (LWP 2807)  0x00927430 in __kernel_vsyscall ()

  6 Thread 0x9e8cfb70 (LWP 2806)  0x00927430 in __kernel_vsyscall ()

  5 Thread 0xa2ed6b70 (LWP 2763)  0x00927430 in __kernel_vsyscall ()

  4 Thread 0x9fcd1b70 (LWP 2804)  0x00927430 in __kernel_vsyscall ()

  3 Thread 0xc7d11b70 (LWP 2701)  0x00927430 in __kernel_vsyscall ()

  2 Thread 0xa1ad4b70 (LWP 2765)  0x00927430 in __kernel_vsyscall ()

* 1 Thread 0xc9113b70 (LWP 2699)  0x03654b06 in __strlen_sse2_bsf () from /lib/libc.so.6

 

If you have smtracedefault.log, then you can track the LWP 2699 which has hexadecimal Thread ID 0xc9113b70 and see what it was doing, if it logged sufficient message before the crash.

 

Load calculator in "Programmer" mode.

Select "Hex" button then enter the hexadecimal ThreadID value c9113b70

Then select "Dec" button to convert it to numeric value.

ThreadID 3373349744 in your smtracedefault.log need to be compared with this stack trace to get to the root cause of crash.

 

Sometimes(eg, stack overflow), the dump file do not capture all information(actually it can get overwritten(aka stomping) with garbage data) and makes itself useless. (At least it is very common when gdb reports corrupt stack, and it will tell you if it is)

 

So, it is always safer to collect as many (packaged) core files if at all possible.

And core dump analysis may not always give all information to find the root cause.

Having smtracedefault.log with sufficient component and data always fills the gap.

 

pkgapp and gdb are not developed by CA. Please contact the people who developed them if you need more information.

Outcomes