Anyone seen bug "Crash occurred in function FREEBL_GetVector from module /opt/local/sys/websvr7/lib

3579954

Hi is anyone aware of this bug?

In Oracle iPlanet webserver

Oracle iPlanet Web Server 7.0.27 B11/06/2017 14:24 (SunOS DOMESTIC)

on host

bash-3.2$ uname -a

SunOS intpd1web3 5.10 Generic_150400-59 sun4v sparc sun4v

bash-3.2$ cat /etc/release

                   Oracle Solaris 10 1/13 s10s_u11wos_24a SPARC

  Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.

                            Assembled 17 January 2013

We are getting the following crash  of the https listener process on port 443. No issues with the http listener (port 80)...

[17/May/2019:14:07:22] catastrophe (22775): CORE3260: Server crash detected (signal SIGBUS)

[17/May/2019:14:07:22] info (22775): CORE3261: Crash occurred in NSAPI SAF send-file

[17/May/2019:14:07:22] info (22775): CORE3262: Crash occurred in function FREEBL_GetVector from module /opt/local/sys/websvr7/lib/sparcv9/libfreebl_64fpu_3.so

[17/May/2019:14:07:51] info (29391): CORE3050: Spawning replacement for worker process 22775

[17/May/2019:14:07:51] info (24694): HTTP3072: http-listener-1: https://163.189.218.46:443 ready to accept requests

[17/May/2019:14:07:51] info (24694): CORE3274: successful server startup

We see the crash frequency increase at higher https: connections rates and decrease again at lower connection rates.

Https Listener configured for (TLS 1.2/1.1/1.0) handshake.

This has been happening for several months since we added a https listener to this site for secure connections; to mitigate Chrome marking the site as insecure.

We don't see this issue on our other low volume sites on the same host.

We migrated the virtual webserver to a dedicated https listener in its own iplanet configuration on the same host to isolate the issue.

We suspected it was the higher volume site that had the issue and the issue did move to the fresh install.

Our vendor supplied Oracle the crash dump for analysis but Oracle were unable to assist since the product is now End of Life.

In the last 18 hour period we have had 515 crashes on the production webserver

bash-3.2$ grep -c catastrophe  errors

515

23548:[17/May/2019:18:28:22] catastrophe (18153): CORE3260: Server crash detected (signal SIGBUS)

23588:[17/May/2019:18:29:19] catastrophe (19287): CORE3260: Server crash detected (signal SIGBUS)

23614:[17/May/2019:18:30:17] catastrophe (20056): CORE3260: Server crash detected (signal SIGBUS)

23655:[17/May/2019:18:31:24] catastrophe (20959): CORE3260: Server crash detected (signal SIGBUS)

23727:[17/May/2019:18:33:20] catastrophe (21715): CORE3260: Server crash detected (signal SIGBUS)

23784:[17/May/2019:18:34:34] catastrophe (23669): CORE3260: Server crash detected (signal SIGBUS)

23806:[17/May/2019:18:35:16] catastrophe (24544): CORE3260: Server crash detected (signal SIGBUS)

23874:[17/May/2019:18:36:20] catastrophe (25357): CORE3260: Server crash detected (signal SIGBUS)

23901:[17/May/2019:18:37:22] catastrophe (26161): CORE3260: Server crash detected (signal SIGBUS)

23936:[17/May/2019:18:39:18] catastrophe (27293): CORE3260: Server crash detected (signal SIGBUS)

.

.

OS and webserver install are both 64 bit.

Webserver library mentioned in the crash libfreebl_64fpu_3.so is installed in two location on the server; however file sizes are identical:-

bash-3.2$ ls -al /opt/local/sys/websvr7/lib/sparcv9/libfree*

-rw-r--r--   1 root     root         899 Dec 19  2017 /opt/local/sys/websvr7/lib/sparcv9/libfreebl_64fpu_3.chk

-rwxr-xr-x   1 root     root      696168 Dec 19  2017 /opt/local/sys/websvr7/lib/sparcv9/libfreebl_64fpu_3.so

-rw-r--r--   1 root     root         899 Dec 19  2017 /opt/local/sys/websvr7/lib/sparcv9/libfreebl_64int_3.chk

-rwxr-xr-x   1 root     root      579760 Dec 19  2017 /opt/local/sys/websvr7/lib/sparcv9/libfreebl_64int_3.so

bash-3.2$ ls -al /usr/lib/mps/sparcv9/libfree*

-rwxr-xr-x   1 root     bin          899 Aug  8  2017 /usr/lib/mps/sparcv9/libfreebl_64fpu_3.chk

-rwxr-xr-x   1 root     bin       696168 Aug  8  2017 /usr/lib/mps/sparcv9/libfreebl_64fpu_3.so

-rwxr-xr-x   1 root     bin          899 Aug  8  2017 /usr/lib/mps/sparcv9/libfreebl_64int_3.chk

-rwxr-xr-x   1 root     bin       579760 Aug  8  2017 /usr/lib/mps/sparcv9/libfreebl_64int_3.so

Full listing of the directories are shown below:-

bash-3.2$ ls -al /opt/local/sys/websvr7/lib/sparcv9/

total 89864

drwxr-xr-x   2 root     root        1536 Dec 19  2017 .

drwxr-xr-x  13 root     root        3584 Dec 19  2017 ..

-rwxr-xr-x   1 root     root      235856 Dec 19  2017 certutil

-rwxr-xr-x   1 root     root       44584 Dec 19  2017 Cgistub

-rwxr-xr-x   1 root     root       52136 Dec 19  2017 libares3.so

-rwxr-xr-x   1 root     root      413152 Dec 19  2017 libdavplugin.so

-rw-r--r--   1 root     root         899 Dec 19  2017 libfreebl_64fpu_3.chk

-rwxr-xr-x   1 root     root      696168 Dec 19  2017 libfreebl_64fpu_3.so

-rw-r--r--   1 root     root         899 Dec 19  2017 libfreebl_64int_3.chk

-rwxr-xr-x   1 root     root      579760 Dec 19  2017 libfreebl_64int_3.so

-rwxr-xr-x   1 root     root       13304 Dec 19  2017 libgetprop.so

-rwxr-xr-x   1 root     root     9917472 Dec 19  2017 libicudata.so.3

-rwxr-xr-x   1 root     root     1711240 Dec 19  2017 libicui18n.so.3

-rwxr-xr-x   1 root     root     1400200 Dec 19  2017 libicuuc.so.3

-rwxr-xr-x   1 root     root      475312 Dec 19  2017 libj2eeplugin.so

-rwxr-xr-x   1 root     root      470496 Dec 19  2017 libldap60.so

-rwxr-xr-x   1 root     root      103976 Dec 19  2017 liblibdbm.so

-rwxr-xr-x   1 root     root      338096 Dec 19  2017 liblibsi18n.so

-rwxr-xr-x   1 root     root     7046800 Dec 19  2017 libns-httpd40.so

-rwxr-xr-x   1 root     root       75744 Dec 19  2017 libnsfc.so

-rwxr-xr-x   1 root     root      400672 Dec 19  2017 libnspr4.so

-rwxr-xr-x   1 root     root      118576 Dec 19  2017 libnsprwrap.so

-rwxr-xr-x   1 root     root     1626736 Dec 19  2017 libnss3.so

-rwxr-xr-x   1 root     root      804960 Dec 19  2017 libnssckbi.so

-rw-r--r--   1 root     root         899 Dec 19  2017 libnssdbm3.chk

-rwxr-xr-x   1 root     root      239288 Dec 19  2017 libnssdbm3.so

-rwxr-xr-x   1 root     root      262256 Dec 19  2017 libnssutil3.so

-rwxr-xr-x   1 root     root       38600 Dec 19  2017 libnstime.so

-rwxr-xr-x   1 root     root       20784 Dec 19  2017 libnstp.so

-rwxr-xr-x   1 root     root       74856 Dec 19  2017 libpcre.so.0

-rwxr-xr-x   1 root     root       41056 Dec 19  2017 libplc4.so

-rwxr-xr-x   1 root     root       19848 Dec 19  2017 libplds4.so

-rwxr-xr-x   1 root     root       40808 Dec 19  2017 libprldap60.so

-rwxr-xr-x   1 root     root      166392 Dec 19  2017 libsasl.so

-rwxr-xr-x   1 root     root       11160 Dec 19  2017 libShtml.so

-rwxr-xr-x   1 root     root      269840 Dec 19  2017 libsmime3.so

-rw-r--r--   1 root     root         899 Dec 19  2017 libsoftokn3.chk

-rwxr-xr-x   1 root     root      366936 Dec 19  2017 libsoftokn3.so

-rwxr-xr-x   1 root     root      837064 Dec 19  2017 libsqlite3.so

-rwxr-xr-x   1 root     root      522928 Dec 19  2017 libssl3.so

-rwxr-xr-x   1 root     root       64704 Dec 19  2017 libssldap60.so

-rwxr-xr-x   1 root     root      149416 Dec 19  2017 libsupport.so

-rwxr-xr-x   1 root     root      230064 Dec 19  2017 libwebdav.so

-rwxr-xr-x   1 root     root     7157048 Dec 19  2017 libxalan-c.so.19

-rwxr-xr-x   1 root     root       43776 Dec 19  2017 libxalanMsg.so.19

-rwxr-xr-x   1 root     root     7709632 Dec 19  2017 libxerces-c.so.26

-rwxr-xr-x   1 root     root       31312 Dec 19  2017 libxp.so

-rwxr-xr-x   1 root     root       76480 Dec 19  2017 libxsd2cpp.so

-rwxr-xr-x   1 root     root      138768 Dec 19  2017 libz.so

-rwxr-xr-x   1 root     root      297416 Dec 19  2017 modutil

-rwxr-xr-x   1 root     root      132344 Dec 19  2017 pk12util

-rwxr-xr-x   1 root     root       10472 Dec 19  2017 true

-rwxr-xr-x   1 root     root       14360 Dec 19  2017 webservd

-rwxr-xr-x   1 root     root      119368 Dec 19  2017 webservd-wdog

bash-3.2$ ls -al /usr/lib/mps/sparcv9/

total 14502

drwxr-xr-x   2 root     bin         1024 Jun 19  2018 .

drwxr-xr-x   5 root     bin         1024 Jun 19  2018 ..

-rwxr-xr-x   1 root     bin          899 Aug  8  2017 libfreebl_64fpu_3.chk

-rwxr-xr-x   1 root     bin       696168 Aug  8  2017 libfreebl_64fpu_3.so

-rwxr-xr-x   1 root     bin          899 Aug  8  2017 libfreebl_64int_3.chk

-rwxr-xr-x   1 root     bin       579760 Aug  8  2017 libfreebl_64int_3.so

-rwxr-xr-x   1 root     bin       306968 Nov  5  2004 libjss3.so

-rwxr-xr-x   1 root     bin       287400 Aug  8  2017 libjss4.so

-rwxr-xr-x   1 root     bin       400672 Aug  8  2017 libnspr4.so

-rwxr-xr-x   1 root     bin      1626736 Aug  8  2017 libnss3.so

-rwxr-xr-x   1 root     bin       804960 Aug  8  2017 libnssckbi.so

-rw-r--r--   1 root     bin          899 Aug  8  2017 libnssdbm3.chk

-rwxr-xr-x   1 root     bin       239288 Aug  8  2017 libnssdbm3.so

-rwxr-xr-x   1 root     bin       262256 Aug  8  2017 libnssutil3.so

-rwxr-xr-x   1 root     bin        41056 Aug  8  2017 libplc4.so

-rwxr-xr-x   1 root     bin        19848 Aug  8  2017 libplds4.so

-rwxr-xr-x   1 root     bin       269840 Aug  8  2017 libsmime3.so

-rwxr-xr-x   1 root     bin          899 Aug  8  2017 libsoftokn3.chk

-rwxr-xr-x   1 root     bin       366936 Aug  8  2017 libsoftokn3.so

-rwxr-xr-x   1 root     bin       837064 Aug  8  2017 libsqlite3.so

-rwxr-xr-x   1 root     bin       522928 Aug  8  2017 libssl3.so

--------------------------------------------