After upgrading to Tuxedo 12, we started getting core dumps in the new libcrypto.so lib (after applied workaround to ensure the correct lib is used). It was just a simple tpinit call that used to work when Tuxedo 11 was used. We've searched Tuxedo documentation and release notes, and didn't see any functionality change could cause this happening. The following is the stack trace:
#0 0x00007fc946503391 in SHA256_Final () from ..tuxedo/product/12.1/lib/libcrypto.so.1.0.0
#1 0x00007fc945713f99 in SHA256Final () from ..tuxedo/product/12.1/lib/libengine.so
#2 0x00007fc9457069df in AHChooseDigestFinal () ftom ..tuxedo/product/12.1/lib/libengine.so
#3 0x00007fc94576eecf in _e_cryp_AES256_genKey () from ..tuxedo/product/12.1/lib/libengine.so
#4 0x00007fc945bf6054 in _tm_e87 () from ..tuxedo/product/12.1/lib/libtux.so
#5 0x00007fc945bf61f7 in _tm_e103 () from ..tuxedo/product/12.1/lib/libtux.so
#6 0x00007fc945ba961e in atnlcl_gss_init_sec_context () from ..tuxedo/product/12.1/lib/libtux.so
#7 0x00007fc9457b5b83 in _e_sec_gss_init_sec_context_ex ()
#8 0x00007fc945adb316 in _tm_self_authenticate_ex () from ..tuxedo/product/12.1/lib/libtux.so
#9 0x00007fc945add4b7 in _tminit () from ..tuxedo/product/12.1/lib/libtux.so
#10 0x00007fc945ade105 in _tpinit () from ..tuxedo/product/12.1/lib/libtux.so
#11 0x00007fc945bbc894 in tpinit ()
Any ideas why it's happening?
One difference may be a change in the tpusr file. If you have not applied a patch then please apply the latest rolling patch level available for your platform,
recreate the tpusr file, and rerun your test(RP001 has a change for tpusr but there are some other security related fixes in later RPs which you may also want).
Otherwise, open a service request and provide an example/testcase/reproducer/steps to recreate.
There are many Tuxedo 12 rolling patches listed on the My Oracle Support website.
Following is an Oracle knowledge management (aka KM) document with instructions to locate one (i.e. choose the highest RP# available for your platform):
Tuxedo Patches Can Now Be Directly Downloaded Without Having To Open A Service Request NOTE 1075805.1
Let me know if you have any problems. (If so let me know the platform you are looking to find an RP for)