Looks like the stack got corrupted for some reason. Could be buffer overflow or something like that. It doesn't seem that dbx's guess about overflow is right because it only showed 4 frames that look normal.
It's impossible to say why it worked on Solaris 9 and is crashing on 11.3; the reason may not be connected with the OS version at all.
I'd start with recompiling for debugging (preferrably without optimization) and running the program under dbx to catch it misbehaving earlier than it crashed.
The culprit may be a new compiler, rather than new Operating System. It poorly treats C++ of 2014. - Many bugs.
Try to change number of bits (32, 64), optimisation, level of C++ (..., 2011, 2014).
Check which libraries it links to. A new library may be not compatible with the old. For a new library you may need to use a new header, and sometimes change the code substantially.