I am wondering whether there is a way to prevent the JVM from dying when the JNI native code hits an assertion.
There is no way to prevent the VM from exiting if native code does anything that causes an exit.
An assertion is only one way that can happen.
So to prevent the VM from exiting - don't run native code. The safe way to execute OS native library is to do the following.
1. Wrap the library in an executable.
2. Create a communications API for the executable.
3. Manage the executable via Java Runtime.exec/ProcessBuilder.
4. Talk to the executable using the communications API from 2 in the java code.
The above is safe because if the library exits it exits the executable, not the VM.