I am only trying to delete the entry using deleteEntry (in this case sub-ca-node-mgmt), then store an updated version of it (you could imagine this as replacing an expired certificate with the updated version) using setKeyEntry. If I add additional code to reload the keyStore after deleting the entry, this does not make any difference in the result. If I rerun the code over again, I can get as many separate copies of sub-ca-node-mgmt as you could imagine. If I comment out the call to setKeyEntry, the problem stops happening.
Thus I suspect that setKeyEntry is either not working as advertised, or I am misusing the API. Does anyone have a working example of how to delete and overwrite an existing entry in the PKCS11 KeyStore?
Found the root cause for this one. The code which generated the RSA key for the cert was not quite right, and mismatched the certs and keys. When this happened it broke the NSS DB, because the association between the keys and certs got damaged, and Java could not access the DB reliably anymore. However, Java is still part of the problem, because it fails to check if the CKA_ID (NSS / PKCS11 Key ID), which is calculated from the RSA modulus inside the public key / cert and the private key, are properly matching or not.