Should/Can we remove old keys from wallet after rotating to a new key

We have multiple databases set up using TDE and keys are stored in an Oracle Key Vault. We are required by our corporate policy to rotate the keys on a regular basis and we do this by running the following.

ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY **********;

Once we do this a new key is added the wallet. Is there are way to then remove the old key(s) from the wallet? It seems by still having the old keys that there it is a security concern. We also have other applications(QLIK), that read the data and need the key information added into them. As we get more and more keys, we will have a long list of keys added to that application.