Apparently, somehow your secondary index DB has become out of sync with your primary. Normally this is caused by not using a transactional store (EntityStore.setTransactional). But whatever the cause, I'll describe how to correct the situation by rebuilding the index.
1) Take your application off-line so that no other operations are occurring.
2) Make a backup in case something goes wrong during this procedure.
3) Do not open the EntityStore yet.
4) Remove the index database that is out of sync (persist#gdlogs#test.TableWhProductStorageCard#ProductId) by calling Environment.removeDatabase with this name.
5) Rebuild the index database by simply opening the EntityStore. This will take longer than usual, since the index will be rebuilt before the EntityStore constructor returns.
6) Confirm that the index was rebuilt correctly.
7) Bring your application back on-line.
Thanks a lot!
One more question: Anyway to detect this? because when this happens, no way to detect that the secondary index are out of sync with primary unless the deleted key is out of the range of the secondary key. even insertion will not cause any exception or cause the index to auto sync with the primary.
We don't have any tools for scanning and matching primary records with to secondary indexes, although you could write one. When accessing records you may get the SecondaryIntegrityException (as you know), but this is probably not a desirable way to detect the problem.
Are you calling StoreConfig.setTransactional(true)?
No. I do not use transaction. Because the data has already been imported to the database and I am just wring some program to read the data, during this period, I have added some secondary indexes into some of the entities and then I found that some of the secondary indexes are not completed. So, for this kind of operation (adding indexes to entities by annotations), is it true that using transaction can 100% avoiding secondary index corruption?
Yes, you should use transactions. See:
All you have to do is call StoreConfig.setTransactional(true). You don't need to create an explicit transaction, since auto-commit will be used.