6 Replies Latest reply: Jul 12, 2012 7:45 PM by greybird RSS

    com.sleepycat.je.SecondaryIntegrityException when deleting

    943066
      I seem to be cursed with hitting obscure issues (or am just getting worse at debugging).

      Here goes -

      As a follow-on from my adventures with partial comparators (see: LOG_FILE_NOT_FOUND bug possible in current BDB JE? I've moved to working with a secondary database to maintain a secondary ordering (ie: uniqueness being handled with one key, ordering being handled by a secondary).

      Everything seems to work perfectly on puts(), however, as soon as I attempt a delete(), I get:

      ----
      Exception in thread "main" com.sleepycat.je.SecondaryIntegrityException: (JE 5.0.55) Secondary is corrupt: the primary record contains a key that is not present in the secondary
           at com.sleepycat.je.SecondaryDatabase.deleteKey(SecondaryDatabase.java:938)
           at com.sleepycat.je.SecondaryDatabase.updateSecondary(SecondaryDatabase.java:853)
           at com.sleepycat.je.SecondaryTrigger.databaseUpdated(SecondaryTrigger.java:41)
           at com.sleepycat.je.Database.notifyTriggers(Database.java:2144)
           at com.sleepycat.je.Database.deleteInternal(Database.java:930)
           at com.sleepycat.je.Database.delete(Database.java:828)
           at com.xxx.yyy.db.BDBCalendarStorageBackend.indexCalendar(BDBCalendarStorageBackend.java:113)
           at com.xxx.yyy.indexer.TicketIndexer.indexDeltaLogs(TicketIndexer.java:239)
           at com.xxx.yyy.cli.DeltaLogTool.run(DeltaLogTool.java:63)
           at com.xxx.yyy.cli.AbstractBaseCommand.execute(AbstractBaseCommand.java:114)
           at com.xxx.yyy.cli.DeltaLogTool.main(DeltaLogTool.java:85)

      ----

      I've done a fair amount of reading, and believe I'm doing things correctly (though obviously, not quite :) - Specifically:

      - Both DBs are transactional
      - The secondary key creator has no external state/is totally deterministic
      - I've started from a clean DB (so creator/comparators aren't deserialized)
      - I'm only writing to the primary DB (not touching secondary directly at all)

      Basic DB opening code is this:

      ----
           // Setup primary DB config
           DatabaseConfig dbConfig = new DatabaseConfig();
           dbConfig.setAllowCreate(true);
           dbConfig.setSortedDuplicates(false);
           dbConfig.setTransactional(true);
           dbConfig.setBtreeComparator(new IndexKeyBinding.PrimaryKeyComparator());
                     
           // Setup secondary DB config
           SecondaryConfig db2Config = new SecondaryConfig();
           db2Config.setAllowCreate(true);
           db2Config.setTransactional(true);
           db2Config.setSortedDuplicates(true); // Required for secondary (<route><cost> is not unique)
           db2Config.setBtreeComparator(new IndexKeyBinding.SecondaryKeyComparator());
           db2Config.setKeyCreator(new IndexKeyBinding.RouteCostKeyCreator());
      ----

      Key creator code is viewable here: https://gist.github.com/8b49086b57bc80ab4a15

      I've fiddled my key format quite a bit, because I was having some interesting issues with the primaryKeyEntry returning more bytes than are actually in the key. Despite my efforts, nothing seems to have made any difference.

      My secondary key is entirely created as a composite of my primaryKey and primaryValue (data) and I've put in a bunch of debugging statements in the key creator to ensure that the data being encoded is as I expect and it's all looking OK, so I'm a bit stuck as to what to look at next.

      Update: I've tried removing the custom secondary btree comparator (as I'm not practically using it) and it sadly makes no difference.

      Edited by: fb on Jul 11, 2012 1:58 AM

      Edited by: fb on Jul 16, 2012 7:32 PM
        • 1. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
          greybird
          Did you start the test with a fresh/empty environment directory, when you added the secondary db? If there is existing data (prior to adding the secondary db), you have to populate the new index. See SecondaryConfig.setAllowPopulate.

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
          • 2. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
            943066
            Yup, definitely from a totally clean DB.

            I can replicate it with a single put/delete too (ie: it's not something that happens after thousands of deletes or anything).
            • 3. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
              greybird
              My guess is that the problem is related to using a partial key comparator for the primary key. You're storing C1:C2:C3 in the primary key, but C2 can be considered a data field (since it's ignored by the comparator). If you do an update and change C2 (but not C1 or C3), I suspect the internal trigger it not firing (or not working properly in this case) and the secondary index is not being updated. Primary key partial comparators are very unusual and only supported fairly recently, and I've never seen them used in this way.

              I suggest that you don't use a partial primary key comparator, because it is adding complexity and (I believe) no benefit. Instead the primary key can contain C1:C3 only, and you may not even need a custom comparator, and C2 can be stored in the data. The secondary key creator still has access to all fields (it is passed the primary key and data), so it can create the C1:C2 index key.

              I'm wondering if you could give this a try, and hopefully it will get you over this hump.

              --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
              • 4. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
                greybird
                I've confirmed that if you're using a partial comparator and you change the primary key, and you change a primary key field that is used in a secondary key, JE won't properly update the secondary index. I consider this a bug, but it probably won't be fixed immediately.

                Please let me know if moving the C2 field into the record data worked out for you.

                --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                • 5. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
                  943066
                  Aaah, great insight, thanks heaps, Mark.

                  I did actually find a related oddity when I had a slightly different implementation, where if you based the secondary key entirely of partial portions of the primary key (ie: C1:C3) the value you received through in your KeyCreator was the key specified in the get(), not the the key returned by the get().

                  I realised this was probably expected behaviour anyway, but the cause was quite tricky to track down (ie: inserts always worked, gets sometimes worked, but often failed).

                  You're right that in my current implementation I'm not using the partial comparator anyway, so it should be relatively easy to change my implementation appropriately.

                  Thanks again for your help,

                  fb.
                  • 6. Re: com.sleepycat.je.SecondaryIntegrityException when deleting
                    greybird
                    I did actually find a related oddity when I had a slightly different implementation, where if you based the secondary key entirely of partial portions of the primary key (ie: C1:C3) the value you received through in your KeyCreator was the key specified in the get(), not the the key returned by the get().

                    I realised this was probably expected behaviour anyway, but the cause was quite tricky to track down (ie: inserts always worked, gets sometimes worked, but often failed).
                    This is incorrect behavior, but again it all has to do with the partial comparator. Without a partial comparator the key passed to get() is necessarily identical to the get returned by get().

                    FTR, I've filed internal ticket [#21592] for this, and I'll add this latest issue to the ticket.

                    --mark