This discussion is archived
2 Replies Latest reply: Feb 25, 2009 7:06 AM by 679075 RSS

BUG + PATCH: SecondaryKeyCreator is passed partial data

679075 Newbie
Currently Being Moderated
If you put a DatabaseEntry containing only partial data into a primary database, and that primary database has an associated secondary database, then when the secondary key is being recomputed the SecondaryKeyCreator is given the very same partial DatabaseEntry object instead of a new "merged" DatabaseEntry containing the entire new data value being put.

This is obviously unacceptable, because the secondary key can, in general, be based on any or all of the primary data, and the SecondaryKeyCreator therefore needs to be able to see all of it.

The patch below fixes this (including some refactoring to eliminate duplicate code):
diff -ur bdb/com/sleepycat/je/DatabaseEntry.java src/java/com/sleepycat/je/DatabaseEntry.java
--- bdb/com/sleepycat/je/DatabaseEntry.java     2008-06-10 10:52:08.000000000 -0500
+++ src/java/com/sleepycat/je/DatabaseEntry.java        2009-02-24 14:25:42.073159061 -0600
@@ -470,4 +471,48 @@
         }
         return hash;
     }
+
+    /**
+     * Merge this DatabaseEntry's partial data into the given existing data,
+     * truncating or extending it as required.
+     *
+     * @param origdata buffer containing the original data; will not be modified by this method
+     * @param origoffset offset of the original data in the origdata buffer
+     * @param origlen length of the original data in the origdata buffer
+     * @return the merge of the given data and this entry in a newly allocated buffer
+     * @throws NullPointerException if origdata is null
+     * @throws IllegalArgumentException if this entry is not partial
+     */
+    public byte[] mergeInto(byte[] origdata, int origoffset, int origlen) {
+
+        /* Check we're partial */
+        if (!partial)
+            throw new IllegalArgumentException("data is not partial");
+
+        /* Compute old and new lengths and allocate new buffer */
+        int oldlen = (doff + dlen > origlen) ? doff + dlen : origlen;
+        int len = oldlen - dlen + size;
+        byte[] newData = new byte[len];
+
+        /* Keep 0..doff of the old data (truncating if doff > length) */
+        int pos = 0;
+        int slicelen = (doff < origlen) ? doff : origlen;
+        if (slicelen > 0)
+            System.arraycopy(origdata, origoffset, newData, pos, slicelen);
+        pos += doff;
+
+        /* Copy in the new data */
+        slicelen = size;
+        System.arraycopy(data, offset, newData, pos, slicelen);
+        pos += slicelen;
+
+        /* Append the rest of the old data (if any). */
+        slicelen = origlen - (doff + dlen);
+        if (slicelen > 0)
+            System.arraycopy(origdata, origoffset + doff + dlen, newData, pos, slicelen);
+
+        /* Done */
+        return newData;
+    }
 }
+
diff -ur bdb/com/sleepycat/je/SecondaryDatabase.java src/java/com/sleepycat/je/SecondaryDatabase.java
--- bdb/com/sleepycat/je/SecondaryDatabase.java 2008-06-10 10:52:08.000000000 -0500
+++ src/java/com/sleepycat/je/SecondaryDatabase.java    2009-02-24 14:25:09.712446250 -0600
@@ -713,6 +713,24 @@
             return;
         }

+        /*
+         * Make sure we don't give partial data to the key creator.
+         * Assume old data is not partial, but new data might be.
+         */
+        assert oldData == null || !oldData.getPartial();
+        if (newData != null && newData.getPartial()) {
+            if (oldData == null) {
+                newData = new DatabaseEntry(newData.getData(),
+                                            newData.getOffset(),
+                                            newData.getSize());
+            } else {
+                byte[] buf = newData.mergeInto(oldData.getData(),
+                                               oldData.getOffset(),
+                                               oldData.getSize());
+                newData = new DatabaseEntry(buf);
+            }
+        }
+
         SecondaryKeyCreator keyCreator = secondaryConfig.getKeyCreator();
         if (keyCreator != null) {
             /* Each primary record may have a single secondary key. */
diff -ur bdb/com/sleepycat/je/dbi/CursorImpl.java src/java/com/sleepycat/je/dbi/CursorImpl.java
--- bdb/com/sleepycat/je/dbi/CursorImpl.java    2008-05-20 11:27:34.000000000 -0500
+++ src/java/com/sleepycat/je/dbi/CursorImpl.java       2009-02-24 14:38:38.524268376 -0600
@@ -1275,50 +1275,17 @@
             byte[] newData;

             /* Resolve partial puts. */
-            if (data.getPartial()) {
-                int dlen = data.getPartialLength();
-                int doff = data.getPartialOffset();
-                int origlen = (foundDataBytes != null) ?
-                    foundDataBytes.length : 0;
-                int oldlen = (doff + dlen > origlen) ? doff + dlen : origlen;
-                int len = oldlen - dlen + data.getSize();
-
-                if (len == 0) {
-                    newData = LogUtils.ZERO_LENGTH_BYTE_ARRAY;
-                } else {
-                    newData = new byte[len];
-                }
-                int pos = 0;
-
-                /*
-                 * Keep 0..doff of the old data (truncating if doff > length).
-                 */
-                int slicelen = (doff < origlen) ? doff : origlen;
-                if (slicelen > 0)
-                    System.arraycopy(foundDataBytes, 0, newData,
-                                     pos, slicelen);
-                pos += doff;
-
-                /* Copy in the new data. */
-                slicelen = data.getSize();
-                System.arraycopy(data.getData(), data.getOffset(),
-                                 newData, pos, slicelen);
-                pos += slicelen;
-
-                /* Append the rest of the old data (if any). */
-                slicelen = origlen - (doff + dlen);
-                if (slicelen > 0)
-                    System.arraycopy(foundDataBytes, doff + dlen, newData, pos,
-                                     slicelen);
+            if (data.getPartial() && foundDataBytes != null) {
+                newData = data.mergeInto(foundDataBytes, 0, foundDataBytes.length);
             } else {
                 int len = data.getSize();
                 if (len == 0) {
                     newData = LogUtils.ZERO_LENGTH_BYTE_ARRAY;
                 } else {
                     newData = new byte[len];
+                    System.arraycopy(data.getData(), data.getOffset(),
+                                     newData, 0, len);
                 }
-                System.arraycopy(data.getData(), data.getOffset(),
-                                 newData, 0, len);
             }

             if (databaseImpl.getSortedDuplicates()) {
  • 1. Re: BUG + PATCH: SecondaryKeyCreator is passed partial data
    greybird Expert
    Currently Being Moderated
    You are correct that this is a bug. Thanks for pointing this out! We will fix it in a future release.

    In the mean time I suggest working around this by not using a partial DatabaseEntry for put() methods, when secondaries are associated with the primary database.

    Please note that using a partial DatabaseEntry for a put() method has very little performance advantage and is very rarely used. This feature is included primarily for compatibility with the original C edition of the BDB product.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 2. Re: BUG + PATCH: SecondaryKeyCreator is passed partial data
    679075 Newbie
    Currently Being Moderated
    Thanks for the response and clarification.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points