This discussion is archived
5 Replies Latest reply: Feb 29, 2012 6:54 PM by greybird RSS

SecondaryIntegrityException using collections

914457 Newbie
Currently Being Moderated
Hello,

I am using BDB JE 4.1.17. I am running into "com.sleepycat.je.SecondaryIntegrityException: (JE 4.1.17) Secondary refers to a missing key in the primary database". Please help me understand what I might be doing wrong.
I have read the special considerations for secondary databases but it didnt seem to help me.I am using the collections api and my primary and secondary database configurations are set to use transactions. I also have a custom SecondaryKeyCreator and know that the secondary keys are being generated properly.

The sequence of operations is:
1)write entry into primary stored map
2)remove the same entry from primary stored map. Immediately after the remove call I try to iterate over the secondary map to check if the secondary key is removed. I then see the above exception.

I was under the impression that the secondary index would be automatically maintained but this doesnt seem to be the case.

Stacktrace:* ( The exception happens on the itr.hasNext()) in the Removal code below. )

2012 Feb 29 15:13:09:510 GMT -8 rt Error [Thread-12] - [root] com.sleepycat.je.SecondaryIntegrityException: (JE 4.1.17) Secondary refers to a missing key in the primary database
2012 Feb 29 15:13:09:510 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.je.SecondaryDatabase.secondaryRefersToMissingPrimaryKey(SecondaryDatabase.java:1124)
2012 Feb 29 15:13:09:511 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.je.SecondaryCursor.readPrimaryAfterGet(SecondaryCursor.java:1507)
2012 Feb 29 15:13:09:511 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.je.SecondaryCursor.position(SecondaryCursor.java:1392)
2012 Feb 29 15:13:09:512 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.je.SecondaryCursor.getFirst(SecondaryCursor.java:394)
2012 Feb 29 15:13:09:512 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.util.keyrange.RangeCursor.doGetFirst(RangeCursor.java:877)
2012 Feb 29 15:13:09:513 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.util.keyrange.RangeCursor.getFirst(RangeCursor.java:266)
2012 Feb 29 15:13:09:513 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.collections.DataCursor.getFirst(DataCursor.java:423)
2012 Feb 29 15:13:09:514 GMT -8 rt Error [Thread-12] - [root]      at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:299)

My code for the creation of databases and the write/removal are below.

Database/Storedmap creation:*

public SecondaryDatabase createSecondaryDatabase(String databaseName, Database primaryDb,
Class keyClass,Class valueClass,Class indexKeyClass) throws InterruptedException {

SecondaryDatabase _secondaryDb = null;

if(logger.isEnabledFor(Level.TRACE) ) {
logger.log(Level.TRACE,"Created secondary database ["+databaseName+"]");
}
//Set the environment's secondary config
SecondaryConfig secConfig = new SecondaryConfig();
secConfig.setTransactional(true);
secConfig.setAllowCreate(true);
secConfig.setKeyCreator(new BDBSecondaryKeyCreator(databaseName,_javaCatalog,keyClass,valueClass,indexKeyClass));

// Now pass secConfig to Environment.openSecondaryDatabase
secondaryDb = env.openSecondaryDatabase(null,databaseName,primaryDb,secConfig);
return _secondaryDb;
}
     
     public StoredMap createStoredMap(Database db,DatabaseConfig cfg,Class keyClass,Class valueClass,boolean writeAllowed) throws InterruptedException {

//Create the key bindings
EntryBinding keyBinding = new SerialBinding(_javaCatalog, keyClass);
EntryBinding valueBinding = new SerialBinding(_javaCatalog, valueClass);
StoredMap map = null;

try {
map = new StoredMap(db,keyBinding,valueBinding,writeAllowed);
logger.log(Level.INFO,"Created Berkeley DB storedmap for: "+db.getDatabaseName());
} catch (Exception e) {
e.printStackTrace();
}
return map;
}
     
     
Addition:*
if( key!=null && value!=null ) {
try {
this._persistenceMap.put(key,value );
} catch ( Exception u) {
logger.log(Level.ERROR,"Put exception for database ["+u.getSecondaryDatabaseName()+"]");
throw new RuntimeException(u);
}
                              }

Removal:

try {
this._persistenceMap.remove(key);

if( _secondaryMap!=null ) {
Set entries = this._secondaryMap.entrySet();
Iterator itr = entries.iterator();
logger.log(Level.INFO,"---------------------");
while(itr.hasNext()) {
Map.Entry entry = (Map.Entry) itr.next();
logger.log(Level.INFO,"Secondary map has:"+(String)entry.getKey());
}
logger.log(Level.INFO,"---------------------");
}

} catch (SecondaryIntegrityException e) {
logger.log(Level.ERROR," Exception at Primary key="+e.getPrimaryKey ()+",Secondary key="+e.getPrimaryKey() );
e.printStackTrace();
}
  • 1. Re: SecondaryIntegrityException using collections
    greybird Expert
    Currently Being Moderated
    I was under the impression that the secondary index would be automatically maintained but this doesnt seem to be the case.
    It is maintained automatically. When using transactions, we don't know of any bugs where the secondary index becomes out of date. Of course, there is a first time for everything, and you should save a copy of your environment directory in case we need to look at the log files.

    Your message indicates that you're doing everything correctly. However, the first step is to double-check. Could you please send the code for opening the primary database and your secondary key creator code?

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 2. Re: SecondaryIntegrityException using collections
    914457 Newbie
    Currently Being Moderated
    Here is the code for the creation of the primary database and stored map:*

    +public ActionResult onOpen(OpenAction openAction) {+*
    logger.log(Level.INFO, "Creating BDB persister for space "openAction.getSpaceName());+
    +try {+
    db = provider.createDatabase(openAction.getSpaceName());+*
    +//Create a secondary index only if the space is a data space+
    +if( _persisterType == ASPersistenceProvider.PersisterType.Data ) {+
    secondaryDb = provider.createSecondaryDatabase(openAction.getSpaceName()+BDBPersistenceConstants.SecondaryDbNameSuffix,_db,_keyClass,_valueClass,_secIndexClass);+*
    +//also create a secondary storedmap+
    secondaryMap = provider.createSecondaryStoredMap(_secondaryDb,_secIndexClass,_valueClass,true);+*
    BDBSecondaryKeyCreator keyCreator = (BDBSecondaryKeyCreator)_secondaryDb.getConfig().getKeyCreator();
    +if( keyCreator != null ) {+
    keyCreator.setSecondaryStoredMap(_secondaryMap);
    +}+
    +}+
    persistenceMap = provider.createStoredMap(_db, keyClass,valueClass,true);+*
    logger.log(Level.INFO, "Created BDB persister for space "openAction.getSpaceName());+
    +} catch (InterruptedException e) {+
    throw new RuntimeException(e.getMessage());
    +}+
    return ActionResult.create();
    +}+

    +public Database createDatabase(String databaseName) throws RuntimeException,InterruptedException  {+*
    return createDatabase(databaseName,_dbConfig);
    +}+

    +//Allow the user to specify his own config+
    +public Database createDatabase(String databaseName,DatabaseConfig cfg) throws RuntimeException,InterruptedException {+*
    Database db = null;

    +try {+
    +if( _env == null ) {+
    throw new RuntimeException("BDB environment not configured. Aborting database creation!");
    +}+
    +db = _env.openDatabase(null, databaseName, cfg);+
    logger.log(Level.INFO,"Created Berkeley DB store: "databaseName);+
    _dbNameToDatabase.put(databaseName,db);+

    +} catch (Exception e) {+
    throw new RuntimeException(e.getMessage());
    +}finally {+
    return db;
    +}+
    +}+

    +public StoredMap createStoredMap(Database db,Class keyClass,Class valueClass,boolean writeAllowed) throws InterruptedException {+*
    return createStoredMap(db,_dbConfig,keyClass,valueClass,writeAllowed);
    +}+
    +     +
    +public StoredMap createStoredMap(Database db,DatabaseConfig cfg,Class keyClass,Class valueClass,boolean writeAllowed) throws InterruptedException* {+

    +//waitForInitialized();+
    +//Create the key bindings+
    EntryBinding keyBinding = new SerialBinding(_javaCatalog, keyClass);
    EntryBinding valueBinding = new SerialBinding(_javaCatalog, valueClass);
    StoredMap map = null;

    +try {+
    map = new StoredMap(db,keyBinding,valueBinding,writeAllowed);
    logger.log(Level.INFO,"Created Berkeley DB storedmap for: "db.getDatabaseName());+
    +} catch (Exception e) {+
    e.printStackTrace();
    +}+
    _dbNameToStoredMap.put(db.getDatabaseName(), map);+
    return map;
    +}     +

    Here is the code for the SecondarykeyCreator implementation:*

    +public class BDBSecondaryKeyCreator extends SerialSerialKeyCreator {+*

    +private Logger _logger = LogManagerFactory.getLogManager().getLogger(BDBSecondaryKeyCreator.class);+
    +private String _databaseName;+
    +private ObjectCodecHook _objectcodecHook = null;+
    +private Cluster _cluster = null;+
    +private SecondaryDatabase _secondaryDb = null;+
    +private StoredMap _secondaryStoredMap = null;+

    public BDBSecondaryKeyCreator(String databaseName,ClassCatalog catalog,
    Class primaryKeyClass,
    Class valueClass,
    +Class indexKeyClass) {+

    super(catalog, primaryKeyClass, valueClass, indexKeyClass);
    _databaseName = databaseName;+
    _cluster = RuleServiceProviderManager.getInstance().getDefaultProvider().getCluster();+
    ASDaoProvider provider = (ASDaoProvider)_cluster.getDaoProvider();
    DefaultTupleCodec defaultTupleCodec = (DefaultTupleCodec)provider.getTupleCodec();

    +if( defaultTupleCodec != null ) {+
    _objectcodecHook = defaultTupleCodec.getObjectCodec();+
    +}+
    +}+

    +public Object createSecondaryKey(Object key, Object value) {+*
    +//+
    +// Extract the secondary key from the primary key and+
    +// data, and set the secondary key into the result parameter.+
    +//+

    Entity deserEntity= null;
    +if( _objectcodecHook!= null ) {+

    +try {+
    deserEntity= (Entity)_objectcodecHook.decode((byte[])value,null,"v");
    +} catch (Exception e) {+
    logger.log(Level.ERROR,"Unable to decode object for secondary index insert for ["+databaseName+"]");+
    throw new RuntimeException(e.getMessage());
    +}+

    +if( deserEntity!=null ) {+

    String extId = deserEntity.getExtId();
    +if (  extId == null ) {+
    return null;
    +}+
    +//Make sure the key doesnt already exist+
    +//If it does simply return null+
    +if( _secondaryStoredMap != null ) {+
    +if( _secondaryStoredMap.containsKey(extId) ) {+
    +if ( _logger.isEnabledFor(Level.TRACE)) {+
    logger.log(Level.TRACE,deserEntity.getExtId()+" already exists in secondary database ["+databaseName+"]");+
    +}+
    return null;
    +}+
    +}+
    +if ( _logger.isEnabledFor(Level.TRACE)) {+
    logger.log(Level.TRACE,"Creating secondary index "+deserEntity.getExtId()+" for database ["+databaseName+"]");+
    +}+
    return extId;
    +}+

    +}+
    return null;
    +}                              +
    +     +
    +}+


    -Prakash
  • 3. Re: SecondaryIntegrityException using collections
    greybird Expert
    Currently Being Moderated
    Thanks for sending this. Unfortunately, I can't tell whether it is correct or not, because it uses several external classes and is abstract.

    At this point I think the best way for me to help you is for you to first send a reproducible test case that shows the problem you're having, in a single self-contained source file. Please post the file here, or email the file to mark.hayes at o.com (o == oracle).

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 4. Re: SecondaryIntegrityException using collections
    914457 Newbie
    Currently Being Moderated
    It was a bug in my implementation. Thank you for your help Mark.

    -Prakash
  • 5. Re: SecondaryIntegrityException using collections
    greybird Expert
    Currently Being Moderated
    You're welcome. I'm glad to hear it's resolved.
    --mark                                                                                                                                                                                                       

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points