This discussion is archived
12 Replies Latest reply: Mar 29, 2007 11:45 AM by 561346 RSS

thoughts on adding ConcurrentMap methods to StoredMap?

561346 Newbie
Currently Being Moderated
as far as i can tell, StoredMap has the potential to implement the extra methods in the jdk 1.5 ConcurrentMap interface. Database already has a putNoOverwrite method which would be the implementation of putIfAbsent. Personally, however, i am more interested in the other methods, all of which basically do their work only if the current value matches a given old value. this enables safe concurrent updates to a StoredMap even outside of transactions, basically using an optimistic scenario:

- read existing key,value (hold no locks)
- copy current value and modify the copy
- write modified value only if current value matches what is now in the database (this line within the context of a RMW lock)

any idea if this could be incorporated in the je codebase anytime soon? i may end up implementing it in our codebase, although the je api is not easy to extend as most classes/useful methods are package protected (arg!).
  • 1. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    as far as i can tell, StoredMap has the potential to
    implement the extra methods in the jdk 1.5
    ConcurrentMap interface.
    Yes, implementing ConcurrentMap would be really nice, and not very difficult. The main problem we have right now is that we do not depend on Java 1.5, so we could not actually implement the 1.5 ConcurrentMap interface.

    We could add the methods, however, without implementing the interface. Are you more interested in the methods (the functionality) or the implementation of the standard Java ConcurrentMap interface?

    If it is important to implement the standard Java ConcurrentMap interface, then perhaps we could implement it as separate Java 1.5-only classes -- StoredConcurrentMap and StoredConcurrentSortedMap. This would extend StoredMap and StoredSortedMap, and would only be usable from Java 1.5 and later.
    Database already has a
    putNoOverwrite method which would be the
    implementation of putIfAbsent.
    Yes, this would be easy.
    Personally, however,
    i am more interested in the other methods, all of
    which basically do their work only if the current
    value matches a given old value. this enables safe
    concurrent updates to a StoredMap even outside of
    transactions, basically using an optimistic
    scenario:

    - read existing key,value (hold no locks)
    - copy current value and modify the copy
    - write modified value only if current value matches
    what is now in the database (this line within the
    context of a RMW lock)
    Since we need to write-lock (RMW lock) the record before writing it, I don't see any advantage to reading the record without locking initially. We might as well just:

    - read with RMW
    - if the current value equals the given value, then write

    This implementation requires the use of a cursor or a transaction.
    any idea if this could be incorporated in the je
    codebase anytime soon?
    We don't have any plans to do this, but I'm happy to help you if I can understand better exactly what your needs are.
    i may end up implementing it
    in our codebase, although the je api is not easy to
    extend as most classes/useful methods are package
    protected (arg!).
    Yes, those classes were not designed to be extended, although you can always do so by modifying the source code.

    Mark
  • 2. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    as far as i can tell, StoredMap has the potential
    to
    implement the extra methods in the jdk 1.5
    ConcurrentMap interface.
    Yes, implementing ConcurrentMap would be really nice,
    and not very difficult. The main problem we have
    right now is that we do not depend on Java 1.5, so we
    could not actually implement the 1.5 ConcurrentMap
    interface.

    We could add the methods, however, without
    implementing the interface. Are you more interested
    in the methods (the functionality) or the
    implementation of the standard Java ConcurrentMap
    interface?
    We're only interested in the functionality, not the interface itself.
    Personally, however,
    i am more interested in the other methods, all of
    which basically do their work only if the current
    value matches a given old value. this enables
    safe
    concurrent updates to a StoredMap even outside of
    transactions, basically using an optimistic
    scenario:

    - read existing key,value (hold no locks)
    - copy current value and modify the copy
    - write modified value only if current value
    matches
    what is now in the database (this line within the
    context of a RMW lock)
    Since we need to write-lock (RMW lock) the record
    before writing it, I don't see any advantage to
    reading the record without locking initially. We
    might as well just:

    - read with RMW
    - if the current value equals the given value, then
    write

    This implementation requires the use of a cursor or a
    transaction.
    any idea if this could be incorporated in the je
    codebase anytime soon?
    We don't have any plans to do this, but I'm happy to
    help you if I can understand better exactly what your
    needs are.
    well, our basic need is to support concurrent modification of a database, where there is a very small chance of concurrent modification of the same record. (and, before i continue, i'll admit i don't completely understand the relationship between transactions and locking). i can certainly implement this in a transactional environment, but i was hoping to avoid using transactions since the chance of collision is so small. i'm assuming that locking is still used in the non-transactional environment (or else you'd get seriously horked data?), so i was hoping that using concurrentmap-like methods would enable me to concurrently modify a database without transactions, but while still handling the occasional collision. that is why my description of the operations involve the RMW lock only on the write call, so there is no need for cursors (at least, outside of the write method) or transactions. however, i could be misunderstanding the functionality available so please tell me if i'm way off base with what i'm looking for.
  • 3. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    Hi,
    We're only interested in the functionality, not the
    interface itself.
    OK. Then I'll just explain how to do this using the existing APIs.
    well, our basic need is to support concurrent
    modification of a database, where there is a very
    small chance of concurrent modification of the same
    record. (and, before i continue, i'll admit i don't
    completely understand the relationship between
    transactions and locking). i can certainly implement
    this in a transactional environment, but i was hoping
    to avoid using transactions since the chance of
    collision is so small.
    Why not use transactions anyway? Their overhead is very small. If you don't need durability on each operation, call EnvironmentConfig.setTxnNoSync, and you'll see roughly the same performance with transactions as without transactions.
    i'm assuming that locking is
    still used in the non-transactional environment (or
    else you'd get seriously horked data?), so i was
    Yes, locking is always used unless you specify READ_UNCOMMITTED (dirty read). Using a cursor (without transactions), the lock is held until the cursor moves or is closed. When using transactions, the lock is held until the transaction ends (abort or commit).
    hoping that using concurrentmap-like methods would
    enable me to concurrently modify a database without
    transactions, but while still handling the occasional
    collision. that is why my description of the
    operations involve the RMW lock only on the write
    call, so there is no need for cursors (at least,
    outside of the write method) or transactions.
    however, i could be misunderstanding the
    functionality available so please tell me if i'm way
    off base with what i'm looking for.
    If you don't use transactions, then you should use a cursor whenever you need to do an update where you process the existing data in some way -- a read-modify-write cycle. The cursor is needed to hold the lock on the record you're modifying, so that another thread cannot modify it. If you don't use a cursor or transactions, then JE cannot help you to prevent conflicts between multiple threads trying to read-modify-write the same record -- you would have to do that completely on your own.

    When reading via a cursor, you have the option of using RMW when you read the record. If you use RMW, this reduces the chance of deadlocks caused when another thread is trying to read-modify-write the same record. However, after you read the record if you decide not to write it, then the RMW lock is wasted and concurrency is reduced unnecessarily.

    Therefore, if there is a possibility that multiple threads will read-modify-write the same record, and you always know you will write the record after reading it, use RMW when reading. If you will write it only sometimes, or if there is only one writer thread, then do not use RMW.

    If there are multiple writer threads and you do not use RMW, be prepared to handle a deadlock. When a DeadlockException is thrown, you must close all cursors and retry the operation from the beginning. If the operation is very simple -- a single read-modify-write cycle -- this is easy. If there are multiple write operations involved, "undoing" the operation will be difficult and you really should be using transactions. Be aware that if you write to a primary database that has secondary databases configured, multiple write operations are being performed and you cannot easily undo that operation when a deadlock occurs. When using secondaries, the use of transactions is strongly recommended.

    All of this is much simpler when using transactions. Please first decide whether you will use transactions or a cursor to perform the read-modify-write cycle. Then if you have questions about how to do this with the collections API, let me know and I can help with that.

    Mark
  • 4. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    If you don't use transactions, then you should use a
    cursor whenever you need to do an update where you
    process the existing data in some way -- a
    read-modify-write cycle. The cursor is needed to
    hold the lock on the record you're modifying, so that
    another thread cannot modify it. If you don't use a
    cursor or transactions, then JE cannot help you to
    prevent conflicts between multiple threads trying to
    read-modify-write the same record -- you would have
    to do that completely on your own.
    I may not be explaining myself well here. Like i mentioned before, I'm working from a scenario where write conflicts should be very infrequent. thus, i'm hoping to use an optimistic strategy, similar to what i would use with a concurrentmap. the way i usually update a concurrent map (using an optimistic strategy):

    - read the existing value (map.get()). after this call, you hold no locks (so someone could modify the value before you write, but i'm assuming infrequent conflicts)
    - create new value (maintaining handle to existing value)
    - attempt to write new value only if existing value is unchanged (map.replace(key, old, new)) (if this call fails, you loop back to the first step and try again)

    this is obviously sub-optimal if you expect frequent, colliding updates. however, we are not. and, the nice part about this api is that the caller does not need to do any locking or transaction management.
    When reading via a cursor, you have the option of
    using RMW when you read the record. If you use RMW,
    this reduces the chance of deadlocks caused when
    another thread is trying to read-modify-write the
    same record. However, after you read the record if
    you decide not to write it, then the RMW lock is
    wasted and concurrency is reduced unnecessarily.

    Therefore, if there is a possibility that multiple
    threads will read-modify-write the same record, and
    you always know you will write the record after
    reading it, use RMW when reading. If you will write
    it only sometimes, or if there is only one writer
    thread, then do not use RMW.
    we will always be writing the record after we read it. your reasoning definitely makes sense. however, i'm working with the collections api, and i do not see any obvious way to do this. am i missing something?
    If there are multiple writer threads and you do not
    use RMW, be prepared to handle a deadlock. When a
    DeadlockException is thrown, you must close all
    cursors and retry the operation from the beginning.
    If the operation is very simple -- a single
    read-modify-write cycle -- this is easy. If there
    are multiple write operations involved, "undoing"
    the operation will be difficult and you really
    should be using transactions. Be aware that if you
    write to a primary database that has secondary
    databases configured, multiple write operations are
    being performed and you cannot easily undo that
    operation when a deadlock occurs. When using
    secondaries, the use of transactions is strongly
    recommended.

    All of this is much simpler when using transactions.
    Please first decide whether you will use
    transactions or a cursor to perform the
    read-modify-write cycle. Then if you have questions
    about how to do this with the collections API, let
    me know and I can help with that.
    the reason that the concurrentmap interface is attractive for someone using the collections api (which we are), is that all that complexity is hidden behind the api. and, by adding these methods, you allow users of the collections api to make safe updates without having to delve into transactions. we don't need to group a bunch of updates together nor update secondary databases or i would definitely use transactions. we just want to make safe, independent updates to single records in a primary database. again, i could certainly use transactions for this, but i'm using the collections api, and was hoping to get away with just that.
  • 5. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    - read the existing value (map.get()). after this
    call, you hold no locks (so someone could modify
    the value before you write, but i'm assuming
    infrequent conflicts)
    - create new value (maintaining handle to existing
    value)
    - attempt to write new value only if existing value
    is unchanged (map.replace(key, old, new)) (if this
    call fails, you loop back to the first step and try
    again)
    This has almost no performance advantage over using a transaction or a cursor. No matter what you do, a write lock must be taken when you update the record. So you might as well take the lock when you read the record. Delaying the lock for a very small amount of time will have almost no advantage.

    If you do performance measurements and you find that holding an RMW (write) lock for this extra interval is a performance problem, the solution is to not use RMW. If you do not use RMW, then a read lock (not a write lock) will be taken when reading the record, which will avoid reducing concurrency. When you write, you may get a DeadlockException but this will be extremely rare, since you say there is very little chance of conflicts among writers.

    Another way of saying this is: If we were to implement this API, we would have to use a cursor, and lock the record when we read it. Therefore, this API (while perhaps convenient) has no performance advantage.

    These are the factors from a performance perspective, not an API perspective. More on the API below.
    the reason that the concurrentmap interface is
    attractive for someone using the collections api
    (which we are), is that all that complexity is hidden
    behind the api. and, by adding these methods, you
    allow users of the collections api to make safe
    updates without having to delve into transactions.
    we don't need to group a bunch of updates together
    nor update secondary databases or i would definitely
    use transactions. we just want to make safe,
    independent updates to single records in a primary
    database. again, i could certainly use transactions
    for this, but i'm using the collections api, and was
    hoping to get away with just that.
    I don't understand why using transactions is difficult or undesirable, it's quite easy, especially for the operation you describe. Can you explain further?

    In any case, if you do not use transactions and you want to use the collections API, then you have two options:

    1) If you want the concurrent map interface or new methods that are equivalent, you'll have to extend the source code yourself to get this capability. Adding this is a good idea as a convenience method for non-transactional use. However, because this functionality is currently available and easy to use with transactions, it will probably not be a high priority for us to add this, although we'll definitely keep your request in mind during future planning and we'll watch for other users that request this. It may also be important to add support for the ConcurrentMap interface, for interface compatibility reasons, in the future; but so far, we have not had any requests for this. I described earlier how this extension to StoredMap could be implemented using cursors, if you decide to do it yourself.

    2) To use the current collections API to do a read-write-modify without transactions you can do the following. A StoredIterator is effectively a cursor. To use RMW, call StoredIterator.setReadModifyWrite(true).
    Object myKey;
    StoredSortedMap myMap;
    StoredSortedMap subMap = (StoredSortedMap) myMap.sublMap(myKey, true, myKey, true);
    StoredIterator i = subMap.storedIterator();
    try {
        if (i.hasNext()) {
            Object val = i.next();
            // do some processing on val
            i.set(val);
        }
    } finally {
        i.close();
    }
    Please let me know what you think.

    Mark
  • 6. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    This has almost no performance advantage over using a
    transaction or a cursor. No matter what you do, a
    write lock must be taken when you update the record.
    So you might as well take the lock when you read the
    record. Delaying the lock for a very small amount
    of time will have almost no advantage.
    i completely understand.
    I don't understand why using transactions is
    difficult or undesirable, it's quite easy, especially
    for the operation you describe. Can you explain
    further?
    to expand further on our usage, bdb code in our system is actually hidden behind our own "persistent collection" api. currently, this api exposes no transaction related stuff. in order to utilize transactions in our client code, we would have to create some sort of generic transaction api for our persistent collection api.

    as for hiding the transaction behind a method implementation, perhaps you could explain for me the implications of opening a database (and environment for that matter) as "transactional" vs "non-transactional". in order to utilize transactions within certain methods behind our persistent collection api, i would have to open every env/db as "transactional". however, i was assuming there was some sort of overall performance implication in this (otherwise, why have the option?). i mean, if the system will perform the same way if i just open every env/db as transactional, i could easily implement the map.replace() method using transactions instead of using the method you include below.
    In any case, if you do not use transactions and you
    want to use the collections API, then you have two
    options:

    1) If you want the concurrent map interface or new
    methods that are equivalent, you'll have to extend
    the source code yourself to get this capability.
    Adding this is a good idea as a convenience method
    for non-transactional use. However, because this
    functionality is currently available and easy to use
    with transactions, it will probably not be a high
    priority for us to add this, although we'll
    definitely keep your request in mind during future
    planning and we'll watch for other users that
    request this. It may also be important to add
    support for the ConcurrentMap interface, for
    interface compatibility reasons, in the future; but
    so far, we have not had any requests for this. I
    described earlier how this extension to StoredMap
    could be implemented using cursors, if you decide to
    do it yourself.

    2) To use the current collections API to do a
    read-write-modify without transactions you can do the
    following. A StoredIterator is effectively a cursor.
    To use RMW, call
    StoredIterator.setReadModifyWrite(true).
    pre]
    Object myKey;
    StoredSortedMap myMap;
    StoredSortedMap subMap = (StoredSortedMap)
    myMap.sublMap(myKey, true, myKey, true);
    StoredIterator i = subMap.storedIterator();
    try {
    if (i.hasNext()) {
    Object val = i.next();
    // do some processing on val
    i.set(val);
    }
    } finally {
    i.close();
    /pre]
    ah, beautiful. i didn't realize you could enable RMW through the use of a storediterator. that should provide a very simple implementation of concurrentmap.replace() without the use of transactions. which brings me back to my question above. does enabling transactions by default affect overall performance?
  • 7. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    Hi,

    In BDB JE, enabling transactions adds no overhead when an environment is opened. The reason for the EnvironmentConfig.setTransactional API is to enable the transaction feature For our commercial customers (we have a dual license product) we sell the transactional version of the product separately from the non-transactional version -- they are two products. Whether transactions are configured or not is the distinguishing factor.

    When each transaction is committed, a very small commit record is written to the log -- I doubt that will have an impact on performance for you.

    The main overhead for transactions is to provide durability. Without transactions, durability is only guaranteed when you call Environment.sync -- at that time, we checkpoint, flush all buffers and fsync to disk.

    With transactions, we flush and fsync at every commit. This is very expensive and is required for strict durability.

    But if you are using transactions and you don't need that durability, you can simply call EnvironmentConfig.setTxnNoSync(true) and the durability (and overhead) with transactions will be the same as without transactions. In this case, you get the atomicity of transactions without the durability.

    Note that when you configure a database as transactional, we will use auto-commit if you don't explicitly begin a transaction. So you don't have to change your operation code except in cases where you want to use a transaction, such as for this read-modify-write cycle.

    Mark
  • 8. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    But if you are using transactions and you don't need
    that durability, you can simply call
    EnvironmentConfig.setTxnNoSync(true) and the
    durability (and overhead) with transactions will be
    the same as without transactions. In this case, you
    get the atomicity of transactions without the
    durability.
    okay, that makes a lot more sense now. i actually played a bit already with transactions and the various levels of durability, and that is pretty much what i saw in my limited tests. but, i couldn't figure out why transactions still had to be explicitly enabled. you might want to add that to the docs somewhere so others don't come to the same erroneous conclusions that i did.

    So anyway, working from your example, i've implemented 3 of the 4 concurrentmap methods. however, putIfAbsent() eludes me as the collections api does not seem to expose a way to call this on the underlying database? This is what i have so far, does it look reasonable?:
      private StoredIterator getIteratorForKey(Object key)
      {
        StoredSortedMap subMap = (StoredSortedMap)
          ((StoredSortedMap)_map).subMap(key, true, key, true);
        return ((StoredEntrySet)subMap.entrySet()).storedIterator();
      }
      
      public V putIfAbsent(K key, V value)
      {
        // FIXME
        return null;
      }
    
      public boolean remove(Object key, Object value)
      {
        StoredIterator iter = getIteratorForKey(key);
        try {
          iter.setReadModifyWrite(true);
          if(!iter.hasNext()) {
            return false;
          }
          Map.Entry e = (Map.Entry)iter.next();
          Object curVal = e.getValue();
          if(!ObjectUtils.equals(value, curVal)) {
            // was modified
            return false;
          }
          // good to go (we have the write lock already)
          iter.remove();
        } finally {
          iter.close();
        }
        return true;
      }
    
      @SuppressWarnings("unchecked")
      public boolean replace(K key, V oldValue, V newValue)
      {
        StoredIterator iter = getIteratorForKey(key);
        try {
          iter.setReadModifyWrite(true);
          if(!iter.hasNext()) {
            return false;
          }
          Map.Entry e = (Map.Entry)iter.next();
          Object curVal = e.getValue();
          if(!ObjectUtils.equals(oldValue, curVal)) {
            // was modified
            return false;
          }
          // set new value (we have the write lock already)
          e.setValue(newValue);
        } finally {
          iter.close();
        }
        return true;
      }
    
      @SuppressWarnings("unchecked")
      public V replace(K key, V value)
      {
        Object curValue = null;
        StoredIterator iter = getIteratorForKey(key);
        try {
          iter.setReadModifyWrite(true);
          if(iter.hasNext()) {
            Map.Entry e = (Map.Entry)iter.next();
            curValue = e.getValue();
            // set new value (we have the write lock already)
            e.setValue(value);
          }
        } finally {
          iter.close();
        }
        return (V)curValue;
      }
  • 9. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    okay, that makes a lot more sense now. i actually
    played a bit already with transactions and the
    various levels of durability, and that is pretty much
    what i saw in my limited tests. but, i couldn't
    figure out why transactions still had to be
    explicitly enabled. you might want to add that to
    the docs somewhere so others don't come to the same
    erroneous conclusions that i did.
    OK, will do.
    So anyway, working from your example, i've
    implemented 3 of the 4 concurrentmap methods.
    however, putIfAbsent() eludes me as the collections
    api does not seem to expose a way to call this on
    the underlying database? This is what i have so
    far, does it look reasonable?:
    These look good!

    You're correct that putNoOverwrite is not available in the Collections API.

    Since you cannot access package-private members, the only way to do this is to implement it at the base API level. You will have to save the Database and bindings that you used to create the StoredMap, and use them to call Database.putNoOverwrite.

    Another way is to configure the Serializable isolation level and simply put() if get() returns null, but this requires using transactions.

    Mark
  • 10. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    You're correct that putNoOverwrite is not available
    in the Collections API.

    Since you cannot access package-private members, the
    only way to do this is to implement it at the base
    API level. You will have to save the Database and
    bindings that you used to create the StoredMap, and
    use them to call Database.putNoOverwrite.
    First, thanks for all your help with this!

    So, in an effort to see if i could write these methods "correctly", i dove into the lower level api. I ended up reimplementing the previous methods and implementing the remaining method. (for one thing, the previous implementations did not work if transactions were enabled for the db). here is my final impl. i'm not quite sure about the putifabsent() method (don't know all the scenarios that might be encountered with/without transactions). how does this look (and, if reasonable, could it be rolled into existing codebase)?:
    public class StoredConcurrentMap extends StoredMap
    {
    
      public StoredConcurrentMap(
          Database database, EntryBinding keyBinding,
          EntryBinding valueEntityBinding, boolean writeAllowed)
      {
        super(database, keyBinding, valueEntityBinding, writeAllowed);
      }
    
      public Object putIfAbsent(Object key, Object value)
      {
        Object oldValue = null;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true);
          while(true) {
            if(OperationStatus.SUCCESS ==
               cursor.putNoOverwrite(key, value, false)) {
              // we succeeded
              break;
            } else if(OperationStatus.SUCCESS ==
                      cursor.getSearchKey(key, null, false)) {
              // someone else beat us to it
              oldValue = cursor.getCurrentValue();
              break;
            }
    
            // we couldn't put and we couldn't get, try again
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return oldValue;
      }
    
      public boolean remove(Object key, Object value)
      {
        boolean removed = false;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            Object curValue = cursor.getCurrentValue();
            if(ObjectUtils.equals(curValue, value)) {
              cursor.delete();
              removed = true;
            }
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return removed;
      }
    
      public boolean replace(Object key, Object oldValue, Object newValue)
      {
        boolean replaced = false;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            Object curValue = cursor.getCurrentValue();
            if(ObjectUtils.equals(curValue, oldValue)) {
              cursor.putCurrent(newValue);
              replaced = true;
            }
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return replaced;
      }
    
      public Object replace(Object key, Object value)
      {
        Object curValue = null;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            curValue = cursor.getCurrentValue();
            cursor.putCurrent(value);
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return curValue;
      }
    }
  • 11. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    greybird Expert
    Currently Being Moderated
    Sorry the slow reply.
    First, thanks for all your help with this!
    You're welcome, thanks for your effort and ideas!
    So, in an effort to see if i could write these
    methods "correctly", i dove into the lower level api.
    I ended up reimplementing the previous methods and
    implementing the remaining method. (for one thing,
    the previous implementations did not work if
    transactions were enabled for the db). here is my
    final impl. i'm not quite sure about the
    putifabsent() method (don't know all the scenarios
    that might be encountered with/without
    transactions). how does this look (and, if
    reasonable, could it be rolled into existing
    codebase)?:
    Thanks again for these ideas. We'll go ahead and add an implementation of the ConcurrentMap methods for a future release.

    I can't see anything wrong with your implementation. The loop you're using for putIfAbsent is correct -- it works with and without transactions. With transactions, the loop is not necessary if the Serializable isolation level is configured, but this is not the default -- what you've done should work in all cases.

    Mark
  • 12. Re: thoughts on adding ConcurrentMap methods to StoredMap?
    561346 Newbie
    Currently Being Moderated
    hey,
    just wanted to follow up on this a little. after using the concurrent implementation for a while (with transactions enabled), i experimented with extending the api to add a "transaction-like" extension to the concurrentmap api. this basically blends the concurrentmap api with the option of better performance when transactions are enabled in the underlying bdb. there are three methods added, which basically allow for "locking" a given key and "committing" or "rolling back" the updates as appropriate (if transactions are not enabled, the new methods are largely no-ops, and you get normal concurrent map behavior). thought i'd post the new code in case you were interested in providing this facility to other users (or, you could just stick with the original version posted above).

    Example usage:
    Object key;
    try {
      Object value = map.getForUpdate(key);
      // ... run concurrent ops on value ...
      map.replace(key, value, newValue);
      map.completeUpdate(key);
    } finally {
      map.closeUpdate(key);
    }
    New implementation:
    public class StoredConcurrentMap extends StoredMap
    {
      private final ThreadLocal<UpdateState> _updateState =
        new ThreadLocal<UpdateState>();
    
      public StoredConcurrentMap(
          Database database, EntryBinding keyBinding,
          EntryBinding valueEntityBinding, boolean writeAllowed)
      {
        super(database, keyBinding, valueEntityBinding, writeAllowed);
      }
      
      public Object getForUpdate(Object key)
      {
        if(_updateState.get() != null) {
          throw new IllegalStateException(
              "previous update still outstanding for key " +
              _updateState.get()._key);
        }
        UpdateState updateState = new UpdateState(key, beginAutoCommit());
        _updateState.set(updateState);
        
        DataCursor cursor = null;
        Object value = null;
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            // takeover ownership of the cursor
            value = updateState.loadCurrentValue(cursor);
            cursor = null;
          }
          closeCursor(cursor);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, false);
        }
        return value;
      }
    
      public void completeUpdate(Object key)
      {
        UpdateState updateState = getUpdateState(key);
        if(updateState != null) {
          try {
            updateState.clearCurrentValue(this);
            commitAutoCommit(updateState._doAutoCommit);
            // only clear the reference if everything succeeds
            _updateState.set(null);
          } catch(DatabaseException e) {
            throw new RuntimeExceptionWrapper(e);
          }
        }
      }
    
      public void closeUpdate(Object key)
      {
        UpdateState updateState = getUpdateState(key);
        if(updateState != null) {
          // this op failed, abort (clear the reference regardless of what happens
          // below)
          _updateState.set(null);
          try {
            updateState.clearCurrentValue(this);
            view.getCurrentTxn().abortTransaction();
          } catch(DatabaseException ignored) {
          }
        }
      }
      
      public Object putIfAbsent(Object key, Object value)
      {
        UpdateState updateState = getUpdateState(key);
        if(valueExists(updateState)) {
          return updateState._curValue;
        }
        
        Object oldValue = null;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true);
          while(true) {
            if(OperationStatus.SUCCESS ==
               cursor.putNoOverwrite(key, value, false)) {
              // we succeeded
              break;
            } else if(OperationStatus.SUCCESS ==
                      cursor.getSearchKey(key, null, false)) {
              // someone else beat us to it
              oldValue = cursor.getCurrentValue();
              break;
            }
    
            // we couldn't put and we couldn't get, try again
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return oldValue;
      }
    
      public boolean remove(Object key, Object value)
      {
        UpdateState updateState = getUpdateState(key);
        if(valueExists(updateState)) {
          if(ObjectUtils.equals(updateState._curValue, value)) {
            try {
              updateState._cursor.delete();
              updateState.clearCurrentValue(this);
              return true;
            } catch (Exception e) {
              throw handleException(e, false);
            }
          } else {
            return false;
          }
        }
    
        boolean removed = false;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            Object curValue = cursor.getCurrentValue();
            if(ObjectUtils.equals(curValue, value)) {
              cursor.delete();
              removed = true;
            }
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return removed;
      }
    
      public boolean replace(Object key, Object oldValue, Object newValue)
      {
        UpdateState updateState = getUpdateState(key);
        if(valueExists(updateState)) {
          if(ObjectUtils.equals(updateState._curValue, oldValue)) {
            try {
              updateState.replaceCurrentValue(newValue);
              return true;
            } catch (Exception e) {
              throw handleException(e, false);
            }
          } else {
            return false;
          }
        }
        
        boolean replaced = false;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            Object curValue = cursor.getCurrentValue();
            if(ObjectUtils.equals(curValue, oldValue)) {
              cursor.putCurrent(newValue);
              replaced = true;
            }
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return replaced;
      }
    
      public Object replace(Object key, Object value)
      {
        UpdateState updateState = getUpdateState(key);
        if(valueExists(updateState)) {
          try {
            return updateState.replaceCurrentValue(value);
          } catch (Exception e) {
            throw handleException(e, false);
          }
        }
    
        Object curValue = null;
        DataCursor cursor = null;
        boolean doAutoCommit = beginAutoCommit();
        try {
          cursor = new DataCursor(view, true, key);
          if(OperationStatus.SUCCESS == cursor.getNextNoDup(true)) {
            curValue = cursor.getCurrentValue();
            cursor.putCurrent(value);
          }
          closeCursor(cursor);
          commitAutoCommit(doAutoCommit);
        } catch (Exception e) {
          closeCursor(cursor);
          throw handleException(e, doAutoCommit);
        }
        return curValue;
      }
    
      
      /**
       * @return the current UpdateState for the given key, if any, {@code null}
       *         otherwise.
       */
      private UpdateState getUpdateState(Object key)
      {
        UpdateState updateState = _updateState.get();
        if((updateState != null) && (ObjectUtils.equals(updateState._key, key))) {
          return updateState;
        }
        return null;
      }
      
      /**
       * @return {@code true} if the update state exists and found a value (which
       *         is currently locked)
       */
      private boolean valueExists(UpdateState updateState)
      {
        return((updateState != null) && (updateState._cursor != null));
      }
    
    
      /**
       * Maintains state about an object loaded in a
       * {@link StoredConcurrentMap#getForUpdate} call.
       */
      private static class UpdateState
      {
        public final Object _key;
        public final boolean _doAutoCommit;
        public DataCursor _cursor;
        public Object _curValue;
    
        private UpdateState(Object key, boolean doAutoCommit) {
          _key = key;
          _doAutoCommit = doAutoCommit;
        }
    
        /**
         * Loads the current value from the given cursor, and maintains a
         * reference to the given cursor and the loaded value.
         */
        public Object loadCurrentValue(DataCursor cursor)
          throws DatabaseException
        {
          _cursor = cursor;
          _curValue = _cursor.getCurrentValue();
          return _curValue;
        }
    
        /**
         * Replaces the current value in the current cursor with the given value.
         * @return the old value
         */
        public Object replaceCurrentValue(Object newValue)
          throws DatabaseException
        {
          Object oldValue = _curValue;
          _cursor.putCurrent(newValue);
          _curValue = newValue;
          return oldValue;
        }
    
        /**
         * Closes the current curstor and clears the references to the cursor and
         * current value.
         */
        public void clearCurrentValue(StoredConcurrentMap map) {
          try {
            map.closeCursor(_cursor);
          } finally {
            _cursor = null;
            _curValue = null;
          }
        }
        
      }
      
    }