This discussion is archived
4 Replies Latest reply: Jun 24, 2013 11:47 PM by user567421 RSS

Custom Expiry Implementation Advice Needed

user567421 Newbie
Currently Being Moderated

Hi Guys,
I need help identifying best implementation approach.  A requirement is to transitionally persist data that is getting expired from a cache to either MQ queue or a database.  Here I need to make sure if persistence operation fails (ie database/mq is down), entry should be retained in cache until other systems are operational again.

 

Implementation Options:

1) Periodically scan all cached entries and manually identify which ones are ready for expiration.  Then use coherence transactions and link it through XA with persistence operation.

2) Periodically scan all cached entries and manually identify which ones are ready for expiration.  Remove entry only if persistence operation is successful.

3) Use Custom Eviction Policy?  I didn't see a good way of plugging in there.

4) Extend LocalCache.Entry and override isExpired method.  Not too sure if that is a good option or if that will even work.

5) ??

 

Thanks,

Dmitriy

  • 1. Re: Custom Expiry Implementation Advice Needed
    Jonathan.Knight Expert
    Currently Being Moderated

    Hi Dmitriy,

     

    Personally I wouldn't do anything off of the back of Coherence's built in expiry. Going through your options...

    1) While you are scanning the entries Coherence has probably evicted them. Whenever you do an action on a cache the first thing Coherence does is evict anything that has expired. So if you tried to scan a cache Coherence would evict the entries before you could scan them. A second point, I wouldn't use Coherence transactions for what you suggest as you will find they are too restrictive to be much use.

     

    2) See, 1 above - again you will have trouble scanning the cache

     

    3) Custom eviction policies are really just ways to tell Coherence if it can evict something

     

    4) This could get tricky and trying to do some sort of persistence off of the back of isExpired is just wrong - methods should do what the name says, you have no idea how often Coherence will call your overriden method and if it is doing persistence you could have a big impact on performance of your cluster.

     

    5) I would choose this option

     

    I am pretty sure that triggers get fired when an entry is evicted so you could add a trigger to the cache that will do the persistence when an entry is evicted. When an entry is evicted you basically get a synthetic remove event which you can detect in the trigger (although is it not nice code as you need to use reflection to get to the isSynthetic() method). A trigger that looks like this might be workable...

     

    import com.tangosol.util.MapTrigger;
    import com.tangosol.util.ValueExtractor;
    import com.tangosol.util.extractor.ReflectionExtractor;
    public class MyTrigger implements MapTrigger {
        private static final ValueExtractor syntheticExtractor = new ReflectionExtractor("isSynthetic");
      @Override
        public void process(Entry entry) {
            if (entry.isOriginalPresent() && !entry.isPresent() && isSyntheticEntryEvent(entry)) {
                // The entry is being evicted...
            }
        }
      private boolean isSyntheticEntryEvent(Entry entry) {
            boolean isSynthetic;
            try {
                Object synthetic = syntheticExtractor.extract(entry);
                isSynthetic = (synthetic instanceof Boolean) ? (Boolean) synthetic : false;
            } catch (Throwable e) {
                isSynthetic = false;
            }
            return isSynthetic;
        }
    }

     

    JK

  • 2. Re: Custom Expiry Implementation Advice Needed
    user567421 Newbie
    Currently Being Moderated

    Oh yeah.  I forgot to list MapTrigger option.  I did look into it, but the issue here is that if persistence fails for one reason or another, entry will no longer exist in cache.

     

    For options 1 & 2 I meant to disable automated expiry altogether and have some type of timer thread scan cache entries to decide when they are ready for expiration and transactionally make sure that both persistence and removal from cache complete successfully.

     

    For eviction policy, it is kind of what I need.  I do not want entry to be removed from cache if persistence fails even if it was expired.

  • 3. Re: Custom Expiry Implementation Advice Needed
    Jonathan.Knight Expert
    Currently Being Moderated

    Is there a reason that you only want to persist something when it is about to expire from the cache? Why don't you persist data as it is put into the cache then you don;t need to worry about expiry? The project I work on at my current client does persistence on a trigger so if anything goes wrong with the persistence call, which is rare as it goes to a very reliable message queue, then the put/update will fail. This way we know that if the data is in the cache then it is also persisted.

     

    JK

  • 4. Re: Custom Expiry Implementation Advice Needed
    user567421 Newbie
    Currently Being Moderated

    Yeah.  In this case cache is meant to be used as a buffer to accumulate data over time or until buffer(cache size) fills up with specific filters per specified duration/expiration.  Upon expiration an action needs to be taken.

     

    Messages cannot be lost, but extreme scenarios such as whole coherence cluster going down are acceptable. 

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points