11 Replies Latest reply on May 15, 2012 8:18 PM by 934910

    Oracle Coherence database sync while table refresh

    934910
      Hello,

      I am using Oracle Coherence to cache a database table for read only puposes. However, my question is that this table needs to be fully refreshed every night and during this time I want to retain the cache in Oracle Coherence and do not want it to be recycled as well since it has to be available 24X7. Is there a way to do that? I am thinking of using a custom CacheStore class that will have its own local cache of the table or the other approach will be to look up in the shadow table while the primary table is being refreshed? Are there any other optimum ways to do this? The primary consideration for me is fast response.

      Any help will be appreciated.

      Thanks,
      Jay
        • 1. Re: Oracle Coherence database sync while table refresh
          User738616-Oracle
          Hi Jay,

          I would suggest you to create a new cache for the new data (Cache1v2) and a markerCache (replicated-scheme). Update the markerCache after the nightly referesh to use cache (Cache1v2). Please ensure that before accessing the Cache you check which cache to be used from the markerCache.

          During Nightly Refresh:
          - Continue using Cache1v1 ; MarkerCache {Cache1v1}
          - Create a new cache Cache1v2 and load the data
          - Update the MarkerCache {Cache1v2}
          - Wait till all the existing requests to Cache1v1 has been completed and destroy cache Cache1v1

          Hope this helps!

          Cheers,
          NJ
          • 2. Re: Oracle Coherence database sync while table refresh
            934910
            Thanks for your reply NJ. The question is that how will I seemlessly transition from Cache1v1 to Cache1v2. If I am understanding correctly you are suggesting me to stop one JVM server process and then start the other JVM server process on the same IP address and port so that client do not have to change anything. If there is some request that comes during the server stop and start what will happen to them? This is not a distribued environment that will be able to handle it automatically (Sorry forgot to mention that.. My apologies..)
            Please correct me if I am not following something..

            Thanks again,
            Jay
            • 3. Re: Oracle Coherence database sync while table refresh
              User738616-Oracle
              Hi Jay,

              No, I am not suggesting to stop or start the JVMs. Everything will continue run as it was running ...

              The Nightly refresh process that loads the data will load in non-active version of the cache (dataCache). Once the load is completed, it will update the MarkerCache (markerCache) with this version. All the client requests will follow something like this:
              //Lookup in markerCache which version of the data cache is active
              NamedCache markerCache=CacheFactory.getCache("markerCache");
              String str=markerCache.get("dataCacheName");
              
              //The cache which has the refreshed and updated data (dataCachev1 or dataCachev2)
              NamedCache dataCache=CacheFactory.getCache(str);
              
              For the nightly refresh process, you will do something like this:
              NamedCache markerCache=CacheFactory.getCache("markerCache");
              String str=markerCache.get("dataCacheName");
              
              if ("dataCachev1".equals(str)){
                     NamedCache newDataCache=CacheFactory.getCache("dataCachev2");
                     //load the data{ }
                     //update the marker Cache with this newly loaded cache; from now on the client will start using this newly loaded cache
                     markerCache.get("dataCacheName","dataCachev2");
                     // wait for few minutes to let all the old request complete
                     //destroy the old cache
                     CacheFactory.getCache("dataCachev1").destroy();
              } else{
                  //do the reverse for dataCachev1 and dataCachev2
              }
              The only thing you need to note is that there might still be some requests using the dataCachev1 when it is destroyed so you need to handle the exception gracefully at the client application.

              Please note, that you should optimize the code to minimize the time spent in lookup of the markerCache by using Maplisteners as appropriate.

              Hope this helps!

              Cheers,
              NJ
              • 4. Re: Oracle Coherence database sync while table refresh
                Jonathan.Knight
                Hi Jay,

                Can you expand on your requirements a little more? NJ's solution may be a bit over complicated depending on what you actually want to achieve.

                My understanding is that you have a read-only cache that is loaded from a DB table and you want to update the DB table without affecting the cache.

                I have a few questions...
                <li>Do you load all the data into the cache when your application starts or do you use a CacheStore to load data into the cache on demand?
                <li>How do applications access the data? Is it just gets/queries? Do you have listeners or ContinuousQueryCaches?
                <li>Do you need applications that read the cache to read consistent data? That is - does all the data in the cache need to be updated atomically in one go?

                The answer to the last one is quite important. If you answer "yes", Do you have situations like this, where an application needs to read multiple entries from the cache before doing something...
                <li>Application reads some data
                <li>You reload the new data into cache (or flip the cache in NJ's solution).
                <li>Application reads the next piece of data
                <li>Application processes data but the two bits of data are out of sync

                Going back to NJs solution, assuming you did not have listeners, CQCs or anything else that would fail if you suddenly destroyed the cache. I would wrap all of the code the works out what cache to call inside a NamedCache wrapper and then use the cache configuration to return the wrapper rather than the real caches. The client code would then just use the cache like normal without having to do all of the messy code to work out which real cache to go to.

                Once you've posted some more details on the requirement we can give some better advice.

                JK
                • 5. Re: Oracle Coherence database sync while table refresh
                  934910
                  Thanks NJ for your answer. I was able to understand your suggestion much better now.

                  Jonathan,

                  Let me try to answer your questions:

                  ->Do you load all the data into the cache when your application starts or do you use a CacheStore to load data into the cache on demand?
                  I wish to load the data into the cache as the application starts and then update the cache continuously based on expiration of cache entries for better response time to the client requests.

                  How do applications access the data? Is it just gets/queries? Do you have listeners or ContinuousQueryCaches?
                  I have real time client with near cache.

                  Do you need applications that read the cache to read consistent data? That is - does all the data in the cache need to be updated atomically in one go?
                  Not all the data need to be updated atomically in the cache.

                  I like your idea of creating the Named Cache wrapper on the server side instead of client side so that the details are abstract from the client for what NJ recommended.

                  Thanks,
                  Jay
                  • 6. Re: Oracle Coherence database sync while table refresh
                    934910
                    NJ,

                    In the marker cache switch case, if my client is real time client with near cache then what will happen to my near cache? Will that be updated automatically as a push from the server or there will be near cache created for both the caches (cache 1 and cache2)?

                    Thanks,
                    Jay
                    • 7. Re: Oracle Coherence database sync while table refresh
                      User738616-Oracle
                      Hi Jay,

                      There will be seperate near cache for cache1 and cache2 so once the markerCache is updated your real time client will start reading from the appropriate cache.

                      Hope this helps!

                      Cheers,
                      NJ
                      • 8. Re: Oracle Coherence database sync while table refresh
                        Jonathan.Knight
                        As the Cache Switching Pattern seems to get suggested as a solution to a few things on this forum here is a write up of how to do the whole cache switching thing transparently to the client code http://thegridman.com/coherence/switching-cache-pattern-in-oracle-coherence/ along with some of the caveats to the pattern.

                        JK
                        • 9. Re: Oracle Coherence database sync while table refresh
                          934910
                          Thanks NJ for the clarification.

                          Jonathan,

                          Thanks for hosting such a great article on cache switching. However, I have a C++ client so guess the cache-config what you have will not work since it uses "com.thegridman.coherence.cache.SwitchingNamedCache" java class. Is there a way to have SwitchingNamedCache on the server side?

                          Thanks,
                          Jay

                          Edited by: 931907 on May 14, 2012 1:31 PM
                          • 10. Re: Oracle Coherence database sync while table refresh
                            Jonathan.Knight
                            Hi Jay,

                            Ah... it's a while since I did C++ and all I remember from that time was Visual Studio driving me mad. I assume it is possible to write the same SwitchingNamedCache wrapper class in C++ that I wrote in Java as the same classes and interfaces I used are present in the C++ client. I don't know if class-scheme works in the C++ cache config either though. Maybe one day when I'm in the mood for some self punishment I might get round to trying out a bit of C++ again :-)

                            You might be able to put the SwitchingNamedCache on the Extend Proxy node that your client connects to, providing your cluster has separate storage disabled Extend Proxy nodes. If you did this though you would not be able to use anything that relied on events including near caches on the C++ client.

                            JK
                            • 11. Re: Oracle Coherence database sync while table refresh
                              934910
                              Thanks JK. So going by all the facts the best option for me will be to go with the C++ implementation of SwitchingNamedCache and doing the below since I am not sure if class-scheme works for C++.

                              NamedCache cacheA = CacheFactory.getCache("cache-a");
                              NamedCache cacheB = CacheFactory.getCache("cache-b");
                              NamedCache flagCache = CacheFactory.getCache("flag-cache");
                              NamedCache cache = new SwitchingNamedCache(cacheA, cacheB, flagCache, "cache-a-b-flag");
                              ....

                              Thanks,
                              Jay