1 2 Previous Next 23 Replies Latest reply: Jul 3, 2012 5:12 PM by User738616-Oracle RSS

    store vs. storeAll and write behind

    785631
      Hi, so we have these write-behind caches, backed with a cache store, that receive calls into the store() and storeAll() methods.

      I'm trying to understand when one is used vs. another. Our write-behind delay is 10 seconds.

      Now, i can see how the code would always decide to use storeAll(), since everything is written at intervals (everything in the last 10 secs gets one storeAll() call made), but we see both store() and storeAll() get called.

      So how does coherence decide which to call? Eg, does it have anything to with how the application calls the cache (put(), vs. putAll(), vs. entry processor logic)?

      Thx.
        • 1. Re: store vs. storeAll and write behind
          User738616-Oracle
          Hi,

          This is mainly due to the <bundling-config> configuration in your cachestore configuration.

          Hope this helps!

          Cheers,
          NJ
          • 2. Re: store vs. storeAll and write behind
            robvarga
            user738616 wrote:
            Hi,

            This is mainly due to the <bundling-config> configuration in your cachestore configuration.

            Hope this helps!

            Cheers,
            NJ
            Hi NJ,

            he uses write-behind, so IMHO it has nothing to do with bundling-config.

            Best regards,

            Robert
            • 3. Re: store vs. storeAll and write behind
              robvarga
              user9222505 wrote:
              Hi, so we have these write-behind caches, backed with a cache store, that receive calls into the store() and storeAll() methods.

              I'm trying to understand when one is used vs. another. Our write-behind delay is 10 seconds.
              Write-behind can decide to use store() if there is only one ripe/soft-ripe entry. It depends on timing.

              Also, you did not mention how many cache nodes you have, but if you have multiple, it is possible even case there were multiple updates that only one entry of those changed were owned on a node which leads to store() instead of storeAll().

              It also gives relevant information if you write out the thread-name for store() operations.
              Now, i can see how the code would always decide to use storeAll(), since everything is written at intervals (everything in the last 10 secs gets one storeAll() call made), but we see both store() and storeAll() get called.

              So how does coherence decide which to call? Eg, does it have anything to with how the application calls the cache (put(), vs. putAll(), vs. entry processor logic)?

              Thx.
              It has more to do with entry-ownership and timing. Did you specify write-batch-factor? Specifying a value closer to 1 will increase the probability of storeAll, whereas if you have a low or zero write-batch-factor, then there is less likelyhood that you will get soft-ripe entries when the first entry becomes fully-ripe. If the puts are spaced apart more than (the time the write operation takes plus write-delay*write-batch-factor), then it is conceivable that you will get single writes all the time.

              You may want to look at the documentation of write-batch-factor, too.

              Best regards,

              Robert

              Edited by: robvarga on Jun 26, 2012 6:17 PM
              • 4. Re: store vs. storeAll and write behind
                785631
                Ok, so it sounds like it has nothing to do with how the application calls the cache; it's all about timing on the storage nodes. And we do have quite a number, > 100 storage nodes.

                So because of the timing, maybe you'll get a storeAll(), or just a store().

                The behavior we're trying to explain, is that the rate of store() calls increases over time. This is over several days/weeks of a system being in production. The rate of storeAll() calls does not, it remains constant. We only see this with one of many write-behind caches; all other caches tend to be constant store()/storeAll() calls.

                Of course, the first suspect is that our puts are also increasing over time, but that's not the case; our puts are constant. It's almost as if the objects in this particular cache are being "moved around" so that the density goes down within the nodes, and thus more store() calls are issued due to the timing.

                Does that make any sense?
                • 5. Re: store vs. storeAll and write behind
                  robvarga
                  user9222505 wrote:
                  Ok, so it sounds like it has nothing to do with how the application calls the cache; it's all about timing on the storage nodes. And we do have quite a number, > 100 storage nodes.

                  So because of the timing, maybe you'll get a storeAll(), or just a store().

                  The behavior we're trying to explain, is that the rate of store() calls increases over time. This is over several days/weeks of a system being in production. The rate of storeAll() calls does not, it remains constant. We only see this with one of many write-behind caches; all other caches tend to be constant store()/storeAll() calls.

                  Of course, the first suspect is that our puts are also increasing over time, but that's not the case; our puts are constant. It's almost as if the objects in this particular cache are being "moved around" so that the density goes down within the nodes, and thus more store() calls are issued due to the timing.

                  Does that make any sense?
                  It would be more relevant if you did not only look at the frequence of the storeAll calls, but also the total number of entries passed to storeAll calls.

                  Actually there is one more thing which can cause store() calls: eviction from the backing map due to size/expiry also causes the evicted entry to be written out synchronously if it was evicted. Do you have size-based or time-based eviction configured for that one cache or any others? Also, the thread which would execute the store() calls would never be the write-behind thread, instead it would be a worker thread or the service thread if you don't have a thread-pool configured with specifying the thread-count for the service.

                  As I mentioned, it is also worth checking which thread the store/storeAll method was executed on.

                  Best regards,

                  Robert

                  Edited by: robvarga on Jun 27, 2012 5:35 PM
                  • 6. Re: store vs. storeAll and write behind
                    785631
                    Yes, i understand i should be looking at the total, it's just that our metrics system (which is very generic) doesn't know how to do that. But we should look into it.

                    Anyways, when you say "eviction will cause store() to be called synchronously", you do mean only in the case where the write-behind thread didn't have a chance to write the object yet. Our delay is 10 seconds, so if an object got put, and evicted within those 10 seconds, then the synch write would kick in; right? You would not be calling store() on every eviction, right?

                    But certainly something to look into.
                    • 7. Re: store vs. storeAll and write behind
                      User738616-Oracle
                      Hi,

                      AFAIU <bundling-config> is used for bundling the ripe entries and it goes for write-behind cache strategy as well. I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.

                      Hope this helps!

                      Cheers,
                      NJ
                      • 8. Re: store vs. storeAll and write behind
                        robvarga
                        user738616 wrote:
                        Hi,

                        AFAIU <bundling-config> is used for bundling the ripe entries and it goes for write-behind cache strategy as well.
                        Nope. It is write-delay and write-batch-factor which governs batching for soft-ripe and ripe entries on the write-behind thread. I don't think bundling-config would be used for grouping eviction-caused store() calls either.

                        The reason why I think this is so is that bundling-config is for grouping multiple operations from multiple threads together whereas eviction is operating under a lock.

                        I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.
                        And I am pretty sure that the write-batch-factor governs from what proportion of the write-delay period the not-yet-ripe entries are considered soft-ripe to be added to the fully-ripe entry/entries.

                        The only effect operation-bundling is supposed to have on write-behind is that storeAll is going to be called for even a single entry if operation-bundling is configured, but write-batch-factor and write-delay still decides what is ripe and soft-ripe.


                        You might want to read the documentation of write-batch-factor on page http://docs.oracle.com/cd/E24290_01/coh.371/e22837/appendix_cacheconfig.htm


                        Best regards,

                        Robert
                        • 9. Re: store vs. storeAll and write behind
                          User738616-Oracle
                          Hi Rob,

                          I am not sure why you disagree with me as I meant what you have written in your last post:
                          The only effect operation-bundling is supposed to have on write-behind is that storeAll is going to be called for even a single entry if operation-bundling is configured, but write-batch-factor and write-delay still decides what is ripe and soft-ripe.
                          Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.

                          Cheers,
                          NJ
                          • 10. Re: store vs. storeAll and write behind
                            robvarga
                            user738616 wrote:
                            Hi Rob,

                            I am not sure why you disagree with me as I meant what you have written in your last post:
                            The only effect operation-bundling is supposed to have on write-behind is that storeAll is going to be called for even a single entry if operation-bundling is configured, but write-batch-factor and write-delay still decides what is ripe and soft-ripe.
                            Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.

                            Cheers,
                            NJ
                            Hi NJ,

                            Because you may have meant what I said but what you wrote even in the last post was contradictory to what I wrote, regardless of however you actually meant it. And people read what you write and not what you mean. And what you wrote was incorrect.

                            Let me try to explain it in a bit more detail:

                            Configuration A: You don't have operation-bundling configured:

                            A1: You have 1 ripe record in the queue.
                            A2: You have multiple ripe and/or soft-ripe records in the queue.

                            Configuration B: You have operation-bundling configured:

                            B1: You have 1 ripe record in the queue.
                            B2: You have multiple ripe and/or soft-ripe records in the queue.


                            Let me quote you again:

                            >
                            I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.
                            >

                            This means that storeAll() is going to be called in B2 case. It is true.


                            Quoting you again:

                            >
                            Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.
                            >

                            I can't fail to interpret this in any other way, that you claim that if you did not have bundling configuration (A1 and A2 case) then storeAll() would not be called.

                            And it is definitely not correct.

                            My point was exactly that it is going to be called in A2 case, so operation-bundling does not make a bit of difference at all if you have multiple ripe/soft-ripe entries (A2 and B2 cases).


                            My other point was that storeAll() is also going to be called in B1 case, and this is the only difference bundling-config makes for write-behind: if you have only a single ripe entry, then if you have operation-bundling, then storeAll() is going to be called, if you do not have bundling-config then store() is going to be called.

                            So when bundling-config allows storeAll() to be called is not the multiple-entry but the single-entry case.


                            So in short, you effectively wrote that storeAll() is called in B2 case and is not called in A2 case, and I (and the documentation) claimed that it is called in A2, B1 and B2 cases.

                            Hope it is clear for you now.


                            Best regards,

                            Robert
                            • 11. Re: store vs. storeAll and write behind
                              robvarga
                              user9222505 wrote:
                              Yes, i understand i should be looking at the total, it's just that our metrics system (which is very generic) doesn't know how to do that. But we should look into it.

                              Anyways, when you say "eviction will cause store() to be called synchronously", you do mean only in the case where the write-behind thread didn't have a chance to write the object yet. Our delay is 10 seconds, so if an object got put, and evicted within those 10 seconds, then the synch write would kick in; right? You would not be calling store() on every eviction, right?

                              But certainly something to look into.
                              Eviction only calles store() on entries for which the last change have not been successfully written by the write-behind thread.

                              Best regards,

                              Robert
                              • 12. Re: store vs. storeAll and write behind
                                User738616-Oracle
                                Hi Rob,

                                Thanks for this detailed explanation!
                                Because you may have meant what I said but what you wrote even in the last post was contradictory to what I wrote, regardless of however you actually meant it. And people read what you write and not what you mean. And what you wrote was incorrect.

                                Let me try to explain it in a bit more detail:

                                Configuration A: You don't have operation-bundling configured:

                                A1: You have 1 ripe record in the queue.
                                A2: You have multiple ripe and/or soft-ripe records in the queue.

                                Configuration B: You have operation-bundling configured:

                                B1: You have 1 ripe record in the queue.
                                B2: You have multiple ripe and/or soft-ripe records in the queue.


                                Let me quote you again:

                                >
                                I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.
                                >

                                This means that storeAll() is going to be called in B2 case. It is true.


                                Quoting you again:

                                >
                                Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.
                                >

                                I can't fail to interpret this in any other way, that you claim that if you did not have bundling configuration (A1 and A2 case) then storeAll() would not be called.
                                I guess you misinterpreted! I never meant and never mentioned that storeAll() will not be called in A1/A2 case but I meant that storeAll() will be called randomly in A2 case (it is possible that all the write are done using store() as well ) and you can use bundling-config to bundle your writes to the DB much more effectively (which is B2 case).

                                >
                                And it is definitely not correct.

                                My point was exactly that it is going to be called in A2 case, so operation-bundling does not make a bit of difference at all if you have multiple ripe/soft-ripe entries (A2 and B2 cases).


                                My other point was that storeAll() is also going to be called in B1 case, and this is the only difference bundling-config makes for write-behind: if you have only a single ripe entry, then if you have operation-bundling, then storeAll() is going to be called, if you do not have bundling-config then store() is going to be called.

                                So when bundling-config allows storeAll() to be called is not the multiple-entry but the single-entry case.


                                So in short, you effectively wrote that storeAll() is called in B2 case and is not called in A2 case, and I (and the documentation) claimed that it is called in A2, B1 and B2 cases.
                                No I did not write that it wont be called in A2 case (though, def. it is possible that all write are done using store()) but I wrote that bundling-config would guarantee that bundles are used for writing to DB.

                                >
                                Hope it is clear for you now.


                                Best regards,

                                Robert
                                Cheers,
                                NJ

                                Edited by: user738616 on Jun 28, 2012 12:10 PM
                                Changed from A2 to B2 (avoid more confusion)
                                • 13. Re: store vs. storeAll and write behind
                                  robvarga
                                  user738616 wrote:
                                  Hi Rob,

                                  Thanks for this detailed explanation!
                                  Because you may have meant what I said but what you wrote even in the last post was contradictory to what I wrote, regardless of however you actually meant it. And people read what you write and not what you mean. And what you wrote was incorrect.

                                  Let me try to explain it in a bit more detail:

                                  Configuration A: You don't have operation-bundling configured:

                                  A1: You have 1 ripe record in the queue.
                                  A2: You have multiple ripe and/or soft-ripe records in the queue.

                                  Configuration B: You have operation-bundling configured:

                                  B1: You have 1 ripe record in the queue.
                                  B2: You have multiple ripe and/or soft-ripe records in the queue.


                                  Let me quote you again:

                                  >
                                  I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.
                                  >

                                  This means that storeAll() is going to be called in B2 case. It is true.


                                  Quoting you again:

                                  >
                                  Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.
                                  >

                                  I can't fail to interpret this in any other way, that you claim that if you did not have bundling configuration (A1 and A2 case) then storeAll() would not be called.
                                  I guess you misinterpreted! I never meant and never mentioned that storeAll() will not be called in A1/A2 case but I meant that storeAll() will be called randomly in A2 case (it is possible that all the write are done using store() as well ) and you can use bundling-config to bundle your writes to the DB much more effectively (which is B2 case).
                                  Sorry, but you are now trying to confuse the situation. Once again, let me reiterate, I don't know what you meant. I can only look at what you wrote.

                                  >
                                  bundling-config allows to use storeAll() to write these entries
                                  >

                                  is equivalent with "in the absence of bundling-config storeAll() is not allowed to be called". There is no other way you can logically interpret allows in that sentence while it still retaining any meaning and having some effect on behaviour.

                                  Also, let's see what you now claim, that:

                                  >
                                  storeAll() will be called randomly in A2 case.
                                  >

                                  That is also not correct. In A2 case storeAll() will be called. Period. Not randomly, reliably.


                                  >>
                                  And it is definitely not correct.

                                  My point was exactly that it is going to be called in A2 case, so operation-bundling does not make a bit of difference at all if you have multiple ripe/soft-ripe entries (A2 and B2 cases).


                                  My other point was that storeAll() is also going to be called in B1 case, and this is the only difference bundling-config makes for write-behind: if you have only a single ripe entry, then if you have operation-bundling, then storeAll() is going to be called, if you do not have bundling-config then store() is going to be called.

                                  So when bundling-config allows storeAll() to be called is not the multiple-entry but the single-entry case.


                                  So in short, you effectively wrote that storeAll() is called in B2 case and is not called in A2 case, and I (and the documentation) claimed that it is called in A2, B1 and B2 cases.
                                  No I did not write that it wont be called in A2 case (though, def. it is possible that all write are done using store()) but I wrote that bundling-config would guarantee that bundles are used for writing to DB.
                                  See above. And in A2 case (multiple entries needs storing and not removing) store() is definitely not going to be called. In the A2 case storeAll() is going to be called. This behaviour was present way before bundling-configuration was ever introduced in the context of cache-stores, and before write-through ever supported storeAll().


                                  >
                                  Bundling config guarantees that the storeAll() will be invoked.
                                  >

                                  But the only situation where this guarantee actually makes a difference is when you only have exactly 1 ripe entry to write (and not delete).

                                  When you have multiple entries to write (and not delete), then write-behind is going to call storeAll() on its own, regardless of bundling-configuration, and has been doing so ever since versions released ages ago.


                                  So sorry, but it is quite hard to understand why you are sticking to your claims of bundling-configuration being the thing which allows a behaviour which was present in Coherence for several years before Coherence introducing bundling configuration to be specified or relevant for write-behind and actually cache stores at all.

                                  Just the whole idea of bundling-configuration allowing something which was allowed before bundling-configuration (and thus its absence (i.e. in an existing configuration file) not allowing it) would mean that an upgrade to the version where the configuration option was introduced would be backward-incompatible.

                                  Best regards,

                                  Robert
                                  • 14. Re: store vs. storeAll and write behind
                                    User738616-Oracle
                                    robvarga wrote:
                                    user738616 wrote:
                                    Hi Rob,

                                    Thanks for this detailed explanation!
                                    Because you may have meant what I said but what you wrote even in the last post was contradictory to what I wrote, regardless of however you actually meant it. And people read what you write and not what you mean. And what you wrote was incorrect.

                                    Let me try to explain it in a bit more detail:

                                    Configuration A: You don't have operation-bundling configured:

                                    A1: You have 1 ripe record in the queue.
                                    A2: You have multiple ripe and/or soft-ripe records in the queue.

                                    Configuration B: You have operation-bundling configured:

                                    B1: You have 1 ripe record in the queue.
                                    B2: You have multiple ripe and/or soft-ripe records in the queue.


                                    Let me quote you again:

                                    >
                                    I am sure that if you configure the <operation-bundling> in your <cachestore-scheme> and if there are more than 1 ripe entries within bundle-delay it will call storeAll() method.
                                    >

                                    This means that storeAll() is going to be called in B2 case. It is true.


                                    Quoting you again:

                                    >
                                    Yes the entries that are ready to be written are decided by write-delay and write-batch factor but bundling-config allows to use storeAll() to write these entries.
                                    >

                                    I can't fail to interpret this in any other way, that you claim that if you did not have bundling configuration (A1 and A2 case) then storeAll() would not be called.
                                    I guess you misinterpreted! I never meant and never mentioned that storeAll() will not be called in A1/A2 case but I meant that storeAll() will be called randomly in A2 case (it is possible that all the write are done using store() as well ) and you can use bundling-config to bundle your writes to the DB much more effectively (which is B2 case).
                                    Sorry, but you are now trying to confuse the situation. Once again, let me reiterate, I don't know what you meant. I can only look at what you wrote.
                                    I read what I posted in my last post and makes complete sense to me what I wrote .. not sure what is confusing for you ..

                                    >
                                    >
                                    bundling-config allows to use storeAll() to write these entries
                                    >

                                    is equivalent with "in the absence of bundling-config storeAll() is not allowed to be called". There is no other way you can logically interpret allows in that sentence while it still retaining any meaning and having some effect on behaviour.

                                    Also, let's see what you now claim, that:

                                    >
                                    storeAll() will be called randomly in A2 case.
                                    >

                                    That is also not correct. In A2 case storeAll() will be called. Period. Not randomly, reliably.
                                    This is not what I have seen from my experience ... In A2 case, even if you have 100's of entries waiting for DB write .. it will not call storeAll() always ... it will be ramdomly calling store() and storeAll(). So, I mentioned it is random and I have seen only store() been called even there are bunch of entries queued for DB writes.

                                    >
                                    >
                                    >>>
                                    And it is definitely not correct.

                                    My point was exactly that it is going to be called in A2 case, so operation-bundling does not make a bit of difference at all if you have multiple ripe/soft-ripe entries (A2 and B2 cases).
                                    It is not always true .. you will see a combination of store() and storeAll() even if you have pending entries to be written (A2 case)... An example of A2 case (20 entries in the writebehind queue)

                                    store()
                                    storeAll() - 5 entries
                                    store()
                                    store()
                                    store()
                                    storeAll() - 10 entries
                                    store()

                                    An example of B2 case (20 entries in the writebehind queue):

                                    storeAll() - 13 entries
                                    storeAll() - 7 entries

                                    >>>
                                    >>>
                                    My other point was that storeAll() is also going to be called in B1 case, and this is the only difference bundling-config makes for write-behind: if you have only a single ripe entry, then if you have operation-bundling, then storeAll() is going to be called, if you do not have bundling-config then store() is going to be called.

                                    So when bundling-config allows storeAll() to be called is not the multiple-entry but the single-entry case.
                                    I haven't suggested using bundling-config for single entries but to make sure that the storeAll() is called more reliably.
                                    >>>
                                    >>>
                                    So in short, you effectively wrote that storeAll() is called in B2 case and is not called in A2 case, and I (and the documentation) claimed that it is called in A2, B1 and B2 cases.
                                    No I did not write that it wont be called in A2 case (though, def. it is possible that all write are done using store()) but I wrote that bundling-config would guarantee that bundles are used for writing to DB.
                                    See above. And in A2 case (multiple entries needs storing and not removing) store() is definitely not going to be called. In the A2 case storeAll() is going to be called. This behaviour was present way before bundling-configuration was ever introduced in the context of cache-stores, and before write-through ever supported storeAll().


                                    >
                                    Bundling config guarantees that the storeAll() will be invoked.
                                    >

                                    But the only situation where this guarantee actually makes a difference is when you only have exactly 1 ripe entry to write (and not delete).

                                    When you have multiple entries to write (and not delete), then write-behind is going to call storeAll() on its own, regardless of bundling-configuration, and has been doing so ever since versions released ages ago.


                                    So sorry, but it is quite hard to understand why you are sticking to your claims of bundling-configuration being the thing which allows a behaviour which was present in Coherence for several years before Coherence introducing bundling configuration to be specified or relevant for write-behind and actually cache stores at all.

                                    Just the whole idea of bundling-configuration allowing something which was allowed before bundling-configuration (and thus its absence (i.e. in an existing configuration file) not allowing it) would mean that an upgrade to the version where the configuration option was introduced would be backward-incompatible.
                                    I hope my example above will help why I am recommending it.

                                    >
                                    Best regards,

                                    Robert
                                    Cheers,
                                    NJ
                                    1 2 Previous Next