9 Replies Latest reply: Apr 20, 2012 6:23 AM by 925966 RSS

    Should keep application thin with most logic run in cache server JVMs?

    925966
      Hi all,

      We are implementing an securities trading OMS. We decided to separate application server from Coherence cache server. After studying coherence doc, I believe we should have most of logic run in the JVM of cache servers in the form of Entry Processor(may use backing map directly) or Invocable(invocation service) instead of application JVMs. In this way, application server is thin while the processing load is heavy on cache server JVMs.

      Just wonder whether my rationale above is correct or not. Appreciate for any comments. Thanks.

      Henry
        • 1. Re: Should keep application thin with most logic run in cache server JVMs?
          Jonathan.Knight
          Hi Henry,

          That is not an easy question to answer as where you put functionality really depends on your application. We have quite a complex trade cache and I would say we have quite a spread of where the logic is located. Some is in the storage nodes, some is in storage disabled Extend proxy nodes, some is in other storage disabled cluster members and finally some is in Extend client processes.

          If you think of your Coherence application in a similar way to a DB application, what sort of logic would you put into the DB in say stored procedures, triggers etc. Primarily you want to locate logic that has to deal with querying or mutating data in the cluster, as that is where the data is. More general logic, for example we have a lot of feed handlers, does not usually belong on the storage enabled nodes, but can still go in the cluster on storage disabled members.

          Taking our application as an example. We take trade feeds from a lot of external systems, validate and process the data, providing events and a query API to client.
          <li>Putting the data into the cluster is done with Invocalbles (i.e. in the cluster) - mainly as we have a custom implementation of putAll
          <li>Validation is done with triggers, as is other functionality where we want to apply muations to the data on the way into the cluster - e.g. time stamping the entries.
          <li>We have functionality to mark in-use reference data the runs in the cluster in CacheStore implementations
          <li>We persist updates to the DB by posting messages to a topic which is done in triggers on the storage enabled nodes
          <li>We have logic based on state changes of Trades that also happens in Triggers on storage nodes
          <li>We have custom eviction logic which is done using EntryProcessors
          <li>We have a custom query API which is implemented using Invocables which then run Aggregators on the storage nodes
          <li>We have feed handlers, which I have already mentioned, that run as storage disabled cluster members
          <li>We have a few report generators that run as Extend client processes
          <li>We have a thin GUI, which runs as an Extend client. The actual business logic happens on the Extend proxy node that the GUI connects to

          So, as you can see, the logic is spread all over the place. You really need to look at each piece of functionality on a case by case basis and decide the best place for it.

          JK
          • 2. Re: Should keep application thin with most logic run in cache server JVMs?
            925966
            Hi JK,

            Thanks for detailed explanation. Most of our logic deal with data inside the cluster, so it is more appropriate to "send" the code to the storage-enabled node rather grabbing data to storage-disabled application node and process to avoid network hop traffic, right?

            Besides, what is the difference in functionality between storage disabled cluster members and Extend client processes(non-clustered)? I can invoke Entry Processor and Invocable from either of them, right?

            thanks in advance.
            Henry
            • 3. Re: Should keep application thin with most logic run in cache server JVMs?
              alexey.ragozin
              Hi Henry,

              Performance is tricky thing, sometimes it is better to process data "in place", sometimes it is not. When you processing "in place" you consume storage node CPU and put pressure to Java GC. Storage node's CPU is valuable resource, it may run out quickly. On the other side data transfer does not consume much CPU (your objects are stored as binary blobs on storage node) and modern network hardware are good.

              In one of my project we have put a dedicated layer of storage disabled nodes, extensively caching and processing data, keeping storage nodes dump (and less CPU disabled). This way we were able to scale request through put much better - it is always easier to scale data-less processes even if Coherence doing all heavy lifting for you. On the other hand, more complicated topology - more burden to operations.

              There are lot of factors and lot of trade offs, so my advise is do not thing about such thing too much and do not try all "advanced" Coherence techniques in your first project. Do are prototype, keep it simple, find a real issues, and ask on this forum for help (and you will get it :) ).
              Besides, what is the difference in functionality between storage disabled cluster members and Extend client processes(non-clustered)? I can invoke Entry Processor and Invocable from either of them, right?
              EntryProcessor - yes
              Invocable - yes, but you cannot choose a member, it will be invoked on proxy you are connecting to. I'm extensively using this to expose rich service API via invocation service in RPC style.

              Regards,
              Alexey
              • 4. Re: Should keep application thin with most logic run in cache server JVMs?
                925966
                Hi Alexey,

                For Invocable, in case of Extend client, if one cannot choose a member and it will always invoked on the proxy, then Invocable cannot be used if we want it to run on the node that processing data resides on, isn't it? Thanks.

                Henry
                • 5. Re: Should keep application thin with most logic run in cache server JVMs?
                  alexey.ragozin
                  Hi Henry,
                  For Invocable, in case of Extend client, if one cannot choose a member and it will always invoked on the proxy, then Invocable cannot be used if we want it to run on the node that processing data resides on, isn't it? Thanks.
                  That is correct, but you can work this around but having InvocableTask from Extend client to use invocation API and send another task/tasks. Proxy is a member of cluster so your first task will have full access to cluster invocation API.

                  BTW are you sure you really need invocation with data affinity?

                  Regards,
                  Alexey
                  • 6. Re: Should keep application thin with most logic run in cache server JVMs?
                    Jonathan.Knight
                    alexey.ragozin wrote:
                    BTW are you sure you really need invocation with data affinity?
                    That's a strange question. I would have said that having data affinity makes it even more likely that you will use invocables. You can efficiently target an invocable to a specific cluster member and be sure that all the data that the invocable needs is on that member. Even from an extend client it is easy to do a two stage invocable that can target specific members. Our project relies heavily on data affinity and our entire client API uses invocables, aggregators and occasionaly entry processors. The invocables allow the client side to be a very thin layer with minimal direct cache interaction - most of the work is done on extend proxies.

                    JK
                    • 7. Re: Should keep application thin with most logic run in cache server JVMs?
                      925966
                      Hi JK,

                      In case of Extend client, can the client use explicit locking namedCache.lock("aKey", -1) ? If yes, the lock is held by the proxy on cluster member that client connects to, right? If then, the Extend client crashes, is the lock held indefinitely?

                      thanks,
                      Henry
                      • 8. Re: Should keep application thin with most logic run in cache server JVMs?
                        Jonathan.Knight
                        Hi Henry,

                        If I remember correctly locks from Extend clients are not a good idea and by default locking from the client is disabled in the proxy service.

                        There are two settings for how ownership of a lock works, which are "Thread" or "Member", the default is "Thread" (see lease-granularity here: http://docs.oracle.com/cd/E24290_01/coh.371/e22837/appendix_cacheconfig.htm#BABDCBED)

                        This basically means that by default if you took out a lock from an Extend client then the lock can only be released by a request that comes back on the same service thread on the proxy service as the original lock request - which is unlikely as you have a number of service threads. To get around this you can change the lease-granularity to Member but this means that any unlock request from an extend proxy node can remove any lock that was made by any client connected to that Extend proxy.

                        JK
                        • 9. Re: Should keep application thin with most logic run in cache server JVMs?
                          925966
                          Hi JK,

                          Thanks for quick response. I have some use cases that I need to lock both keys before processing, but EntryProcessor can only be invoked on one key only. Any workaround? If not, using explicit locking seems the only choice. In Extend client case, with "Member" ownership of lock, is there any way an Extend client can determine the lock is held by it (on behalf)? Then it will only unlock when it held the lock and will not unlock those locked by other clients.

                          thanks.
                          Henry