5 Replies Latest reply on Oct 19, 2012 3:38 PM by Honghsi Lo-Oracle

    Segregating services per server (performance considerations)

      Using Tuxedo 11.1 on Solaris....

      Question is about performance/admin considerations when segregating services per server. We have a development gig that is looking to shave about 250 services down to 11 situations offered as 11 services. The idea is that there is an abundance of duplicate services that roughly return the same customer info and it would be logically better to package this overflow of "customization" (done mainly to meets specific needs of an subscriber) into a number of customer situations (like "give me everything about contact information", or "everything about relationships to other people"). These service situations will likely deliver more data and funnel obviously more calls into a single bottleneck (that has to be scaled horizontally). For example, we have services like "retrieve a customer identification" that get called 20 times a second (average 20ms) across 3 domains (against the same database). Getting the "identification" of somebody can have like 20 different flavors despite the return being somewhat polymorphic (maybe an extra property here and there but the base info is the same).

      What I am wondering is the best way to package these 11 situations/services? Put them all info a single Tuxedo server then clopt out instances with specific services (likely just a single service). Or one service per server for readability? If I pile everything in to a single server what is the memory hit when clopting? Do only the services being clopted get put in memory or everything defined to the server? Not likely a serious issue for us (given the size of our park) but curious.

      A rough ballpark (without detail knowledge of exactly how development is implementing) is that a service might potentially have to deal with 20 c/s * 20 (different flavors today) * 3 (domains) = 1200 calls per second. ;-)

      What is the effect on the gateway domain? A service is published singularly outwards from the gateway but mapped inwards to potentially MANY instances.

      Edited by: MatthewDavidYoung on 2012-mar-23 00:35

      Edited by: MatthewDavidYoung on 2012-mar-23 02:37
        • 1. Re: Segregating services per server (performance considerations)
          Todd Little-Oracle
          Hi Matthew,

          In general the grouping of services in to servers is based upon two considerations. One is that the services are usually logically related and thus share a lot of code. The other consideration is in general to segragate long running services from short running services. Again, this isn't a hard and fast rule, just a good rule of them. The advantage of this is that short running service requests won't get stuck behind long running requests. Clearly setting up MSSQ sets can help aleviate this problem, although it is still possible for short running requests to get stuck behind longer running requests if there is a burst of long running requests that end up making all the servers busy.

          In general how services are divided into servers won't have an impact on throughput. Throughput is determined by having an adequate number of servers configured to utilize the available resources. Where segregation may make sense is to help keep resource utilization more constant. Thus if you have some services that are more CPU bound and others that are more I/O or database bound, by grouping like services togethere into a server, you can control how much concurrency occurs and thus control the maximum level of resource consumption. So if server A contains CPU intensive services so generally consumes mostly CPU time, you would configure no more A servers than you have available CPU resources. Likewise if server B contains database intensive services, you'd configure no more B servers than the database can handle.

          As for memory usage, all servers will share code segments. Thus running 10 copies of server A versus 20 copies of server A will likely not consume a huge amount of more memory. Or another way to put it is, say running 1 copy of server A takes 50 MB, it is likely that the next copy of A running will only consume another 10 MB as most of the processes virtual address space will likely be taken up by code, which will be shared across instances. CLOPT settings in general shouldn't affect memory consumption. If you are really concerned about memory consumption, try to build your code into shareable libraries, which then allows that code to be shared by any server needing that code with only a single copy of the code in memory.

          Regarding the domain gateway, it follows the normal Tuxedo routing algorithm assuming load balancing has been turned on. So imported services from a remote domain are treated just like local services. Exported services again have standard request routing applied to any incoming requests, i.e., the incoming load will be optimized across the servers (actually request queues) that offer the service.

          I'm not sure the above answered your questions. If not, please describe a bit more in detail what you are concerned about.

          Todd Little
          Oracle Tuxedo Chief Architect
          1 person found this helpful
          • 2. Re: Segregating services per server (performance considerations)
            Thanks for the reply.....

            We don't have CPU problems today. The majority of our services (these so called "situations" in the future) will be mainly reading from the database layer, hitting IO when not cached in Veritas Quick IO. So "guess" is that the main concern is dividing into fast/slow services.

            Take from your answer that the only hinder for the domain are the request queues. From here I am speculating (hard, since I mainly do performance testing on Java and SAP environments). Do you mean the gateway has a set of request queues that handover work to a "free" instance? What determines the number of these queues? Is there any way to monitor "overheating" of queues other than pulling info from SNMP? Would be cool to view the throughput of domain queues.

            thanks / Matthew
            • 3. Re: Segregating services per server (performance considerations)
              Todd Little-Oracle
              Hi Matthew,

              Every Tuxedo server, including our domain gateways have a request queue (System V IPC queue) associated with them. This queue may be specific to a single server instance (SSSQ - Single Server Single Queue) or associated with multiple server (MSSQ - Multiple Server Single Queue). In the case of the Tuxedo domain gateways, they operate in an SSSQ configuration, so when requests are routed, they are placed on a request queue that is specific to the domain gateway instance.

              With regards to monitoring, Oracle offers a product called TSAM (Tuxedo System and Application Monitoring) that allows a customer to view in real-time the number of messages queued and number of bytes pending to be sent to a remote domain. This should allow you to see whether the gateways are becoming bottlenecks. If a domain gateway is becoming a bottleneck, multiple domain gateways can be configured to increase the throughput to a remote domain.

              I hope this answers your questions. If not, please elaborate on what else you need to know.

              Todd Little
              Oracle Tuxedo Chief Architect
              • 4. Re: Segregating services per server (performance considerations)

                So the domain gateway has it's SSSQ. And the gateway has an access point (unique) and a group with a TMSCOUNT (default of 3). What we normally do (in production) is have 2 sometimes 3 domain gateways where the every subscribing domain gets local access point to both gateways. Thereafter a percentage of services are exposed on one of the gateways. The segregation in our environment is not likely do to historical performance problems with the gateway rather the ability to shut off communication to sections of the domain or for administrative concerns.

                But if the traffic for customer services (these 11 situations) was so high could it be possible to overheat the TMS or the SSSQ of the gateway? Wondering basically if you have seen that problem with other Tuxedo users?

                Really appreciate the time you have taken. mvh / matthew
                • 5. Re: Segregating services per server (performance considerations)
                  Honghsi Lo-Oracle
                  GWTDOMAIN gateway itself is a TMS, so it handles all the TMS related traffics. You don't need to configure TMS for it. GWTDOMAIN gateway depends on whether DMTLOGDEV is configured or not, if it configured it will create transaction log if necessary and handles all the TMS.