8 Replies Latest reply: Nov 11, 2013 9:42 AM by WaydeNie RSS

    Jolt Client Library Access to Queue Information?

    Dave Fist



      I'm writing a load balancer healthcheck for a PeopleSoft system. At present I have an http servlet running on Weblogic 10.3.4 which uses jolt client libraries (jolt.jar, joltwls.jar, joltjse.jar) to connect to the PeopleSoft app running on Tuxedo 10.3. I'm able to create a session / transactions etc but am wondering if I can get tuxedo client, server and queue information which may help to base an intelligent load balancing algorithm on. Is this possible or do you need to interrogate a process running local on the Tuxedo server? Ideally I'd be hoping  for information similar to the following which was generated using the PeopleSoft application server tool psadmin...


      > Prog Name  Queue Name  Grp Name  ID RqDone Load Done Current Service
      ---------  ----------  --------  -- ------ --------- ---------------
      BBL        224049  cs9appp+   0491 24550 (  IDLE )
      PSDBGSRV   DBGQ    DBGSRV     0     0 (  IDLE )
      PSAPPSRV   APPQ    APPSRV     6   300 (  IDLE )
      PSWATCHSRV WATCH   WATCH      0     0 (  IDLE )
      PSAPPSRV   APPQ    APPSRV     6   300 (  IDLE )
      PSSUBDSP   SUBDQ_dflt  PUBSUB   300  0     0 (  IDLE )
      PSSUBHND   SUBHQ_dflt  PUBSUB   301  0     0 (  IDLE )
      WSL        00001.00020 BASE      20  0     0 (  IDLE )
      PSBRKDSP   BRKDQ_dflt  PUBSUB   100  0     0 (  IDLE )
      PSSAMSRV   SAMQ    APPSRV   100  0     0 (  IDLE )
      PSBRKHND   BRKHQ_dflt  PUBSUB   101  0     0 (  IDLE )
      PSSAMSRV   SAMQ    APPSRV   101  0     0 (  IDLE )
      JREPSVR    00094.00250 JREPGRP  250  5   250 (  IDLE )
      JSL        00095.00200 JSLGRP   200  0     0 (  IDLE )
      PSPUBDSP   PUBDQ_dflt  PUBSUB   200  0     0 (  IDLE )
      PSPUBHND   PUBHQ_dflt  PUBSUB   201  0     0 (  IDLE )



      > Prog Name  Queue Name  # Serve Wk Queued  # Queued  Ave. LenMachine
      ---------  ------------------- ---------  --------  ---------------



      PSAPPSRV   APPQ          2     -     0     - cs9apppj1+
      PSBRKHND   BRKHQ_dflt    1     -     0     - cs9apppj1+
      PSSUBHND   SUBHQ_dflt    1     -     0     - cs9apppj1+
      PSPUBHND   PUBHQ_dflt    1     -     0     - cs9apppj1+
      BBL        224049        1     -     0     - cs9apppj1+
      PSDBGSRV   DBGQ          1     -     0     - cs9apppj1+
      PSPUBDSP   PUBDQ_dflt    1     -     0     - cs9apppj1+
      WSL        00001.00020   1     -     0     - cs9apppj1+
      PSWATCHSRV WATCH         1     -     0     - cs9apppj1+
      PSBRKDSP   BRKDQ_dflt    1     -     0     - cs9apppj1+
      JSL        00095.00200   1     -     0     - cs9apppj1+
      JREPSVR    00094.00250   1     -     0     - cs9apppj1+
      PSSUBDSP   SUBDQ_dflt    1     -     0     - cs9apppj1+
      PSMONITORSRV   MONITOR       1     -     0     - cs9apppj1+
      PSSAMSRV   SAMQ          2     -     0     - cs9apppj1+


      Apologies if this should be posted elsewhere or is an annoyingly newb question. As you may have guessed. I'm a PeopleSoft guy and most of this Tuxedo stuff is usually obfuscated.


      Best Regards,




      David Fist

      PeopleSoft Administrator

      University of Newcastle

        • 1. Re: Jolt Client Library Access to Queue Information?
          Per Lindström

          Hi David,


          as I don't know too much about Peoplesoft this may sound like a stupid question, but what kind of load balancing are you in need of?


          Generally speaking, Tuxedo is very, very good at "load balancing" in it self (or rather "load dispatching", "balance" or "fairness" are not the interesting things here, "efficiency" and "throughput" are :-)). Most attempts of "helping" Tuxedo in this respect are mostly in vain, in my experience.


          So if you could elaborate a little about your actual problem/need I might be able to help you sort out all the obfuscated stuff :-)


          Best regards,


          • 2. Re: Jolt Client Library Access to Queue Information?
            Todd Little-Oracle

            Hi David,


            Per has a good point.  It would be interesting to understand what problem you are trying to solve.


            In any case, to get the information you listed, you could make use of the Tuxedo MIB.  You create and FML32 buffer with the appropriate fields and then call the .TMIB service.  In general you must authenticate yourself as tpsysadm to make MIB calls.  There are examples in this forum for performing the MIB calls, and you can use ud32 to test out your queries before creating them in Jolt.  You'll also need to add an entry in the Jolt repository for the .TMIB service.



            Todd Little

            Oracle Tuxedo Chief Architect

            • 3. Re: Jolt Client Library Access to Queue Information?
              Dave Fist

              Hi Per and Todd,


              Thanks for your replies. The problem I am trying to solve is to create a health check for an F5 Big IP hardware load balancer which fronts a multi-server PeopleSoft system. The health check is responsible for providing information to the F5 so that it can remove pool members (webservers) whenever the web server and their associated application servers (PeopleSoft/tuxedo) become unavailable. A stretch goal would be an F5 iRule to load balance on tuxedo queue lengths. Basically the F5 will call a URL that returns simple text like


              Webserver on <hostname> is alive

              Appserver on <hostname> is alive

              <x> Tuxedo process of type <proc type> are booted

              <y> requests are queued.

              etc etc


              The obfuscation of which I speak is that Tuxedo basically sits underneath PeopleSoft systems and does its job so well you'd hardly know its there. Writing Tuxedo clients is not something I've ever had to do.


              The healthcheck is quite a common problem for PeopleSoft systems with a hardware load balancer and surprisingly there is nothing canned out there. Unfortunately something actually developed in PeopleSoft can't be used in more complex architectures (ie featuring hardware LBRs with SSL offload) because all calls to the PeopleSoft Internet Architecture need to be via an authentication domain....anything not of the form https://system.auth.dom/HRMS/blah will generate an error whereas the health check needs to address the individual instances themselves - ie http://webserver.priv:8123/f5-health-check).


              I'll look into the MIB/FML32 stuff but any suggestions on where to find examples or best practices for using the Jolt Client library in Java to connect to the TMIB would be greatly appreciated. Thanks again for the feedback.



              • 4. Re: Jolt Client Library Access to Queue Information?
                Bobfinan - Oracle-Oracle

                Hello Dave,

                The following knowledge management (aka KM) doc has an example:

                Tux - How to determine the Number of Queued Messages Using the MIB (Doc ID 764638.1)


                Note that the queued messages referred to are not IPC queues, which you are interested in, but Tuxedo /Q queues(i.e. uses T_QUEUE class MIB fields).

                It is though a good example of how to use MIBs queries.


                Bob Finan

                • 5. Re: Jolt Client Library Access to Queue Information?
                  Dave Fist

                  Many thanks to everyone for the help to date.


                  I seem to be running into Re: Help! .TMIB service not returning local attributes, with the TA_NQUEUE value is not being returned for T_QUEUE or T_SVCGRP because the T_DOMAIN:TA_LDBAL value is set to N which problematic as "# Queued" is the value I'm most interested in. Basically I'm just trying to get the same information that tmadmin's "pq" command generates using .TMIB and FML.


                  Is this possible?


                  > pq

                  Prog Name      Queue Name  # Serve Wk Queued  # Queued  Ave. Len    Machine

                  ---------      ------------------- ---------  --------  --------    -------

                  PSBRKHND       BRKHQ_dflt        1         -         0         - cs9apppj1+

                  PSSUBHND       SUBHQ_dflt        1         -         0         - cs9apppj1+

                  JSL            00095.00200       1         -         0         - cs9apppj1+

                  JREPSVR        00094.00250       1         -         0         - cs9apppj1+

                  PSBRKDSP       BRKDQ_dflt        1         -         0         - cs9apppj1+

                  PSPPMSRV       PPMQ2             1         -         0         - cs9apppj1+

                  PSPUBHND       PUBHQ_dflt        1         -         0         - cs9apppj1+

                  PSWATCHSRV     WATCH             1         -         0         - cs9apppj1+

                  PSSUBDSP       SUBDQ_dflt        1         -         0         - cs9apppj1+

                  PSMONITORSRV   MONITOR           1         -         0         - cs9apppj1+

                  PSSAMSRV       SAMQ              2         -         0         - cs9apppj1+

                  WSL            00001.00020       1         -         0         - cs9apppj1+

                  PSAPPSRV       APPQ              2         -         0         - cs9apppj1+

                  PSPUBDSP       PUBDQ_dflt        1         -         0         - cs9apppj1+

                  PSDBGSRV       DBGQ              1         -         0         - cs9apppj1+

                  BBL            153494            1         -         0         - cs9apppj1+


                  Note, I am getting a value back for TA_WKQUEUED... how does this differ from TA_NQUEUED?




                  • 6. Re: Jolt Client Library Access to Queue Information?
                    Per Lindström

                    Hi Dave,


                    In general one would find that WKQUEUED = NQUEUED*50, unless special tuning has been made. But if special tuning has been made, it would mean that WKQUEUED is a more accurate estimation of the time your request will spend waiting in that queue (i e someone has come to the conclusion that all services are not equal in terms of "work" and would induce longer processing times). In any case, using WQQUEUED should be at least as good (or hopefully better) than NQUEUED, unless someone has done a real bad job estimating the load values for each service :-).


                    If you can run the command

                    tmunloadcf | grep LOAD

                    for the Tuxedo domain you're likely to see a lot of


                    meaning these services use the default (50) ratio between N and "work".


                    Still I have some difficulties in seeing what you are trying to achieve. If all http servers end up in the same Tuxedo application, Tuxedo will still do the load balancing for you. Do you really have several Tuxedo domains servicing the same PS application? Sounds like a very advanced setup... but as I said, I don't know too much about PS :-).


                    Hope this helps,


                    • 7. Re: Jolt Client Library Access to Queue Information?
                      Todd Little-Oracle

                      Hi Dave,


                      My suggestion would be to look at the TSAM Plus product from Oracle.  With this product you can get detailed performance information about your Tuxedo applications.  If the F5 has a mechanism for pushing information to it, you could easily establish SLA goals that when not being met would generate a Tuxedo event.  That Tuxedo event could be used to start a command script on the Tuxedo domain that could perform virtually anything as it is just a shell script.


                      Regarding what Per is suggesting, an alternative solution, although not necessarily blessed by PeopleSoft would be to configure your Tuxedo application as an MP configuration and let Tuxedo handle the load balancing should server queues start to get unbalanced.  I know Oracle Consulting Services can help you with this as they have a lot of experience in working with Tuxedo and PeopleSoft.  If you're interesting in working with Oracle Consulting, just send me an e-mail.  My e-mail address is my first name followed by a period followed by my last name at oracle.com.


                      Also, one quick question, why is the LDBAL value for the domain set to N?  It is rare that LDBAL=N has value for an application.



                      Todd Little

                      Oracle Tuxedo Chief Architect

                      • 8. Re: Jolt Client Library Access to Queue Information?

                        Hi Dave,


                        I think we're trying to do largely similar things... I'm using Netscaler's instead of F5's but I'm also trying to find a suitable health monitor method for JOLT/Tuxedo App server and am a bit surprised at the lack of resources (which always makes me wonder what fundamental thing I'm misunderstanding :-) ). I've got an SR (SR 3-7691710291) in with Oracle, but it's pretty stagnant...


                        Basically, we're using VM's with set Weblogic/Tuxedo pairs, using Weblogic psserver=localhost:9000 only. If either the web server and/or the app server are unhealthy, we want the load balancer to stop sending them connections. For a Weblogic health monitor we're using an http monitor to load /PSIGW/PeopleSoftListeningConnector and look for the string "ACTIVE". What we don't have is a similar suitable check for Tuxedo/JOLT health.

                        Preferably, we'd like to connect to port 9000 and run a command (a sort of ping or echo test) and check the result. Something I've been playing a bit with is a cgi script that outputs the results of 'echo pq | tmadmin -r' with some threshold limits for the various queue lengths. I'm not very far along with it, and not sure it's the right path anyway...