I'm writing a load balancer healthcheck for a PeopleSoft system. At present I have an http servlet running on Weblogic 10.3.4 which uses jolt client libraries (jolt.jar, joltwls.jar, joltjse.jar) to connect to the PeopleSoft app running on Tuxedo 10.3. I'm able to create a session / transactions etc but am wondering if I can get tuxedo client, server and queue information which may help to base an intelligent load balancing algorithm on. Is this possible or do you need to interrogate a process running local on the Tuxedo server? Ideally I'd be hoping for information similar to the following which was generated using the PeopleSoft application server tool psadmin...
|> Prog Name||Queue Name Grp Name||ID RqDone Load Done Current Service|
|---------||---------- --------||-- ------ --------- ---------------|
|BBL||224049||cs9appp+||0||491||24550 ( IDLE )|
|PSMONITORSRV MONITOR||MONITOR||1||0||0 ( IDLE )|
|PSDBGSRV||DBGQ||DBGSRV||1||0||0 ( IDLE )|
|PSAPPSRV||APPQ||APPSRV||1||6||300 ( IDLE )|
|PSWATCHSRV||WATCH||WATCH||1||0||0 ( IDLE )|
|PSAPPSRV||APPQ||APPSRV||2||6||300 ( IDLE )|
|PSSUBDSP||SUBDQ_dflt PUBSUB||300||0||0 ( IDLE )|
|PSSUBHND||SUBHQ_dflt PUBSUB||301||0||0 ( IDLE )|
|WSL||00001.00020 BASE||20||0||0 ( IDLE )|
|PSBRKDSP||BRKDQ_dflt PUBSUB||100||0||0 ( IDLE )|
|PSSAMSRV||SAMQ||APPSRV||100||0||0 ( IDLE )|
|PSBRKHND||BRKHQ_dflt PUBSUB||101||0||0 ( IDLE )|
|PSSAMSRV||SAMQ||APPSRV||101||0||0 ( IDLE )|
|JREPSVR||00094.00250 JREPGRP||250||5||250 ( IDLE )|
|JSL||00095.00200 JSLGRP||200||0||0 ( IDLE )|
|PSPUBDSP||PUBDQ_dflt PUBSUB||200||0||0 ( IDLE )|
|PSPUBHND||PUBHQ_dflt PUBSUB||201||0||0 ( IDLE )|
|> Prog Name||Queue Name # Serve Wk Queued # Queued Ave. Len||Machine|
|---------||------------------- --------- -------- --------||-------|
|PSMONITORSRV MONITOR||1||-||0||- cs9apppj1+|
Apologies if this should be posted elsewhere or is an annoyingly newb question. As you may have guessed. I'm a PeopleSoft guy and most of this Tuxedo stuff is usually obfuscated.
University of Newcastle
as I don't know too much about Peoplesoft this may sound like a stupid question, but what kind of load balancing are you in need of?
Generally speaking, Tuxedo is very, very good at "load balancing" in it self (or rather "load dispatching", "balance" or "fairness" are not the interesting things here, "efficiency" and "throughput" are :-)). Most attempts of "helping" Tuxedo in this respect are mostly in vain, in my experience.
So if you could elaborate a little about your actual problem/need I might be able to help you sort out all the obfuscated stuff :-)
Per has a good point. It would be interesting to understand what problem you are trying to solve.
In any case, to get the information you listed, you could make use of the Tuxedo MIB. You create and FML32 buffer with the appropriate fields and then call the .TMIB service. In general you must authenticate yourself as tpsysadm to make MIB calls. There are examples in this forum for performing the MIB calls, and you can use ud32 to test out your queries before creating them in Jolt. You'll also need to add an entry in the Jolt repository for the .TMIB service.
Oracle Tuxedo Chief Architect
Hi Per and Todd,
Thanks for your replies. The problem I am trying to solve is to create a health check for an F5 Big IP hardware load balancer which fronts a multi-server PeopleSoft system. The health check is responsible for providing information to the F5 so that it can remove pool members (webservers) whenever the web server and their associated application servers (PeopleSoft/tuxedo) become unavailable. A stretch goal would be an F5 iRule to load balance on tuxedo queue lengths. Basically the F5 will call a URL that returns simple text like
Webserver on <hostname> is alive
Appserver on <hostname> is alive
<x> Tuxedo process of type <proc type> are booted
<y> requests are queued.
The obfuscation of which I speak is that Tuxedo basically sits underneath PeopleSoft systems and does its job so well you'd hardly know its there. Writing Tuxedo clients is not something I've ever had to do.
The healthcheck is quite a common problem for PeopleSoft systems with a hardware load balancer and surprisingly there is nothing canned out there. Unfortunately something actually developed in PeopleSoft can't be used in more complex architectures (ie featuring hardware LBRs with SSL offload) because all calls to the PeopleSoft Internet Architecture need to be via an authentication domain....anything not of the form https://system.auth.dom/HRMS/blah will generate an error whereas the health check needs to address the individual instances themselves - ie http://webserver.priv:8123/f5-health-check).
I'll look into the MIB/FML32 stuff but any suggestions on where to find examples or best practices for using the Jolt Client library in Java to connect to the TMIB would be greatly appreciated. Thanks again for the feedback.
The following knowledge management (aka KM) doc has an example:
Tux - How to determine the Number of Queued Messages Using the MIB (Doc ID 764638.1)
Note that the queued messages referred to are not IPC queues, which you are interested in, but Tuxedo /Q queues(i.e. uses T_QUEUE class MIB fields).
It is though a good example of how to use MIBs queries.
Many thanks to everyone for the help to date.
I seem to be running into Re: Help! .TMIB service not returning local attributes, with the TA_NQUEUE value is not being returned for T_QUEUE or T_SVCGRP because the T_DOMAIN:TA_LDBAL value is set to N which problematic as "# Queued" is the value I'm most interested in. Basically I'm just trying to get the same information that tmadmin's "pq" command generates using .TMIB and FML.
Is this possible?
Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine
--------- ------------------- --------- -------- -------- -------
PSBRKHND BRKHQ_dflt 1 - 0 - cs9apppj1+
PSSUBHND SUBHQ_dflt 1 - 0 - cs9apppj1+
JSL 00095.00200 1 - 0 - cs9apppj1+
JREPSVR 00094.00250 1 - 0 - cs9apppj1+
PSBRKDSP BRKDQ_dflt 1 - 0 - cs9apppj1+
PSPPMSRV PPMQ2 1 - 0 - cs9apppj1+
PSPUBHND PUBHQ_dflt 1 - 0 - cs9apppj1+
PSWATCHSRV WATCH 1 - 0 - cs9apppj1+
PSSUBDSP SUBDQ_dflt 1 - 0 - cs9apppj1+
PSMONITORSRV MONITOR 1 - 0 - cs9apppj1+
PSSAMSRV SAMQ 2 - 0 - cs9apppj1+
WSL 00001.00020 1 - 0 - cs9apppj1+
PSAPPSRV APPQ 2 - 0 - cs9apppj1+
PSPUBDSP PUBDQ_dflt 1 - 0 - cs9apppj1+
PSDBGSRV DBGQ 1 - 0 - cs9apppj1+
BBL 153494 1 - 0 - cs9apppj1+
Note, I am getting a value back for TA_WKQUEUED... how does this differ from TA_NQUEUED?
In general one would find that WKQUEUED = NQUEUED*50, unless special tuning has been made. But if special tuning has been made, it would mean that WKQUEUED is a more accurate estimation of the time your request will spend waiting in that queue (i e someone has come to the conclusion that all services are not equal in terms of "work" and would induce longer processing times). In any case, using WQQUEUED should be at least as good (or hopefully better) than NQUEUED, unless someone has done a real bad job estimating the load values for each service :-).
If you can run the command
tmunloadcf | grep LOAD
for the Tuxedo domain you're likely to see a lot of
meaning these services use the default (50) ratio between N and "work".
Still I have some difficulties in seeing what you are trying to achieve. If all http servers end up in the same Tuxedo application, Tuxedo will still do the load balancing for you. Do you really have several Tuxedo domains servicing the same PS application? Sounds like a very advanced setup... but as I said, I don't know too much about PS :-).
Hope this helps,
My suggestion would be to look at the TSAM Plus product from Oracle. With this product you can get detailed performance information about your Tuxedo applications. If the F5 has a mechanism for pushing information to it, you could easily establish SLA goals that when not being met would generate a Tuxedo event. That Tuxedo event could be used to start a command script on the Tuxedo domain that could perform virtually anything as it is just a shell script.
Regarding what Per is suggesting, an alternative solution, although not necessarily blessed by PeopleSoft would be to configure your Tuxedo application as an MP configuration and let Tuxedo handle the load balancing should server queues start to get unbalanced. I know Oracle Consulting Services can help you with this as they have a lot of experience in working with Tuxedo and PeopleSoft. If you're interesting in working with Oracle Consulting, just send me an e-mail. My e-mail address is my first name followed by a period followed by my last name at oracle.com.
Also, one quick question, why is the LDBAL value for the domain set to N? It is rare that LDBAL=N has value for an application.
Oracle Tuxedo Chief Architect
I think we're trying to do largely similar things... I'm using Netscaler's instead of F5's but I'm also trying to find a suitable health monitor method for JOLT/Tuxedo App server and am a bit surprised at the lack of resources (which always makes me wonder what fundamental thing I'm misunderstanding :-) ). I've got an SR (SR 3-7691710291) in with Oracle, but it's pretty stagnant...
Basically, we're using VM's with set Weblogic/Tuxedo pairs, using Weblogic psserver=localhost:9000 only. If either the web server and/or the app server are unhealthy, we want the load balancer to stop sending them connections. For a Weblogic health monitor we're using an http monitor to load /PSIGW/PeopleSoftListeningConnector and look for the string "ACTIVE". What we don't have is a similar suitable check for Tuxedo/JOLT health.
Preferably, we'd like to connect to port 9000 and run a command (a sort of ping or echo test) and check the result. Something I've been playing a bit with is a cgi script that outputs the results of 'echo pq | tmadmin -r' with some threshold limits for the various queue lengths. I'm not very far along with it, and not sure it's the right path anyway...