8 Replies Latest reply: Apr 2, 2013 8:39 AM by cool.br33ze RSS

    LIBTUX_CAT:541 + Memory Leaks

    cool.br33ze
      Hi

      Tuxedo 11g (11.1.1.2.0) / Windows 2008 R2

      We have a number of third party supplied tuxedo servers which when the BBL kills and restarts them the following is noted in the ULOG.

      BBL.1168.8884.0: LIBTUX_CAT:541: WARN: Server ThirdParty_Grp/630 terminated

      I understand that this warning is given when the code exited a process without calling any Tuxedo exit routines.

      The memory on these same tuxedo servers grow quite large throughout the day and I was wondering if the poor practice of not calling Tuxedo exit routines are the root cause of the memory leaks.

      If so, what evidence/proof/test is there that we can use to ask the third party to correct the code?

      Thanks

      Edited by: cool.br33ze on 20-Mar-2013 04:43
        • 1. Re: LIBTUX_CAT:541 + Memory Leaks
          user734005
          If they can turn TMTRACE on (export TMTRACE=atmi:ulog) you should see whether tpreturn is called.
          • 2. Re: LIBTUX_CAT:541 + Memory Leaks
            Per Lindström
            Hi,

            are you sure that the BBL actually "kills" them, or do they actually crash on their own and BBL just discovers that they're gone (and restarts them)?

            Are there any core files generated? Sometimes the Tuxedo server is unable to create a core file when it crashes due to directory permissions or file size quotas, but if you find a core file and "file core" tells you it was from one of the supplied servers I'd say you have "the smoking gun" right there.

            Hope this helps,
            /Per
            • 3. Re: LIBTUX_CAT:541 + Memory Leaks
              Todd Little-Oracle
              To add to what Per said, if the BBL kills a server (which it normally only does in the case of a server exceeded service timeout value SVCTIMEOUT) then there should be an entry in the ULOG to that effect. I suspect as Per says, these servers likely have severe memory leaks and possibly memory corruption (although that is speculation on my part) problems. Memory growing in a Tuxedo server is often related to not performing tpfree() calls on buffers the server owns. But without sources, the only option is likely TMTRACE as was already mentioned.

              Regards,
              Todd Little
              Oracle Tuxedo Chief Architect
              • 4. Re: LIBTUX_CAT:541 + Memory Leaks
                cool.br33ze
                Thanks for the responses so far.

                As these are third party services I dont have access to the code for me to see myself.

                Yes, BBL is killing the server after a SVCTIMEOUT, the following is in the ULOG.

                BBL.1168.8884.0: CMDTUX_CAT:1836: WARN: Server(7876) processing terminated with SIGKILL after SVCTIMEOUT

                Straight after the LIBTUX_CAT:541 is logged.

                There are multiple offending services which tells me it isn't in one service where the process exited without calling any Tuxedo exit routines... (Are these just tpreturn() and tpexit()? It's been a while since i have written a tux service. )

                ...Or where the memory leaks are.

                Will the TMTRACE show tpalloc() along with (if any) tpfree()?
                • 5. Re: LIBTUX_CAT:541 + Memory Leaks
                  cool.br33ze
                  Also, is it possible to set TMTRACE() to only log tpreturn(), tpexist(), tpfree(), tpalloc()?

                  If so, what is the command?
                  • 6. Re: LIBTUX_CAT:541 + Memory Leaks
                    Per Lindström
                    Hi,

                    you can definitely search for one specific ATMI call, see

                    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rf5/rf5.html#wp1529614

                    for details regarding TMTRACE where there's an example of logging all calls to tpacall().

                    As I'm not The Universal Master Of All RegExps I don't really know whether you can catch several different calls in the search expression.

                    On another note, I'd try looking into the processing time of your services. I'd say it's more probable that you have a performance problem in general rather than a major bug in the services (if they never call tpreturn() not much work will be done at all...).

                    What is the value for SVCTIMEOUT (that is trespassed every now and then)?

                    If you can add a "-r" in CLOPT you'll get statistics from all services executed in that particular server written to the stderr file (that might be specified with the -e option).

                    Using txrpt you can then get insight into the execution times for the services in question. If they are constantly near the SVCTIMEOUT you may need to adjust the SVCTIMEOUT. If they are varying very much you may need to check for locks in the database or other reasons for "spiky" behaviour.

                    See

                    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rf5/rf5.html#wp1003290

                    for more info on what can be done in CLOPT in the ubbconfig (look for -r and -e options), and also

                    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rfcm/rfcmd.html#wp1002357

                    for more info on how to interpret the statistics that -r creates.

                    It is fairly common to create own-written utilities for interpreting the stderr file, it's not really rocket science if you want to get for instance min and max values out of it, which, by the way, would make a nice enhancement to txrpt in the first place. Product Management: are you listening? :-)

                    Hope this helps,
                    /Per
                    • 7. Re: LIBTUX_CAT:541 + Memory Leaks
                      Todd Little-Oracle
                      Hi

                      Well if all of the LIBTUX_CAT:541 messages are preceded by a CMDTUX_CAT:1836, then it would make sense that exit handlers aren't being called because the process is being killed with SIGKILL which as far as I know can't be caught, so all user exit handlers are bypassed. But what does that have to do with leaking memory? The memory leaks are most certainly not related to exit handlers not being called, as that would be the last thing a server does in any case before the process disappears.

                      You would want to trace service routines, tpreturn(), tpalloc(), tpfree(), and tpcall()/tpacall(). There is no tpexit() routine to trace.

                      Regards,
                      Todd Little
                      Oracle Tuxedo Chief Architect
                      • 8. Re: LIBTUX_CAT:541 + Memory Leaks
                        cool.br33ze
                        Hi
                        Thanks guys.
                        You've been a great help.