This discussion is archived
8 Replies Latest reply: Apr 2, 2013 6:39 AM by 871882 RSS

LIBTUX_CAT:541 + Memory Leaks

871882 Newbie
Currently Being Moderated
Hi

Tuxedo 11g (11.1.1.2.0) / Windows 2008 R2

We have a number of third party supplied tuxedo servers which when the BBL kills and restarts them the following is noted in the ULOG.

BBL.1168.8884.0: LIBTUX_CAT:541: WARN: Server ThirdParty_Grp/630 terminated

I understand that this warning is given when the code exited a process without calling any Tuxedo exit routines.

The memory on these same tuxedo servers grow quite large throughout the day and I was wondering if the poor practice of not calling Tuxedo exit routines are the root cause of the memory leaks.

If so, what evidence/proof/test is there that we can use to ask the third party to correct the code?

Thanks

Edited by: cool.br33ze on 20-Mar-2013 04:43
  • 1. Re: LIBTUX_CAT:541 + Memory Leaks
    782098 Newbie
    Currently Being Moderated
    If they can turn TMTRACE on (export TMTRACE=atmi:ulog) you should see whether tpreturn is called.
  • 2. Re: LIBTUX_CAT:541 + Memory Leaks
    Per Lindström Explorer
    Currently Being Moderated
    Hi,

    are you sure that the BBL actually "kills" them, or do they actually crash on their own and BBL just discovers that they're gone (and restarts them)?

    Are there any core files generated? Sometimes the Tuxedo server is unable to create a core file when it crashes due to directory permissions or file size quotas, but if you find a core file and "file core" tells you it was from one of the supplied servers I'd say you have "the smoking gun" right there.

    Hope this helps,
    /Per
  • 3. Re: LIBTUX_CAT:541 + Memory Leaks
    Todd Little Expert
    Currently Being Moderated
    To add to what Per said, if the BBL kills a server (which it normally only does in the case of a server exceeded service timeout value SVCTIMEOUT) then there should be an entry in the ULOG to that effect. I suspect as Per says, these servers likely have severe memory leaks and possibly memory corruption (although that is speculation on my part) problems. Memory growing in a Tuxedo server is often related to not performing tpfree() calls on buffers the server owns. But without sources, the only option is likely TMTRACE as was already mentioned.

    Regards,
    Todd Little
    Oracle Tuxedo Chief Architect
  • 4. Re: LIBTUX_CAT:541 + Memory Leaks
    871882 Newbie
    Currently Being Moderated
    Thanks for the responses so far.

    As these are third party services I dont have access to the code for me to see myself.

    Yes, BBL is killing the server after a SVCTIMEOUT, the following is in the ULOG.

    BBL.1168.8884.0: CMDTUX_CAT:1836: WARN: Server(7876) processing terminated with SIGKILL after SVCTIMEOUT

    Straight after the LIBTUX_CAT:541 is logged.

    There are multiple offending services which tells me it isn't in one service where the process exited without calling any Tuxedo exit routines... (Are these just tpreturn() and tpexit()? It's been a while since i have written a tux service. )

    ...Or where the memory leaks are.

    Will the TMTRACE show tpalloc() along with (if any) tpfree()?
  • 5. Re: LIBTUX_CAT:541 + Memory Leaks
    871882 Newbie
    Currently Being Moderated
    Also, is it possible to set TMTRACE() to only log tpreturn(), tpexist(), tpfree(), tpalloc()?

    If so, what is the command?
  • 6. Re: LIBTUX_CAT:541 + Memory Leaks
    Per Lindström Explorer
    Currently Being Moderated
    Hi,

    you can definitely search for one specific ATMI call, see

    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rf5/rf5.html#wp1529614

    for details regarding TMTRACE where there's an example of logging all calls to tpacall().

    As I'm not The Universal Master Of All RegExps I don't really know whether you can catch several different calls in the search expression.

    On another note, I'd try looking into the processing time of your services. I'd say it's more probable that you have a performance problem in general rather than a major bug in the services (if they never call tpreturn() not much work will be done at all...).

    What is the value for SVCTIMEOUT (that is trespassed every now and then)?

    If you can add a "-r" in CLOPT you'll get statistics from all services executed in that particular server written to the stderr file (that might be specified with the -e option).

    Using txrpt you can then get insight into the execution times for the services in question. If they are constantly near the SVCTIMEOUT you may need to adjust the SVCTIMEOUT. If they are varying very much you may need to check for locks in the database or other reasons for "spiky" behaviour.

    See

    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rf5/rf5.html#wp1003290

    for more info on what can be done in CLOPT in the ubbconfig (look for -r and -e options), and also

    http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rfcm/rfcmd.html#wp1002357

    for more info on how to interpret the statistics that -r creates.

    It is fairly common to create own-written utilities for interpreting the stderr file, it's not really rocket science if you want to get for instance min and max values out of it, which, by the way, would make a nice enhancement to txrpt in the first place. Product Management: are you listening? :-)

    Hope this helps,
    /Per
  • 7. Re: LIBTUX_CAT:541 + Memory Leaks
    Todd Little Expert
    Currently Being Moderated
    Hi

    Well if all of the LIBTUX_CAT:541 messages are preceded by a CMDTUX_CAT:1836, then it would make sense that exit handlers aren't being called because the process is being killed with SIGKILL which as far as I know can't be caught, so all user exit handlers are bypassed. But what does that have to do with leaking memory? The memory leaks are most certainly not related to exit handlers not being called, as that would be the last thing a server does in any case before the process disappears.

    You would want to trace service routines, tpreturn(), tpalloc(), tpfree(), and tpcall()/tpacall(). There is no tpexit() routine to trace.

    Regards,
    Todd Little
    Oracle Tuxedo Chief Architect
  • 8. Re: LIBTUX_CAT:541 + Memory Leaks
    871882 Newbie
    Currently Being Moderated
    Hi
    Thanks guys.
    You've been a great help.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points