10 Replies Latest reply: May 13, 2013 11:41 AM by 1008625 RSS

    Memory leak issue in tuxedo server

    985321
      Hi,

      I am facing memory issues in our tuxedo application.One of the server memory utilization is gradually increasing after reboot. As a temporary solution , we are making restart of that particular server every week.

      I have installed Valgrind tool.i am trying to call the service ( Around 800 services are part of application ) which i suspect causing memory leak through ud script. like below.

      valgrind ud32 < servicename.ud

      but it is not giving memory leak details in actual service instead it is giving details about ud.

      Could you guys suggest how can we find memory leak in tuxedo services.


      Thanks
      Ramana
        • 1. Re: Memory leak issue in tuxedo server
          user9130018
          hello

          run valgrind on tuxedo server binary like this:

          tmshutdown -s <tux-server-binary>
          cd $APPDIR
          valgrind [...valgrind options...] ./<tux-server-binary> -C dom=domain -g <number> -i <number> ....

          grab actual server parametes from "ps -ef" command

          run your service for example 100 times, shutdown it gracefully with tmshutdown -s <tux-server-binary> and look in valgrind log file

          example:

          cd $APPDIR
          valgrind leak-check=full show-reachable=yes undef-value-errors=yes malloc-fill=0xfe free-fill=0xca alignment=256 trace-children=yes track-fds=yes time-stamp=yes log-file=logfile.out ./tuxserver -C dom=tuxdom -g 1100 -i 2200 -u bloo -U /some/path/log/ULOG -m 0 -A -r -- -other args
          • 2. Re: Memory leak issue in tuxedo server
            985321
            HI Sir,

            Thanks a lot for your response.

            I ran the comand as mentioned by you in your post. but i haven't shutdown the server bcoz it is prod environment.
            It logged huge output to the log file.

            Below is the summary of the logfile.

            ==00:00:00:05.383 23666== LEAK SUMMARY:
            ==00:00:00:05.383 23666== definitely lost: 7,105 bytes in 7 blocks
            ==00:00:00:05.383 23666== indirectly lost: 3,383 bytes in 18 blocks
            ==00:00:00:05.383 23666== possibly lost: 28,653 bytes in 9 blocks
            ==00:00:00:05.383 23666== still reachable: 529,286 bytes in 288 blocks
            ==00:00:00:05.383 23666== suppressed: 0 bytes in 0 blocks
            ==00:00:00:05.383 23666==
            ==00:00:00:05.383 23666== For counts of detected and suppressed errors, rerun with: -v
            ==00:00:00:05.383 23666== Use --track-origins=yes to see where uninitialised values come from
            ==00:00:00:05.383 23666== ERROR SUMMARY: 107 errors from 72 contexts (suppressed: 10 from 10)


            But I am not able to understand the output logfile. Could you please let me know how can we analyze the logfile.

            Thanks in advance.
            • 3. Re: Memory leak issue in tuxedo server
              MBriz
              Hi R828,

              you can read http://valgrind.org/docs/manual/mc-manual.html#mc-manual.leaks

              Regards,
              • 4. Re: Memory leak issue in tuxedo server
                MBriz
                Hi all,

                I have a problem when i Execute in my server:

                "valgrind --leak-check=full --show-reachable=yes --undef-value-errors=yes --malloc-fill=0xfe --free-fill=0xca --alignment=256 --trace-children=yes --track-fds=yes --time-stamp=yes --log-file=logfile.out ./tuxserver -C dom=tuxdom -g 1100 -i 2200 -u bloo -U /some/path/log/ULOG -m 0 -A -r -- -other args"

                The server not up, in the ulog i have these error:

                225830.XXXXX!?proc.14473.133477952.0: LIBTUX_CAT:1253: ERROR: shmat failure: errno = 22, key = 78391
                225830.XXXXX!?proc.14473.133477952.0: LIBTUX_CAT:340: ERROR: Wrong configuration file
                225830.XXXXX!?proc.14473.133477952.0: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : shmat: Success

                I want to start the server with valgrind to launch Load Test, but the server dont started. any suggestion?

                Thank in advanced

                Edited by: MBriz on 28-ene-2013 14:21
                • 5. Re: Memory leak issue in tuxedo server
                  user9130018
                  Hello R828

                  this is what you are looking for:

                  [...]
                  ==00:00:02:58.740 23950== 10,800 bytes in 216 blocks are definitely lost in loss record 242 of 251
                  ==00:00:02:58.740 23950== at 0x4027A68: *** <---- your source files
                  ==00:00:02:58.740 23950== by 0x816985B: *** <---- your source files
                  ==00:00:02:58.740 23950== by 0x80E47EF: *** <---- your source files
                  ==00:00:02:58.740 23950== by 0x80D9C6D: *** <---- your source files
                  ==00:00:02:58.740 23950== by 0x436BAB7: _tmsvcdsp (in /opt/bea/tuxedo11gR1/lib/libtux.so)
                  ==00:00:02:58.740 23950== by 0x43969E5: _tmrunserver (in /opt/bea/tuxedo11gR1/lib/libtux.so)
                  ==00:00:02:58.740 23950== by 0x436943C: _tmstartserver (in /opt/bea/tuxedo11gR1/lib/libtux.so)
                  ==00:00:02:58.740 23950== by 0x80723AF: main (BS-5d76.c:160)
                  [...]
                  ==00:00:02:58.741 23950== LEAK SUMMARY:
                  ==00:00:02:58.741 23950== definitely lost: 16,096 bytes in 221 blocks <--------- leak

                  so I lost 221 blocks of memory and backtrace shown above was responsible for 216 of them and source of a problem
                  • 6. Re: Memory leak issue in tuxedo server
                    user9130018
                    Hello MBriz

                    first try to run it manually from APPDIR (without tmboot, just ./server-name <proper-args>). if you succeed you will be able to run it with valgrind
                    • 7. Re: Memory leak issue in tuxedo server
                      MBriz
                      Hi all,

                      If i run in $APPDIR ./server-name <proper-args> the server start, but if i run" valgrind --tool=memcheck --leak-check=full --log-file=logfile.out./server-name <proper-args>) the server don´t start. I see in the logfile.out

                      ==15036== Warning: client syscall shmat tried to modify addresses 0xc0af000-0xd0aefff

                      Regardss,
                      • 8. Re: Memory leak issue in tuxedo server
                        MBriz
                        any idea?
                        • 9. Re: Memory leak issue in tuxedo server
                          user9130018
                          Hello

                          what is your platform and versions?

                          could you paste more from terminal (whole sequence: tmshutdown -s <server>/tmboot -i <instance>/tmshutdown -i <instance>/valgrind run)

                          also paste ulog from valgrind test (hiding sensitive data if you have to)
                          • 10. Re: Memory leak issue in tuxedo server
                            1008625
                            Hi R828,

                            We are also facing memory leak issues when upgrading our OS from HPUX 32 Bit to Red Hat 6.3 (64Bit) with Tuxedo 11.

                            Could you please confirm your configuration. I didn't see it in the thread.

                            Did you resolve your issues ? - if so could you please advise resolution in case it helps us / others.

                            Regards & Thanks