9 Replies Latest reply: May 14, 2013 2:26 AM by 1008752 RSS

    Tuxedo Migration - Memory issues

    770343
      Dear All,

      We recently migrated our application as below

      Old Environment:
           32 bit Tuxedo 8.1 on HPUX, Patch Level 371
      New Environment:
           64 bit Tuxedo 11.1.1.3.0 on Linux, Patch Level 006

      After this migration our C++ Corba servers are leaking memory at a high rate. Please see below the Valgrind report for one Corba server. I have taken only the "definitely lost" cases which were reported by Valgrind. Please suggest some solution to resolve these errors from Valgrind



      ==16216== Memcheck, a memory error detector
      ==16216== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
      ==16216== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
      ==16216== Command: User -C dom=BET1 -g 100 -i 80 -u ebkg-t -U /app/ebkg/log/ulogs/ebkgt1_ULOG -m 0 -A -P -e ./../log/trc/UserServer.trc -o ../../log/trc/SYS_ERROR_LOG
      ==16216==
      --16216-- WARNING: Serious error when reading debug info
      --16216-- When reading debug info from /app/ebkg/BE/lib/libicudata.so.44:
      --16216-- Can't make sense of .data section mapping
      ==16216== 32 bytes in 1 blocks are definitely lost in loss record 256 of 563
      ==16216== at 0x4A0679C: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:329)
      ==16216== by 0x66ED951: DListPtr::GetNode() (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66ED9DA: DListPtr::InsertBefore(void*, DListPtrNode*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66EDABA: DListPtr::InsertTail(void*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x670B09B: ORBImpl::RegisterThreadExit(ThreadExitInterest*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x6385FBD: Current_Impl::Current_Impl(ORBImpl*) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x639570E: Tobj_BootstrapImpl::Tobj_BootstrapImpl(CORBA::ORB*, char const*) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x63958AD: BEA_create_Tobj_Bootstrap (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x5EACB68: Tobj_Bootstrap::Tobj_Bootstrap(CORBA::ORB*, char const*) (in /app/mw/tuxedo11gR1.06/lib/libenv.so)
      ==16216== by 0x639232D: OrbMain::startup(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x6392E72: server_init(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x797989: tpsvrinit (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 119 (96 direct, 23 indirect) bytes in 1 blocks are definitely lost in loss record 331 of 563
      ==16216== at 0x4A0679C: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:329)
      ==16216== by 0x66D05CB: CORBA::ORB_init(int&, char**, char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x639223A: OrbMain::startup(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x6392E72: server_init(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x797989: tpsvrinit (in /app/ebkg/BE/bin/User)
      ==16216== by 0x7452D99: _tmmain (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AF35: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 161 (160 direct, 1 indirect) bytes in 1 blocks are definitely lost in loss record 352 of 563
      ==16216== at 0x4A0568B: calloc (vg_replace_malloc.c:593)
      ==16216== by 0x74BE9C3: atnlcl_gss_acquire_cred (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0xA83AB8A: esec_gss_acquire_cred_ex (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x74250E1: tmself_authenticate_ex (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x74B71EF: tmenrollpki (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x744E620: _tmenrollsvr (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7452265: _tmstdinit (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x74528B1: _tmmain (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AF35: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 280 (152 direct, 128 indirect) bytes in 1 blocks are definitely lost in loss record 417 of 563
      ==16216== at 0x4A0679C: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:329)
      ==16216== by 0x66EC507: IOPInstrumentHolder::IOPInstrumentHolder() (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66E8767: IOPRoot::GetIOPInstrumentHolder() (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66E88C8: IOPRoot::GetIOPTraceStream() (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x6713DE3: ORBImpl::ORBImpl(unsigned int, int, char**, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66D0396: CORBA::ORB_init(int&, char**, char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x639223A: OrbMain::startup(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x6392E72: server_init(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x797989: tpsvrinit (in /app/ebkg/BE/bin/User)
      ==16216== by 0x7452D99: _tmmain (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AF35: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 1,836 (160 direct, 1,676 indirect) bytes in 1 blocks are definitely lost in loss record 489 of 563
      ==16216== at 0x4A0679C: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:329)
      ==16216== by 0x6ACF318: POAImpl::GetRootPOA(ORBImpl*, OAManager**, ErrorInfo&, char) (in /app/mw/tuxedo11gR1.06/lib/liborbpoa.so)
      ==16216== by 0x6AC2CBD: OBB__CreateRootPOA (in /app/mw/tuxedo11gR1.06/lib/liborbpoa.so)
      ==16216== by 0x670CA5D: ORBImpl::GetRootPOA(ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x670DBF7: ORBImpl::resolve_initial_references(char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x639245F: OrbMain::startup(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x6392E72: server_init(int, char**) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x797989: tpsvrinit (in /app/ebkg/BE/bin/User)
      ==16216== by 0x7452D99: _tmmain (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AF35: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 3,876 (412 direct, 3,464 indirect) bytes in 1 blocks are definitely lost in loss record 502 of 563
      ==16216== at 0x4A0568B: calloc (vg_replace_malloc.c:593)
      ==16216== by 0xA7F2149: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x746E59A: tmfmsgaddtcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7402935: tpallocinternal (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x741942A: _tmmsgrcv (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x745F567: _tmrcvrq (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x746453D: _tmrunserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AFFB: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 3,876 (412 direct, 3,464 indirect) bytes in 1 blocks are definitely lost in loss record 503 of 563
      ==16216== at 0x4A0568B: calloc (vg_replace_malloc.c:593)
      ==16216== by 0xA7F2149: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x746E59A: tmfmsgaddtcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7402935: tpallocinternal (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7402BFB: tpalloc (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7468719: tpenqueueinternal (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7468E34: tpenqueue (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4BC2EF: Login_i::login(char const*, double, char const*, char, DataStructures::Customer_out) (User_inew.cpp:1697)
      ==16216== by 0x4C4DE0: POA_User::Login::invoke(CORBA::ServerRequest*) (User_snew.cpp:1014)
      ==16216== by 0x638C20E: ObjectStateManager::invoke(TPContext*) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x63916AE: ObjectManager::invoke(CORBA::ServerRequest*) (in /app/mw/tuxedo11gR1.06/lib/libnative.so)
      ==16216== by 0x6ACB2B4: POAImpl::ProcessRequest(ServerRequestImpl*, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborbpoa.so)
      ==16216==
      ==16216== 34,594 (34,204 direct, 390 indirect) bytes in 1 blocks are definitely lost in loss record 549 of 563
      ==16216== at 0x4A06AF0: realloc (vg_replace_malloc.c:662)
      ==16216== by 0xA7F20F8: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x746D427: tmfmsgresizeutcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x74023F0: tprealloc (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0xCC2A028: TGIOPBuffer::Extend(unsigned int, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborbtgiop.so)
      ==16216== by 0x66BFD98: Coder::CheckBufferNoneLeft(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C180B: Encoder::AlignCheckChunk(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C1907: Encoder::EncodeOctetArray(unsigned int, unsigned char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C2F74: Encoder::EncodeString(char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66DAA41: OBB::MarshalBuf::operator<<(char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x43414F: operator<<(OBB::MarshalBuf&, DataStructures::StringList&) (DataStructures_cnew.cpp:161412)
      ==16216== by 0x45F3BC: operator<<(OBB::MarshalBuf&, DataStructures::User&) (DataStructures_cnew.cpp:171370)
      ==16216==
      ==16216== 94,384 (12,352 direct, 82,032 indirect) bytes in 44 blocks are definitely lost in loss record 553 of 563
      ==16216== at 0x4A0568B: calloc (vg_replace_malloc.c:593)
      ==16216== by 0xA7F2149: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x747BFBB: tmfmsgpostrecv (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x750C6A4: _tmmbrecvm2 (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x750BD7E: _tmmbrecvm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7419622: _tmmsgrcv (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x745F567: _tmrcvrq (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x746453D: _tmrunserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AFFB: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 201,620 (25,132 direct, 176,488 indirect) bytes in 61 blocks are definitely lost in loss record 558 of 563
      ==16216== at 0x4A0568B: calloc (vg_replace_malloc.c:593)
      ==16216== by 0xA7F2149: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x746E59A: tmfmsgaddtcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x7402935: tpallocinternal (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x745EA4A: _tmrcvrq (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x746453D: _tmrunserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x743AFFB: _tmstartserver (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x4098BE: main (in /app/ebkg/BE/bin/User)
      ==16216==
      ==16216== 11,118,988 (10,727,184 direct, 391,804 indirect) bytes in 313 blocks are definitely lost in loss record 563 of 563
      ==16216== at 0x4A06AF0: realloc (vg_replace_malloc.c:662)
      ==16216== by 0xA7F20F8: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
      ==16216== by 0x746D427: tmfmsgresizeutcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0x74023F0: tprealloc (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
      ==16216== by 0xCC2A028: TGIOPBuffer::Extend(unsigned int, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborbtgiop.so)
      ==16216== by 0x66BFD98: Coder::CheckBufferNoneLeft(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C0BF6: Coder::CheckBuffer(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C01EF: Coder::AlignAdvance(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C0C73: Coder::AlignInternal(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C0C94: Coder::Align(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C17EC: Encoder::AlignCheckChunk(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216== by 0x66C0E0D: Encoder::EncodeLong(int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
      ==16216==
      ==16216== LEAK SUMMARY:
      ==16216== definitely lost: 10,800,296 bytes in 426 blocks
      ==16216== indirectly lost: 659,470 bytes in 1,074 blocks
      ==16216== possibly lost: 906,419 bytes in 110 blocks
      ==16216== still reachable: 1,267,586 bytes in 689 blocks
      ==16216== suppressed: 0 bytes in 0 blocks
      ==16216==
      ==16216== For counts of detected and suppressed errors, rerun with: -v
      ==16216== ERROR SUMMARY: 3135 errors from 338 contexts (suppressed: 14 from 9)

      Best Regards
      Sunil
        • 1. Re: Tuxedo Migration - Memory issues
          Todd Little-Oracle
          Hi Sunil,

          Memory checking tools such as valgrind are likely to report a lot of lost memory as there are numerous places in Tuxedo where memory is not freed. One could argue that we should always free all memory allocated before exiting, but in many cases the memory will be allocated once, used during the life of the process, and then reclaimed by the operating system when the process exits. The only ones of interest are those that occur repeatedly. In the list you sent, only the large count entry (313 blocks totaling ~11MB) would seem to be of interest. Have you run the application for 10 times as long or 10 times as many requests handled and see what you get? Also, you'll need more stack depth printed to see what's really going on there in the Encoder/Decoder. Are you by any chance using value types and if so, are the clients Tuxedo clients or Java clients?

          Regards,
          Todd Little
          Oracle Tuxedo Chief Architect
          • 2. Re: Tuxedo Migration - Memory issues
            770343
            Hello Todd,

            Thanks for your reply.
            The client in our case is a Java Client and we are not using valuetype. It is a simple CORBA Java client and CORBA C++ server.
            We came across this issue during our Load tests where we saw that the complete machine memory was getting utilized within 10 hours of load testing. What is surprising is that there was no code addition and we did only a technical migration to higher version of tuxedo and Oracle DB and migrated the C++ code from HPUX to RHEL. As per your suggestion I run the application for 10 times but there was no increase in the leaks reported by Valgrind.

            Best Regards
            Sunil
            • 3. Re: Tuxedo Migration - Memory issues
              Todd Little-Oracle
              Hi Sunil.

              So I'm not sure what the concern is. You moved from a 32 bit environment to a 64 bit environment so memory usage will be greater. As well you indicate that the processes aren't continuing to grow, or at least lose memory according to Valgrind. If you have processes that continue to grow, can you run Valgrind on them and see where the repeatable memory loss may be occurring?

              Regards,
              Todd Little
              Oracle Tuxedo Chief Architect
              • 4. Re: Tuxedo Migration - Memory issues
                770343
                Hi Todd,

                My last post was not correct.
                The size of the leaks reported is increasing after running the process for multiple times (~30 times)

                ==2707== LEAK SUMMARY:
                ==2707== definitely lost: 56,466,057 bytes in 2,078 blocks
                ==2707== indirectly lost: 2,731,420 bytes in 4,570 blocks
                ==2707== possibly lost: 1,366,271 bytes in 163 blocks
                ==2707== still reachable: 1,267,586 bytes in 689 blocks
                ==2707== suppressed: 0 bytes in 0 blocks

                The most leak is reported in the liborb.so.
                Are we passing an int to some library where it is expecting long ? Or does the EncodeLong method have some bug ?


                ==2707== 58,451,496 (56,377,440 direct, 2,074,056 indirect) bytes in 1,645 blocks are definitely lost in loss record 561 of 561
                ==2707== at 0x4A06AF0: realloc (vg_replace_malloc.c:662)
                ==2707== by 0xA7F20F8: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
                ==2707== by 0x746D427: tmfmsgresizeutcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
                ==2707== by 0x74023F0: tprealloc (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
                ==2707== by 0xCC2A028: TGIOPBuffer::Extend(unsigned int, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborbtgiop.so)
                ==2707== by 0x66BFD98: Coder::CheckBufferNoneLeft(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C0BF6: Coder::CheckBuffer(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C01EF: Coder::AlignAdvance(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C0C73: Coder::AlignInternal(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C0C94: Coder::Align(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C17EC: Encoder::AlignCheckChunk(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                ==2707== by 0x66C0E0D: Encoder::EncodeLong(int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)

                Best Regards
                Sunil
                • 5. Re: Tuxedo Migration - Memory issues
                  Todd Little-Oracle
                  Hi Sunil,

                  The problem with leak detection is that it tells you where the memory was allocated, but it can't tell you where it was SUPPOSED to be deallocated. So without seeing the full stack and the related application code (which I'm guessing is some CORBA method), it's going to be very difficult to see where the problem lies. If you can provide that information here, send it to me, or file a support case, those would be your best option.

                  Regards,
                  Todd Little
                  Oracle Tuxedo Chief Architect

                  PS I doubt EncodeLong has a bug as it is likely one of the most heavily used methods in CORBA method calls.
                  • 6. Re: Tuxedo Migration - Memory issues
                    770343
                    Hi Todd,

                    Please see the below reported leak in the same server. The numbers are low since the server was run only for a brief period

                    ==12708== 755,412 (719,712 direct, 35,700 indirect) bytes in 21 blocks are definitely lost in loss record 570 of 570
                    ==12708== at 0x4A06AF0: realloc (vg_replace_malloc.c:662)
                    ==12708== by 0xA7F20F8: emem_balloc (in /app/mw/tuxedo11gR1.06/lib/libengine.so)
                    ==12708== by 0x746D427: tmfmsgresizeutcm (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
                    ==12708== by 0x74023F0: tprealloc (in /app/mw/tuxedo11gR1.06/lib/libtux.so)
                    ==12708== by 0xCAC3028: TGIOPBuffer::Extend(unsigned int, ErrorInfo&) (in /app/mw/tuxedo11gR1.06/lib/liborbtgiop.so)
                    ==12708== by 0x66BFD98: Coder::CheckBufferNoneLeft(unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                    ==12708== by 0x66C180B: Encoder::AlignCheckChunk(unsigned int, unsigned int) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                    ==12708== by 0x66C1907: Encoder::EncodeOctetArray(unsigned int, unsigned char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                    ==12708== by 0x66C2F74: Encoder::EncodeString(char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                    ==12708== by 0x66DAA41: OBB::MarshalBuf::operator<<(char const*) (in /app/mw/tuxedo11gR1.06/lib/liborb.so)
                    ==12708== by 0x43414F: operator<<(OBB::MarshalBuf&, DataStructures::StringList&) (DataStructures_cnew.cpp:161412)
                    ==12708== by 0x45F3BC: operator<<(OBB::MarshalBuf&, DataStructures::User&) (DataStructures_cnew.cpp:171370)
                    • 7. Re: Tuxedo Migration - Memory issues
                      Todd Little-Oracle
                      Hi Sunil,

                      Unfortunately that doesn't really help as I don't have the source code to DataStructures_cnew.cpp. What is really needed is the IDL associated with the methods that leak and the source code to those methods. Or a simple reproducer that doesn't expose any of your intellectual property.

                      Regards,
                      Todd Little
                      Oracle Tuxedo Chief Architect
                      • 8. Re: Tuxedo Migration - Memory issues
                        770343
                        Hi Todd,

                        We have raised this issue as a Oracle service request (#3-7114643981).
                        More information has been provided by us in the service request including sample code.
                        It would be great if you have a look.

                        Best Regards
                        Sunil
                        • 9. Re: Tuxedo Migration - Memory issues
                          1008752
                          Hi Everyone,

                          just adding that we can reproduce this memory loss easily with a sample program that we have written and supplied to Oracle. The memory is definitely not lost in the application code. It is lost somewhere in the stub actions before and after the remote procedure call. Please note, that the memory loss does not happen every time the procedure is called. It happens every minute or so. Also note, the memory loss is massive, between 80K and 140K per hit. The sample code has been appended to the SR mentioned above.

                          Regards

                          Ian