5 Replies Latest reply: Feb 1, 2012 4:09 AM by Thomas Falkenberg RSS

    Memory Leak WSEndpointImpl

    Thomas Falkenberg
      Hi all,

      using R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64 of jrockit we experience a memory leak in production. So far I only got to obtain an hprof dump, which I analyzed with eclipse MAT. I'm trying to obtain a flight recording as well but I'm not sure if this will shed more light on the problem. The MAT result is pretty clear but I'm not sure what the cause is. There's one object of com.sun.xml.ws.server.WSEndpointImpl with a growing retained heap, at the moment of the memory dump it was 929 MB. Here's the import part of the dominator tree:

      Class Name | Shallow Heap | Retained Heap | Percentage
      ---------------------------------------------------------------------------------------------------------------------------------
      com.sun.xml.ws.server.WSEndpointImpl @ 0xafaf0f70 | 176 | 929.780.784 | 69,61%
      |- com.sun.xml.ws.binding.SOAPBindingImpl @ 0xafaf1160 | 72 | 929.046.360 | 69,56%
      | |- com.sun.xml.ws.binding.WebServiceFeatureList @ 0xafaf13e0 | 56 | 929.044.320 | 69,56%
      | | '- java.util.HashMap @ 0xafaf1e58 | 80 | 929.044.264 | 69,56%
      | | |- java.util.HashMap$Entry[16] @ 0xafaf3058 | 152 | 929.044.160 | 69,56%
      | | | |- java.util.HashMap$Entry @ 0xaf9b8f48 | 48 | 928.998.912 | 69,55%
      | | | | '- weblogic.wsee.jaxws.EndpointCreationInterceptorFeature @ 0xaf9b8f80| 32 | 928.998.864 | 69,55%
      | | | | '- java.util.HashSet @ 0xaf9b92f8 | 24 | 928.998.832 | 69,55%
      | | | | '- java.util.HashMap @ 0xaf9b9528 | 80 | 928.998.808 | 69,55%
      | | | | |- java.util.HashMap$Entry[32768] @ 0x9462cb88 | 262.168 | 928.998.704 | 69,55%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x9b672ae8 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xad2a9360 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa3949490 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa3a332c8 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x954ef698 | 40 | 151.440 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb3acff18 | 40 | 148.256 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa1db8a38 | 40 | 148.000 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb9d18ee8 | 40 | 148.000 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbe6bf4d0 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa0397080 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbf6693f0 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xc3c4ddb0 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x82714140 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa0347178 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x919c9948 | 40 | 143.688 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbb1f6d28 | 40 | 135.960 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x8c3cd118 | 40 | 135.488 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x91d6e950 | 40 | 134.936 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x8abefe80 | 40 | 134.936 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x89f0aaa8 | 40 | 134.736 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb0f116a8 | 40 | 134.600 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x9da69938 | 40 | 134.392 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x96cae018 | 40 | 134.392 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x88691950 | 40 | 134.040 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xc201f268 | 40 | 134.040 | 0,01%
      | | | | | '- Total: 25 of 36.803 entries | | |
      ---------------------------------------------------------------------------------------------------------------------------------

      I would appreciate any hints on how to proceed as these are standard java / weblogic classes. If I drill down into the objects I see some ErrorHandler objects, so these objects might accumulate with erroneus requests, but I'm not positive on this since I only checked a few instances. Before I start to try to reproduce this on a test environment I would like to ask for your help.

      Kind regards,
      Thomas

      PS: Sorry, this is probably not really jrockit related, but I know here are some java experts ;)
        • 1. Re: Memory Leak WSEndpointImpl
          Klara Ward, Java Mission Control Dev-Oracle
          Hi,
          have you tried the Memleak tool, available in the JRockit Mission Control GUI?


          Klara, Mission Control QA
          • 2. Re: Memory Leak WSEndpointImpl
            Thomas Falkenberg
            Thanks, but I don't want to use that in a running production system since the performance overhead is very large.
            • 3. Re: Memory Leak WSEndpointImpl
              Hirt-Oracle
              Hi Thomas,

              It seems your hprof dump already contains the information you need - you know which HashMap is unintentionally retaining the objects causing the leak. I would ask around in the WLS forums to see if someone can assist you in finding the root cause. If you want to reverse engineer what is responsible for allocating the objects being retained in the HashMap, you can fire up memleak and turn on allocation profiling for that particular type (weblogic.wsee.jaxws.ServerLateInitTube). That may however be a bit expensive. A Flight Recording may possibly help you, but only if the allocation of the leaked objects put into the hashmap is causing a considerable fraction of the total allocation pressure in the system.

              Kind regards,
              Marcus
              • 4. Re: Memory Leak WSEndpointImpl
                Thomas Falkenberg
                Hi Marcus,

                thanks. The FlightRecording did not help, the allocation pressure is not high enough. I eventually fired up memleak and found the following allocation trace, which doesn't help me a lot. I will post in WLS forums.

                Trace     Type     Type ID     Class Loader     Class Loader ID     Percent(%)     Count
                weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner)     weblogic.wsee.jaxws.ServerLateInitTube     98     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner)     weblogic.wsee.jaxws.ServerLateInitTube     98     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube)     com.sun.xml.ws.api.pipe.PipeClonerImpl     15773     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.ServerLateInitTube.<init>(ServerLateInitTube, TubeCloner)     weblogic.wsee.jaxws.ServerLateInitTube     98     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner)     weblogic.wsee.jaxws.ServerLateInitTube     98     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner)     weblogic.wsee.jaxws.ServerLateInitTube     98     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube)     com.sun.xml.ws.api.pipe.PipeClonerImpl     15773     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.pipe.helper.AbstractFilterTubeImpl.<init>(AbstractFilterTubeImpl, TubeCloner)     com.sun.xml.ws.api.pipe.helper.AbstractFilterTubeImpl     15774     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.<init>(WseeServerTube, TubeCloner)     weblogic.wsee.jaxws.tubeline.standard.WseeServerTube     2450     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.copy(TubeCloner)     weblogic.wsee.jaxws.tubeline.standard.WseeServerTube     2450     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.copy(TubeCloner)     weblogic.wsee.jaxws.tubeline.standard.WseeServerTube     2450     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube)     com.sun.xml.ws.api.pipe.PipeClonerImpl     15773     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.pipe.TubeCloner.clone(Tube)     com.sun.xml.ws.api.pipe.TubeCloner     15776     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.server.WSEndpointImpl$2.<init>(WSEndpointImpl)     com.sun.xml.ws.server.WSEndpointImpl$2     15319     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.server.WSEndpointImpl.createPipeHead()     com.sun.xml.ws.server.WSEndpointImpl     2523     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.server.Adapter$Toolkit.<init>(Adapter)     com.sun.xml.ws.api.server.Adapter$Toolkit     15770     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.<init>(HttpAdapter)     com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit     15306     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.HttpAdapter.createToolkit()     com.sun.xml.ws.transport.http.HttpAdapter     15467     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.HttpAdapter.createToolkit()     com.sun.xml.ws.transport.http.HttpAdapter     15467     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.server.Adapter$1.create()     com.sun.xml.ws.api.server.Adapter$1     2807     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.api.server.Adapter$1.create()     com.sun.xml.ws.api.server.Adapter$1     2807     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.util.Pool.take()     com.sun.xml.ws.util.Pool     15771     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.HttpAdapter.handle(WSHTTPConnection)     com.sun.xml.ws.transport.http.HttpAdapter     15467     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletContext, HttpServletRequest, HttpServletResponse)     com.sun.xml.ws.transport.http.servlet.ServletAdapter     3251     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(HttpServletRequest, HttpServletResponse, ServletContext)     com.sun.xml.ws.transport.http.servlet.WSServletDelegate     4388     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(HttpServletRequest, HttpServletResponse, ServletContext)     com.sun.xml.ws.transport.http.servlet.WSServletDelegate     4388     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(HttpServletRequest, HttpServletResponse)     com.sun.xml.ws.transport.http.servlet.WSServlet     4743     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)     javax.servlet.http.HttpServlet     15468     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)     javax.servlet.http.HttpServlet     15468     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run()     weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction     15469     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.StubSecurityHelper.invokeServlet(ServletRequest, HttpServletRequest, ServletRequestImpl, ServletResponse, HttpServletResponse, Servlet)     weblogic.servlet.internal.StubSecurityHelper     1044     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.ServletStubImpl.execute(ServletRequest, ServletResponse, FilterChainImpl)     weblogic.servlet.internal.ServletStubImpl     907     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.ServletStubImpl.execute(ServletRequest, ServletResponse)     weblogic.servlet.internal.ServletStubImpl     907     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(ServletStubImpl, HttpServletRequest, HttpServletResponse)     weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction     15310     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run()     weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction     15310     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.security.acl.internal.AuthenticatedSubject.doAs(AbstractSubject, PrivilegedAction)     weblogic.security.acl.internal.AuthenticatedSubject     2125     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.security.service.SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedAction)     weblogic.security.service.SecurityManager     15464     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.WebAppServletContext.securedExecute(HttpServletRequest, HttpServletResponse, boolean)     weblogic.servlet.internal.WebAppServletContext     1108     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.WebAppServletContext.execute(ServletRequestImpl, ServletResponseImpl)     weblogic.servlet.internal.WebAppServletContext     1108     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.servlet.internal.ServletRequestImpl.run()     weblogic.servlet.internal.ServletRequestImpl     67     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.work.ExecuteThread.execute(Runnable)     weblogic.work.ExecuteThread     1085     sun.misc.Launcher$AppClassLoader<0>     0     100     17
                weblogic.work.ExecuteThread.run()     weblogic.work.ExecuteThread     1085     sun.misc.Launcher$AppClassLoader<0>     0     100     17

                Kind regards,
                Thomas
                • 5. Re: Memory Leak WSEndpointImpl
                  Thomas Falkenberg
                  This is a known bug (ID 1303769.1), patch is available.
                  We're running it in production at the moment, but too early to tell if it helps.