1 Reply Latest reply: Feb 1, 2012 4:09 AM by Thomas Falkenberg RSS

    Memory Leak (WSEndpointImpl / weblogic.wsee.jaxws.ServerLateInitTube)

    Thomas Falkenberg
      Hi all,

      I already posted this in the JRockit Forum (Re: Memory Leak WSEndpointImpl maybe you guys can help out.

      Runnin WLS 10.3.5. on JRockit R28.1.3-11-141760-1.6.0_24-20110301-1432-linux-x86_64) we experience a memory leak in production. I got to obtain an hprof dump, which I analyzed with eclipse MAT. The MAT result is pretty clear but I'm not sure what the cause is. There's one object of com.sun.xml.ws.server.WSEndpointImpl with a growing retained heap, at the moment of the memory dump it was 929 MB. Here's the important part of the dominator tree:

      Class Name | Shallow Heap | Retained Heap | Percentage
      com.sun.xml.ws.server.WSEndpointImpl @ 0xafaf0f70 | 176 | 929.780.784 | 69,61%
      |- com.sun.xml.ws.binding.SOAPBindingImpl @ 0xafaf1160 | 72 | 929.046.360 | 69,56%
      | |- com.sun.xml.ws.binding.WebServiceFeatureList @ 0xafaf13e0 | 56 | 929.044.320 | 69,56%
      | | '- java.util.HashMap @ 0xafaf1e58 | 80 | 929.044.264 | 69,56%
      | | |- java.util.HashMap$Entry[16] @ 0xafaf3058 | 152 | 929.044.160 | 69,56%
      | | | |- java.util.HashMap$Entry @ 0xaf9b8f48 | 48 | 928.998.912 | 69,55%
      | | | | '- weblogic.wsee.jaxws.EndpointCreationInterceptorFeature @ 0xaf9b8f80| 32 | 928.998.864 | 69,55%
      | | | | '- java.util.HashSet @ 0xaf9b92f8 | 24 | 928.998.832 | 69,55%
      | | | | '- java.util.HashMap @ 0xaf9b9528 | 80 | 928.998.808 | 69,55%
      | | | | |- java.util.HashMap$Entry[32768] @ 0x9462cb88 | 262.168 | 928.998.704 | 69,55%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x9b672ae8 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xad2a9360 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa3949490 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa3a332c8 | 40 | 152.104 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x954ef698 | 40 | 151.440 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb3acff18 | 40 | 148.256 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa1db8a38 | 40 | 148.000 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb9d18ee8 | 40 | 148.000 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbe6bf4d0 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa0397080 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbf6693f0 | 40 | 147.896 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xc3c4ddb0 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x82714140 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xa0347178 | 40 | 143.792 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x919c9948 | 40 | 143.688 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xbb1f6d28 | 40 | 135.960 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x8c3cd118 | 40 | 135.488 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x91d6e950 | 40 | 134.936 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x8abefe80 | 40 | 134.936 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x89f0aaa8 | 40 | 134.736 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xb0f116a8 | 40 | 134.600 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x9da69938 | 40 | 134.392 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x96cae018 | 40 | 134.392 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0x88691950 | 40 | 134.040 | 0,01%
      | | | | | |- weblogic.wsee.jaxws.ServerLateInitTube @ 0xc201f268 | 40 | 134.040 | 0,01%
                          '- Total: 25 of 36.803 entries           


      A FlightRecording did not help, the allocation pressure is not high enough. I eventually fired up memleak and found the following allocation trace, which doesn't help me a lot.

      Trace Type Type ID Class Loader Class Loader ID Percent(%) Count
      weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner) weblogic.wsee.jaxws.ServerLateInitTube 98 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner) weblogic.wsee.jaxws.ServerLateInitTube 98 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube) com.sun.xml.ws.api.pipe.PipeClonerImpl 15773 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.ServerLateInitTube.<init>(ServerLateInitTube, TubeCloner) weblogic.wsee.jaxws.ServerLateInitTube 98 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner) weblogic.wsee.jaxws.ServerLateInitTube 98 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.ServerLateInitTube.copy(TubeCloner) weblogic.wsee.jaxws.ServerLateInitTube 98 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube) com.sun.xml.ws.api.pipe.PipeClonerImpl 15773 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.pipe.helper.AbstractFilterTubeImpl.<init>(AbstractFilterTubeImpl, TubeCloner) com.sun.xml.ws.api.pipe.helper.AbstractFilterTubeImpl 15774 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.<init>(WseeServerTube, TubeCloner) weblogic.wsee.jaxws.tubeline.standard.WseeServerTube 2450 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.copy(TubeCloner) weblogic.wsee.jaxws.tubeline.standard.WseeServerTube 2450 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.wsee.jaxws.tubeline.standard.WseeServerTube.copy(TubeCloner) weblogic.wsee.jaxws.tubeline.standard.WseeServerTube 2450 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.pipe.PipeClonerImpl.copy(Tube) com.sun.xml.ws.api.pipe.PipeClonerImpl 15773 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.pipe.TubeCloner.clone(Tube) com.sun.xml.ws.api.pipe.TubeCloner 15776 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.server.WSEndpointImpl$2.<init>(WSEndpointImpl) com.sun.xml.ws.server.WSEndpointImpl$2 15319 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.server.WSEndpointImpl.createPipeHead() com.sun.xml.ws.server.WSEndpointImpl 2523 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.server.Adapter$Toolkit.<init>(Adapter) com.sun.xml.ws.api.server.Adapter$Toolkit 15770 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.<init>(HttpAdapter) com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit 15306 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.HttpAdapter.createToolkit() com.sun.xml.ws.transport.http.HttpAdapter 15467 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.HttpAdapter.createToolkit() com.sun.xml.ws.transport.http.HttpAdapter 15467 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.server.Adapter$1.create() com.sun.xml.ws.api.server.Adapter$1 2807 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.api.server.Adapter$1.create() com.sun.xml.ws.api.server.Adapter$1 2807 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.util.Pool.take() com.sun.xml.ws.util.Pool 15771 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.HttpAdapter.handle(WSHTTPConnection) com.sun.xml.ws.transport.http.HttpAdapter 15467 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletContext, HttpServletRequest, HttpServletResponse) com.sun.xml.ws.transport.http.servlet.ServletAdapter 3251 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(HttpServletRequest, HttpServletResponse, ServletContext) com.sun.xml.ws.transport.http.servlet.WSServletDelegate 4388 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(HttpServletRequest, HttpServletResponse, ServletContext) com.sun.xml.ws.transport.http.servlet.WSServletDelegate 4388 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(HttpServletRequest, HttpServletResponse) com.sun.xml.ws.transport.http.servlet.WSServlet 4743 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) javax.servlet.http.HttpServlet 15468 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) javax.servlet.http.HttpServlet 15468 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run() weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction 15469 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(ServletRequest, HttpServletRequest, ServletRequestImpl, ServletResponse, HttpServletResponse, Servlet) weblogic.servlet.internal.StubSecurityHelper 1044 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.ServletStubImpl.execute(ServletRequest, ServletResponse, FilterChainImpl) weblogic.servlet.internal.ServletStubImpl 907 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.ServletStubImpl.execute(ServletRequest, ServletResponse) weblogic.servlet.internal.ServletStubImpl 907 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(ServletStubImpl, HttpServletRequest, HttpServletResponse) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction 15310 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run() weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction 15310 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AbstractSubject, PrivilegedAction) weblogic.security.acl.internal.AuthenticatedSubject 2125 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.security.service.SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedAction) weblogic.security.service.SecurityManager 15464 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.WebAppServletContext.securedExecute(HttpServletRequest, HttpServletResponse, boolean) weblogic.servlet.internal.WebAppServletContext 1108 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.WebAppServletContext.execute(ServletRequestImpl, ServletResponseImpl) weblogic.servlet.internal.WebAppServletContext 1108 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.servlet.internal.ServletRequestImpl.run() weblogic.servlet.internal.ServletRequestImpl 67 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.work.ExecuteThread.execute(Runnable) weblogic.work.ExecuteThread 1085 sun.misc.Launcher$AppClassLoader<0> 0 100 17
      weblogic.work.ExecuteThread.run() weblogic.work.ExecuteThread 1085 sun.misc.Launcher$AppClassLoader<0> 0 100 17

      I assume these Instances are being created when an erroneous webservice (caught by the application) call happens. At least it doesn't happen with every webservice call, the total number of allocations are lower than the total number of requests in the timeframe.

      Any help would be appreciated.

      Kind regards,
      Thomas