1 Reply Latest reply on Apr 16, 2019 7:21 AM by user1039684

    Oracle Coherence more and more slower with time

    3889359

      Hey,everyone,I meet a problem when I use coherence 12.1.3.0.0+p21924701(bug fix package).

      When I restart the coherence server,ervery client put/get data from server very fast(4-6ms,data size 100k),

      but after 10 days later this time increased to 14-16 ms.

      When 2 months later,this time will increase to 140-160ms.

      I have already set the coherence-cache-config-server.xml and use time-limited-backing-map,

      I have no idea why the clients put/get data from cache more and more slower with time,Can anyone help?

       

       

       

       

       

       

      server:8C 32G  * 3  every 1 sever have 1 proxy and 2 servers

       

       

      system:RHEL  6.7

       

       

      client:C++ client  in 15  servers (every sever have 24 clients)

       

       

      JDK:jdk1.7.0_55

       

       

      coherence-cache-config-server.xml:

      <defaults>

      <socket-provider system-property="tangosol.coherence.socketprovider" /-->

      <serializer>

      <instance>

      <class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>

      <init-params>

      <init-param>

      <param-type>String</param-type>

      <param-value>dcache-pof-config.xml</param-value>

      </init-param>

      </init-params>

      </instance>

      </serializer>

      </defaults>

       

       

      <caching-scheme-mapping>

      <cache-mapping>

      <cache-name>DistTokenCache</cache-name>

      <scheme-name>dist_PartitionedPofScheme</scheme-name>

      </cache-mapping>

      </caching-scheme-mapping>

       

       

      <caching-schemes>

      <distributed-scheme>

      <scheme-name>dist_PartitionedPofScheme</scheme-name>

      <service-name>PartitionedPofCache</service-name>

      <thread-count>32</thread-count>

      <partition-count>8191</partition-count>

      <request-timeout>5000ms</request-timeout>

      <backing-map-scheme>

      <local-scheme>

      <scheme-ref>time-limited-backing-map</scheme-ref>

      </local-scheme>

      </backing-map-scheme>

      <autostart>true</autostart>

      </distributed-scheme>

      <local-scheme>

      <scheme-name>time-limited-backing-map</scheme-name>

      <expiry-delay>1200s</expiry-delay>

      </local-scheme>

      <proxy-scheme>

      <service-name>ExtendTcpProxyService</service-name>

      <acceptor-config>

      <tcp-acceptor>

      <local-address>

      <address>localhost</address>

      <port>9188</port>

      </local-address>

      <tcp-delay-enabled>false</tcp-delay-enabled>

      </tcp-acceptor>

      </acceptor-config>

      <load-balancer>proxy</load-balancer>

      <autostart system-property="tangosol.coherence.extend.enabled">false</autostart>

      </proxy-scheme>

      </caching-schemes>

      </cache-config>

       

       

      tangosol-coherence-override.xml:

      <cluster-config>

      <member-identity>

      <cluster-name system-property="tangosol.coherence.cluster">dtoken</cluster-name>

      </member-identity>

       

       

      <multicast-listener>

      <address>224.30.9.40</address>

      <port>3166</port>

      <time-to-live>4</time-to-live>

      </multicast-listener>

       

       

      <tcp-ring-listener>

      <ip-timeout system-property="tangosol.coherence.ipmonitor.pingtimeout">25s</ip-timeout>

      <ip-attempts system-property="tangosol.coherence.ipmonitor.pingtattempts">5</ip-attempts>

      </tcp-ring-listener>

       

       

      <service-guardian>

      <timeout-milliseconds>610000</timeout-milliseconds>

      </service-guardian>

       

       

       

       

      <packet-publisher>

      <packet-delivery>

      <timeout-milliseconds>600000</timeout-milliseconds>

      <flow-control>

      <pause-detection>

      <maximum-packets>30</maximum-packets>

      </pause-detection>

       

       

      </flow-control>

      <packet-bundling>

      <maximum-deferral-time>50us</maximum-deferral-time>

      <aggression-factor>0</aggression-factor>

      </packet-bundling>

      <services>

       

       

       

       

      <service id="7">

      <init-params>

      <init-param id="17">

      <param-name>thread-count-max</param-name>

      <param-value system-property="tangosol.coherence.proxy.threads.max">200</param-value>

      </init-param>

       

       

      <init-param id="18">

      <param-name>thread-count-min</param-name>

      <param-value system-property="tangosol.coherence.proxy.threads.min">32</param-value>

      </init-param>

      </init-params>

      </service>

      </services>

      </cluster-config>

       

       

      <logging-config>

      <destination system-property="tangosol.coherence.log">log4j</destination>

      <logger-name system-property="tangosol.coherence.log.logger">dcachelogger</logger-name>

      <severity-level system-property="tangosol.coherence.log.level">9</severity-level>

      </logging-config>

       

       

      <configurable-cache-factory-config>

      <init-params>

      <init-param>

      <param-type>java.lang.String</param-type>

      <param-value system-property="tangosol.coherence.cacheconfig">coherence-cache-config-server.xml</param-value>

      </init-param>

      </init-params>

      </configurable-cache-factory-config>

      </coherence>

       

       

       

       

       

      jvm startup parameter:

       

       

      PROXY:

      java -server -showversion -Xms2g -Xmx2g -Dtangosol.coherence.extend.enabled=true -Dcache.name=proxy -Dtangosol.coherence.distributed.localstorage=false -XX:NewRatio=4 -XX:SurvivorRatio=32 -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSParallelRemarkEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=50 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/home/ap/test/slog/gc_proxy.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ap/test/slog/dump_proxy.hprof -cp .:/home/ap/test/jdk/jdk1.7.0_55/lib/dt.jar:/home/ap/test/jdk/jdk1.7.0_55/lib/tools.jar:/home/ap/test/conf:/home/ap/test/coherence_12c/coherence/lib/coherence.jar:/home/ap/test/lib/* com.tangosol.net.DefaultCacheServer

       

       

      SERVER1:

      java -server -showversion -Xms10g -Xmx10g -Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true -Dcom.sun.management.jmxremote.port=9288 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcache.name=-jmx -XX:PermSize=256m -XX:NewRatio=4 -XX:SurvivorRatio=32 -XX:+UseConcMarkSweepGC -XX:CMSFullGCsBeforeCompaction=4 -XX:+CMSParallelRemarkEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxTenuringThreshold=5 -XX:CMSInitiatingOccupancyFraction=75 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/home/ap/test/slog/gc_Srv.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ap/test/slog/dump_server.hprof -cp .:/home/ap/test/jdk/jdk1.7.0_55/lib/dt.jar:/home/ap/test/jdk/jdk1.7.0_55/lib/tools.jar:/home/ap/test/conf:/home/ap/test/coherence_12c/coherence/lib/coherence.jar:/home/ap/test/lib/* com.tangosol.net.DefaultCacheServer server1

       

       

      SERVER2:

      java -server -showversion -Xms10g -Xmx10g -Dcache.name=server2 -XX:PermSize=256m -XX:NewRatio=4 -XX:SurvivorRatio=32 -XX:+UseConcMarkSweepGC -XX:CMSFullGCsBeforeCompaction=4 -XX:+CMSParallelRemarkEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxTenuringThreshold=5 -XX:CMSInitiatingOccupancyFraction=75 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/home/ap/test/slog/gc_Srv.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ap/test/slog/dump_server.hprof -cp .:/home/ap/test/jdk/jdk1.7.0_55/lib/dt.jar:/home/ap/test/jdk/jdk1.7.0_55/lib/tools.jar:/home/ap/test/conf:/home/ap/test/coherence_12c/coherence/lib/coherence.jar:/home/ap/test/lib/* com.tangosol.net.DefaultCacheServer server2

       

       

       

       

       

       

      proxy log:

      2019-02-14 15:46:59.939/1115272.398 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 52

      2019-02-14 15:48:00.007/1115332.466 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 32

      2019-02-14 15:48:31.822/1115364.281 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 34

      2019-02-14 15:48:44.496/1115376.955 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 36

      2019-02-14 15:48:56.641/1115389.100 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 38

      2019-02-14 15:49:42.957/1115435.416 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 40

      2019-02-14 15:49:54.554/1115447.013 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 41

      2019-02-14 15:49:58.009/1115450.468 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 43

      2019-02-14 15:49:59.280/1115451.739 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 45

      2019-02-14 15:50:58.462/1115510.921 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 46

      2019-02-14 15:51:10.714/1115523.173 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 48

      2019-02-14 15:51:44.921/1115557.380 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 50

      2019-02-14 15:52:29.129/1115601.588 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 51

      2019-02-14 15:52:31.217/1115603.676 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 53

      2019-02-14 15:52:32.521/1115604.980 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 54

      2019-02-14 15:52:53.879/1115626.338 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 56

      2019-02-14 15:52:59.496/1115631.955 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=7): Worker thread pool size is 58

       

      gc_proxy.log:

       

      11876416.164: [GC11876416.164: [ParNew: 402760K->7112K(407104K), 0.0050180 secs] 782939K->387290K(2084864K), 0.0052780 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]

      11876427.825: [GC11876427.825: [ParNew: 401928K->5776K(407104K), 0.0059950 secs] 782106K->385955K(2084864K), 0.0062100 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]

      11876439.098: [GC11876439.098: [ParNew: 400592K->7649K(407104K), 0.0049230 secs] 780771K->387846K(2084864K), 0.0051750 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876450.615: [GC11876450.615: [ParNew: 402465K->8719K(407104K), 0.0056490 secs] 782662K->388960K(2084864K), 0.0059710 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]

      11876462.489: [GC11876462.489: [ParNew: 403535K->6615K(407104K), 0.0048660 secs] 783776K->386890K(2084864K), 0.0050740 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]

      11876474.174: [GC11876474.174: [ParNew: 401431K->6103K(407104K), 0.0052390 secs] 781706K->386418K(2084864K), 0.0055010 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876485.922: [GC11876485.922: [ParNew: 400919K->7442K(407104K), 0.0065900 secs] 781234K->387804K(2084864K), 0.0068790 secs] [Times: user=0.05 sys=0.00, real=0.00 secs]

      11876497.631: [GC11876497.631: [ParNew: 402258K->5562K(407104K), 0.0057830 secs] 782620K->385948K(2084864K), 0.0060820 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876509.171: [GC11876509.171: [ParNew: 400378K->6412K(407104K), 0.0049350 secs] 780764K->386824K(2084864K), 0.0051680 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]

      11876520.892: [GC11876520.892: [ParNew: 401227K->4043K(407104K), 0.0062310 secs] 781640K->384477K(2084864K), 0.0064890 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]

      11876532.509: [GC11876532.509: [ParNew: 398859K->4723K(407104K), 0.0063790 secs] 779293K->385201K(2084864K), 0.0067260 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]

      11876543.915: [GC11876543.915: [ParNew: 399539K->7610K(407104K), 0.0056990 secs] 780017K->388118K(2084864K), 0.0060380 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876555.420: [GC11876555.420: [ParNew: 402426K->5304K(407104K), 0.0055480 secs] 782934K->385829K(2084864K), 0.0057710 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876566.905: [GC11876566.905: [ParNew: 400113K->4769K(407104K), 0.0048040 secs] 780638K->386390K(2084864K), 0.0050400 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]

      11876578.335: [GC11876578.335: [ParNew: 399585K->3807K(407104K), 0.0054440 secs] 781206K->385458K(2084864K), 0.0056530 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]

      11876590.091: [GC11876590.091: [ParNew: 398623K->3514K(407104K), 0.0056560 secs] 780274K->385196K(2084864K), 0.0058700 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876601.437: [GC11876601.437: [ParNew: 398330K->6985K(407104K), 0.0047840 secs] 780012K->388678K(2084864K), 0.0050440 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876613.193: [GC11876613.193: [ParNew: 401801K->4246K(407104K), 0.0045930 secs] 783494K->385968K(2084864K), 0.0048340 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]

      11876624.977: [GC11876624.977: [ParNew: 399060K->4383K(407104K), 0.0056550 secs] 780782K->386146K(2084864K), 0.0058600 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876636.696: [GC11876636.697: [ParNew: 399046K->7400K(407104K), 0.0060520 secs] 780809K->389191K(2084864K), 0.0062660 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876648.500: [GC11876648.500: [ParNew: 402216K->7373K(407104K), 0.0046310 secs] 784007K->389190K(2084864K), 0.0049310 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876660.065: [GC11876660.065: [ParNew: 402189K->8497K(407104K), 0.0054180 secs] 784006K->390337K(2084864K), 0.0056440 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876671.465: [GC11876671.465: [ParNew: 403313K->8131K(407104K), 0.0048940 secs] 785153K->389971K(2084864K), 0.0051560 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876683.319: [GC11876683.319: [ParNew: 402947K->5404K(407104K), 0.0053560 secs] 784787K->387258K(2084864K), 0.0056730 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876694.797: [GC11876694.797: [ParNew: 400220K->6334K(407104K), 0.0046740 secs] 782074K->388202K(2084864K), 0.0049170 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876706.330: [GC11876706.330: [ParNew: 401150K->4080K(407104K), 0.0051210 secs] 783018K->385975K(2084864K), 0.0053370 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]

      11876717.695: [GC11876717.695: [ParNew: 398896K->6533K(407104K), 0.0046100 secs] 780791K->388437K(2084864K), 0.0048540 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876729.075: [GC11876729.075: [ParNew: 401349K->8102K(407104K), 0.0052860 secs] 783253K->390006K(2084864K), 0.0055260 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876740.377: [GC11876740.378: [ParNew: 402918K->7036K(407104K), 0.0050140 secs] 784822K->388982K(2084864K), 0.0052660 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876751.842: [GC11876751.842: [ParNew: 401852K->5642K(407104K), 0.0052990 secs] 783798K->387625K(2084864K), 0.0055640 secs] [Times: user=0.04 sys=0.00, real=0.00 secs]

      11876763.610: [GC11876763.610: [ParNew: 400458K->5433K(407104K), 0.0047420 secs] 782441K->387448K(2084864K), 0.0049980 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876775.353: [GC11876775.353: [ParNew: 400249K->7511K(407104K), 0.0052350 secs] 782264K->389548K(2084864K), 0.0055150 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876787.174: [GC11876787.174: [ParNew: 402327K->8166K(407104K), 0.0058020 secs] 784364K->390224K(2084864K), 0.0060220 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]

      11876798.856: [GC11876798.856: [ParNew: 402982K->7274K(407104K), 0.0055490 secs] 785040K->389353K(2084864K), 0.0057620 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

      11876810.486: [GC11876810.486: [ParNew: 402090K->8194K(407104K), 0.0052780 secs] 784169K->390297K(2084864K), 0.0055650 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]

      11876822.054: [GC11876822.055: [ParNew: 403010K->6040K(407104K), 0.0062610 secs] 785113K->388184K(2084864K), 0.0065520 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

       

      server log:

      2019-02-14 14:34:05.199/1110897.273 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=PacketPublisher, member=8): Member(Id=11, Timestamp=2019-02-01 18:04:34.719, Address=11.169.35.77:8090, MachineId=48266, Location=site:,machine:testap2016,process:2308, Role=CoherenceServer) has failed to respond to 31 packets; declaring this member as paused.

      2019-02-14 14:34:05.216/1110897.290 Oracle Coherence GE 12.1.3.0.0 <D8> (thread=PacketPublisher, member=8): Experienced a 17 ms communication delay (probable remote GC) with member 11; 158 packets rescheduled, PauseRate=0.0, Threshold=357

      2019-02-14 14:39:21.041/1111213.115 Oracle Coherence GE 12.1.3.0.0 <D9> (thread=PacketPublisher, member=8): Member(Id=17, Timestamp=2019-02-01 18:09:54.275, Address=11.169.35.93:8089, MachineId=59753, Location=site:,machine:testap3016,process:11828, Role=CoherenceServer) has failed to respond to 31 packets; declaring this member as paused.

      2019-02-14 14:39:21.058/1111213.132 Oracle Coherence GE 12.1.3.0.0 <D8> (thread=PacketPublisher, member=8): Experienced a 17 ms communication delay (probable remote GC) with member 17; 98 packets rescheduled, PauseRate=0.0, Threshold=415

       

      gc_Srv.log:

      11869740.739: [GC11869740.739: [ParNew: 2027714K->41395K(2035520K), 0.0553630 secs] 6438646K->4467693K(10424128K), 0.0556260 secs] [Times: user=0.32 sys=0.00, real=0.06 secs]

      11870050.650: [GC11870050.650: [ParNew: 2015283K->50504K(2035520K), 0.0240700 secs] 6441581K->4476803K(10424128K), 0.0243360 secs] [Times: user=0.15 sys=0.00, real=0.02 secs]

      11870390.936: [GC11870390.936: [ParNew: 2024392K->43908K(2035520K), 0.0338310 secs] 6450691K->4487811K(10424128K), 0.0340680 secs] [Times: user=0.21 sys=0.00, real=0.04 secs]

      11870711.292: [GC11870711.292: [ParNew: 2017796K->48474K(2035520K), 0.0326790 secs] 6461699K->4492377K(10424128K), 0.0329130 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]

      11871010.395: [GC11871010.396: [ParNew: 2022362K->34733K(2035520K), 0.0485260 secs] 6466265K->4500696K(10424128K), 0.0488250 secs] [Times: user=0.29 sys=0.00, real=0.05 secs]

      11871332.487: [GC11871332.487: [ParNew: 2008621K->61632K(2035520K), 0.0324370 secs] 6474584K->4531896K(10424128K), 0.0327280 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]

      11871659.212: [GC11871659.212: [ParNew: 2035520K->61632K(2035520K), 0.0664270 secs] 6505784K->4564906K(10424128K), 0.0667170 secs] [Times: user=0.43 sys=0.00, real=0.07 secs]

      11871958.300: [GC11871958.301: [ParNew: 2035520K->61632K(2035520K), 0.0399310 secs] 6538794K->4586388K(10424128K), 0.0402480 secs] [Times: user=0.26 sys=0.00, real=0.04 secs]

      11872216.060: [GC11872216.060: [ParNew: 2035520K->61632K(2035520K), 0.0342550 secs] 6560276K->4612977K(10424128K), 0.0345100 secs] [Times: user=0.23 sys=0.00, real=0.03 secs]

      11872493.837: [GC11872493.837: [ParNew: 2035520K->61632K(2035520K), 0.0350320 secs] 6586865K->4636077K(10424128K), 0.0353250 secs] [Times: user=0.21 sys=0.00, real=0.04 secs]

      11872756.698: [GC11872756.698: [ParNew: 2035520K->61632K(2035520K), 0.0491770 secs] 6609965K->4672559K(10424128K), 0.0494730 secs] [Times: user=0.29 sys=0.00, real=0.05 secs]

      11872995.653: [GC11872995.654: [ParNew: 2035520K->61632K(2035520K), 0.0570740 secs] 6646447K->4725399K(10424128K), 0.0573900 secs] [Times: user=0.31 sys=0.00, real=0.06 secs]

      11873252.075: [GC11873252.075: [ParNew: 2035520K->61632K(2035520K), 0.0403620 secs] 6699287K->4772172K(10424128K), 0.0406990 secs] [Times: user=0.22 sys=0.00, real=0.04 secs]

      11873500.627: [GC11873500.627: [ParNew: 2035520K->61632K(2035520K), 0.0420240 secs] 6746060K->4820035K(10424128K), 0.0423000 secs] [Times: user=0.24 sys=0.00, real=0.04 secs]

      11873762.439: [GC11873762.439: [ParNew: 2035520K->61632K(2035520K), 0.0366580 secs] 6793923K->4856693K(10424128K), 0.0369550 secs] [Times: user=0.22 sys=0.00, real=0.04 secs]

      11874012.083: [GC11874012.084: [ParNew: 2035520K->61632K(2035520K), 0.0833490 secs] 6830581K->4898764K(10424128K), 0.0836060 secs] [Times: user=0.50 sys=0.00, real=0.08 secs]

      11874287.012: [GC11874287.012: [ParNew: 2035520K->61632K(2035520K), 0.0390040 secs] 6872652K->4936219K(10424128K), 0.0393130 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]

      11874567.907: [GC11874567.908: [ParNew: 2035520K->61632K(2035520K), 0.0345790 secs] 6910107K->4967914K(10424128K), 0.0348790 secs] [Times: user=0.21 sys=0.00, real=0.03 secs]

      11874812.440: [GC11874812.440: [ParNew: 2035520K->61632K(2035520K), 0.0469770 secs] 6941802K->5012464K(10424128K), 0.0472970 secs] [Times: user=0.26 sys=0.00, real=0.05 secs]

      11875078.856: [GC11875078.857: [ParNew: 2035520K->61632K(2035520K), 0.0417440 secs] 6986352K->5054254K(10424128K), 0.0420540 secs] [Times: user=0.23 sys=0.00, real=0.04 secs]

      11875339.830: [GC11875339.830: [ParNew: 2035520K->61632K(2035520K), 0.1195210 secs] 7028142K->5103322K(10424128K), 0.1198060 secs] [Times: user=0.72 sys=0.00, real=0.12 secs]

      11875596.962: [GC11875596.962: [ParNew: 2035520K->61632K(2035520K), 0.0676760 secs] 7077210K->5151367K(10424128K), 0.0679970 secs] [Times: user=0.39 sys=0.00, real=0.07 secs]

      11875848.177: [GC11875848.177: [ParNew: 2035520K->61632K(2035520K), 0.0705240 secs] 7125255K->5230762K(10424128K), 0.0708380 secs] [Times: user=0.39 sys=0.00, real=0.07 secs]

      11876130.492: [GC11876130.492: [ParNew: 2035520K->61632K(2035520K), 0.0370790 secs] 7204650K->5278029K(10424128K), 0.0373830 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]

      11876412.726: [GC11876412.726: [ParNew: 2035520K->61632K(2035520K), 0.1412060 secs] 7251917K->5348617K(10424128K), 0.1414880 secs] [Times: user=0.72 sys=0.00, real=0.14 secs]

      11876665.323: [GC11876665.323: [ParNew: 2035520K->61632K(2035520K), 0.0328150 secs] 7322505K->5389631K(10424128K), 0.0331790 secs] [Times: user=0.21 sys=0.00, real=0.04 secs]

       

       

      jstat(after running 3 months):

      jstat.png

        • 1. Re: Oracle Coherence more and more slower with time
          user1039684

          we are also seeing similar errors, is this related to GC happening?

           

          Oracle Coherence GE 12.1.3.0.0 <Warning> (thread=PacketPublisher, member=1): Experienced a 4202 ms communication delay (probable remote GC) with Member(Id=27, Timestamp=2019-04-02 13:12:05.544, Address=X.X.X.X:2152, MachineId=51912, Location=site:RCO,,process:6707, Role=WeblogicServer); 1192 packets rescheduled, PauseRate=0.001, Threshold=1157>