2 Replies Latest reply: May 23, 2013 6:23 AM by rshanker RSS

    Cache Unicast failure

    rshanker
      Hi all,
      Im facing issue while running the Coherence in unicast cluster.

      1. I have configured the unicast clustering in the coherence with Cache stores
      2. Ran one java program from one of the WKA-1 which does some puts and gets.
      3. Ran another java program in the other WKA-2 which does only gets. here the issue is the WKA-2 is getting started but the program to get is not running.

      Also an important query: How to configure mutliple nodes using the Unicast mechanism??

      Following is the configuration:
      Here is the configuration snippets:

      cache-config.xml
      <?xml version="1.0"?>

      <cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
      xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">
      <caching-scheme-mapping>
      <cache-mapping>
      <cache-name>dbexample</cache-name>
      <scheme-name>distributed</scheme-name>
      </cache-mapping>
      </caching-scheme-mapping>

      <caching-schemes>
      <distributed-scheme>
      <scheme-name>distributed</scheme-name>
      <service-name>DistributedCache</service-name>
      <thread-count>4</thread-count>
      <request-timeout>60s</request-timeout>
      <backing-map-scheme>
      <read-write-backing-map-scheme>
      <internal-cache-scheme>
      <local-scheme>
      <scheme-name>SampleMemoryScheme</scheme-name>
      </local-scheme>
      </internal-cache-scheme>
      <cachestore-scheme>
      <class-scheme>
      <class-name>com.coherence.KnDBCacheStore</class-name>
      </class-scheme>
      </cachestore-scheme>
      </read-write-backing-map-scheme>
      </backing-map-scheme>
      <autostart>true</autostart>
      </distributed-scheme>
      <local-scheme>
      <scheme-name>LocalSizeLimited</scheme-name>
      <eviction-policy>LRU</eviction-policy>
      <high-units>1000</high-units>
      <expiry-delay>1h</expiry-delay>
      </local-scheme>
      </caching-schemes>
      </cache-config>

      ==========================

      tangosol-coherence-override.xml

      <?xml version='1.0'?>

      <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
      xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">
      <cluster-config>
      <member-identity>
      <cluster-name system-property="tangosol.coherence.cluster">kn_test</cluster-name>
      </member-identity>
      <unicast-listener>
      <well-known-addresses>
      <socket-address id="1">
      <address>192.168.7.3</address>
      <port>8088</port>
      </socket-address>
      <socket-address id="2">
      <address>192.168.7.4</address>
      <port>8088</port>
      </socket-address>
      </well-known-addresses>
      </unicast-listener>
      </cluster-config>

      <configurable-cache-factory-config>
      <init-params>
      <init-param>
      <param-type>java.lang.String</param-type>
      <param-value system-property="tangosol.coherence.cacheconfig">cache-config.xml</param-value>
      </init-param>
      </init-params>
      </configurable-cache-factory-config>
      </coherence>

      Edited by: rshanker on May 23, 2013 2:18 AM
        • 1. Re: Cache Unicast failure
          rshanker
          The snapshot of the logs from the Server - 2 (192.168.7.4)


          Inside the Hello World1
          2013-05-23 15:56:26.862/3.699 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/RAVI/Coherence/coherence/lib/coherence.jar!/tangosol-coherence.xml"
          2013-05-23 15:56:26.935/3.772 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/RAVI/Coherence/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
          2013-05-23 15:56:26.977/3.814 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/home/RAVI/Coherence/coherence/config/tangosol-coherence-override.xml"
          2013-05-23 15:56:26.979/3.816 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

          Oracle Coherence Version 3.7.1.0 Build 27797
          Grid Edition: Development mode
          Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

          2013-05-23 15:56:27.522/4.359 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /192.168.7.4:8088 using SystemSocketProvider
          2013-05-23 15:56:27.943/4.780 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=3, Timestamp=2013-05-23 15:42:06.84, Address=192.168.7.4:8088, MachineId=28420, Location=site:,process:22097, Role=HelloWorld1, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) joined cluster "kn_test" with senior Member(Id=1, Timestamp=2013-05-23 15:40:40.882, Address=192.168.7.3:8088, MachineId=52287, Location=site:,machine:prov,process:8334, Role=HelloWorld, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1)


          After this im not seeing further continuation of the java program.



          In the WKA-1 running java program the logs are as follows when the WKA2 java program is started:
          2013-05-23 16:08:10.510/19.955 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2013-05-23 16:08:10.32, Address=192.168.7.4:8089, MachineId=28420, Location=site:,process:28151, Role=HelloWorld1) joined Cluster with senior member 1
          • 2. Re: Cache Unicast failure
            rshanker
            Few more exceptions on WKA-2 are as follows:

            2013-05-23 17:06:34.315/278.962 Oracle Coherence GE 3.7.1.0 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}}
            2013-05-23 17:06:34.318/278.965 Oracle Coherence GE 3.7.1.0 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump

            Thread[Signal Dispatcher,9,system]

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[main,5,main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
            com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
            com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
            com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
            com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
            com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
            com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
            com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
            com.kodiak.HelloWorld1.main(HelloWorld1.java:16)

            Thread[KnDBActivePollThread,5,main]
            java.lang.Object.wait(Native Method)
            java.util.TimerThread.mainLoop(Timer.java:552)
            java.util.TimerThread.run(Timer.java:505)

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[PacketReceiver,7,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:722)

            ThreadCluster
            sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
            sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
            sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
            sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
            sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
            com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:11)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onWait(ClusterService.CDB:6)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:722)

            Thread[Logger@9271164 3.7.1.0,3,main]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:722)

            Thread[PacketSpeaker,8,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.queue.ConcurrentQueue.waitForEntry(ConcurrentQueue.CDB:16)
            com.tangosol.coherence.component.util.queue.ConcurrentQueue.remove(ConcurrentQueue.CDB:7)
            com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:21)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:722)

            Thread[Finalizer,8,system]
            java.lang.Object.wait(Native Method)
            java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
            java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
            java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

            Thread[Thread-1,5,main]
            java.lang.Thread.sleep(Native Method)
            org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[DestroyPoolThread,5,main]
            java.lang.Object.wait(Native Method)
            java.util.TimerThread.mainLoop(Timer.java:552)
            java.util.TimerThread.run(Timer.java:505)

            Thread[Reference Handler,10,system]
            java.lang.Object.wait(Native Method)
            java.lang.Object.wait(Object.java:503)
            java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[Dispatcher-Thread-0,1,main]
            java.lang.Object.wait(Native Method)
            java.lang.Object.wait(Object.java:503)
            org.apache.log4j.Dispatcher.run(AsyncAppender.java:293)

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[PacketListener1P,8,Cluster]
            java.net.PlainDatagramSocketImpl.receive0(Native Method)
            java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
            java.net.DatagramSocket.receive(DatagramSocket.java:786)
            com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
            com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:722)

            Thread[DB-MANAGER-THREAD,5,main]
            java.lang.Object.wait(Native Method)
            java.lang.Object.wait(Object.java:503)
            com.kodiak.ems.dbmgr.KnDBManager.run(KnDBManager.java:306)

            Thread[Timer-0,5,main]
            java.lang.Object.wait(Native Method)
            java.util.TimerThread.mainLoop(Timer.java:552)
            java.util.TimerThread.run(Timer.java:505)

            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
            java.lang.Object.wait(Native Method)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

            Thread[PacketPublisher,6,Cluster]
            java.lang.Object.wait(Native Method)
            com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
            java.lang.Thread.run(Thread.java:722)

            Thread[Recovery Thread,5,Cluster]
            java.lang.Thread.dumpThreads(Native Method)
            java.lang.Thread.getAllStackTraces(Thread.java:1618)
            com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
            com.tangosol.internal.net.cluster.DefaultServiceFailurePolicy.onGuardableRecovery(DefaultServiceFailurePolicy.java:44)
            com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$WrapperGuardable.recover(Grid.CDB:1)
            com.tangosol.net.GuardSupport$Context$1.run(GuardSupport.java:653)
            java.lang.Thread.run(Thread.java:722)

            Thread[PacketListener1,8,Cluster]
            java.net.PlainDatagramSocketImpl.receive0(Native Method)
            java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
            java.net.DatagramSocket.receive(DatagramSocket.java:786)
            com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:22)
            com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:1)
            com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:20)
            com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
            java.lang.Thread.run(Thread.java:722)

            2013-05-23 17:06:34.320/278.967 Oracle Coherence GE 3.7.1.0 <Warning> (thread=Recovery Thread, member=n/a): Attempting recovery of Guard{Daemon=IpMonitor}