This discussion is archived
4 Replies Latest reply: Jan 30, 2013 2:48 AM by Angel.Ruiz RSS

How does Coherence decide to send out com.tangosol.util.Binary ..... ?

987923 Newbie
Currently Being Moderated
Hello,

sorry if this is a total newbie question.

Can anyone tell me if there are particular circumstances whereby Coherence will send out "com.tangosol.util.Binary" entities instead of the actual "Objects"?

We have recently tried to switch from using local caches to a distributed cache. We brought up node1 and saw "Created cluster", brought up node2 and saw "joined cluster", brought up node3 and saw "joined cluster". All looked good. And then we started seeing thousands of errors of this form:

java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.blah.blah.blah.......

It seems that Coherence wants us to deserialise the Binary object instead of doing a cast. This is fine, we can make that change, but the weird thing is that we are struggling to reliably reproduce the problem. We only saw the ClassCastException when we went to Production, so now we've gone back to our Dev environment and tried to reproduce it. It occurred yesterday on our 2nd node, and we then added a log statement (i absolutely and completely promise, it is just a log statement!) and restarted. Since then, no error.

Am gettting our Devs to roll back that change but in the absence of that working i'd really appreciate if anyone knows anything about the underlying code and can help me with this.

many thanks,
Darren Strachan
  • 1. Re: How does Coherence decide to send out com.tangosol.util.Binary ..... ?
    user738616 Pro
    Currently Being Moderated
    Hi,

    Coherence serializes the objects and keeps them in Binary form when using distributed cache and keeps in object form when using Local Cache. Is your object serialization/deserialization routine available in the classpath of the nodes?

    Cheers,
    NJ
  • 2. Re: How does Coherence decide to send out com.tangosol.util.Binary ..... ?
    DJW Newbie
    Currently Being Moderated
    Can you post the full stack trace? That might give a clue as to the cause.
  • 3. Re: How does Coherence decide to send out com.tangosol.util.Binary ..... ?
    987923 Newbie
    Currently Being Moderated
    Hi there,

    thanks for the responses. Below is an extract out of the log.

    Thanks,
    Darren

    ------------------------------------------

    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RaceIdAndCountries
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetRequest(DistributedCache.CDB:40)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
         ... 1 more","2013-01-24T11:11:41.396+0000",,,,,,11,24,11,january,41,41,thursday,2013,local,,,"prdfrb001.prd.betfair",,"production-bf-racecard-forms-results-api",19,,"Exception thrown:
    com.betfair.cougar.core.api.exception.CougarServiceException: Exception thrown by service method
         at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:54)
         at com.betfair.cougar.core.impl.ev.BaseExecutionVenue.execute(BaseExecutionVenue.java:103)
         at com.betfair.cougar.transport.impl.AbstractCommandProcessor$1.run(AbstractCommandProcessor.java:109)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RaceIdAndCountries
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetRequest(DistributedCache.CDB:40)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
         ... 1 more","com.betfair.cougar.core.api.exception.CougarServiceException",,396,,,,,,,,,"--_::.:_......__-__:_......:_____t_.......(.:)t_..",,,,,"/var/log/bf-racecard-forms-results-api/prdfrb001.prd.betfair-server.log",server,"ie1lgi003.inf.betfair-root",WARN,,,,,,,
    "10.40.178.30:8088",,,,,3,,"site:prd.betfair",36894,,,BetfairCougarCoreImplMain,,,,,"2013-01-24 11:09:19.277",,,"2013-01-24 11:11:41.275: com.betfair.cougar.core.api.exception.CougarServiceException WARN - Exception thrown:
    com.betfair.cougar.core.api.exception.CougarServiceException: Exception thrown by service method
         at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:54)
         at com.betfair.cougar.core.impl.ev.BaseExecutionVenue.execute(BaseExecutionVenue.java:103)
         at com.betfair.cougar.transport.impl.AbstractCommandProcessor$1.run(AbstractCommandProcessor.java:109)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RaceIdAndCountries
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetAllRequest(DistributedCache.CDB:60)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetAllRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest.onReceived(DistributedCacheRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
         ... 1 more","2013-01-24T11:11:41.275+0000",,,,,,11,24,11,january,41,41,thursday,2013,local,,,"prdfrb001.prd.betfair",,"production-bf-racecard-forms-results-api",19,,"Exception thrown:
    com.betfair.cougar.core.api.exception.CougarServiceException: Exception thrown by service method
         at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:54)
         at com.betfair.cougar.core.impl.ev.BaseExecutionVenue.execute(BaseExecutionVenue.java:103)
         at com.betfair.cougar.transport.impl.AbstractCommandProcessor$1.run(AbstractCommandProcessor.java:109)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RaceIdAndCountries
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetAllRequest(DistributedCache.CDB:60)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetAllRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest.onReceived(DistributedCacheRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
         ... 1 more","com.betfair.cougar.core.api.exception.CougarServiceException",,275,,,,,,,,,"--_::.:_......__-__:_......:_____t_.......(.:)t_..",,,,,"/var/log/bf-racecard-forms-results-api/prdfrb001.prd.betfair-server.log",server,"ie1lgi003.inf.betfair-root",WARN,,,,,,,
    "10.40.178.30:8088",,,,,3,,"site:prd.betfair",36894,,,BetfairCougarCoreImplMain,,,,,"2013-01-24 11:09:19.277",,,"2013-01-24 11:11:40.821: com.betfair.cougar.core.api.exception.CougarServiceException WARN - Exception thrown:
    com.betfair.cougar.core.api.exception.CougarServiceException: Exception thrown by service method
         at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:54)
         at com.betfair.cougar.core.impl.ev.BaseExecutionVenue.execute(BaseExecutionVenue.java:103)
         at com.betfair.cougar.transport.impl.AbstractCommandProcessor$1.run(AbstractCommandProcessor.java:109)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RunnerKey
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetRequest(DistributedCache.CDB:40)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
         ... 1 more","2013-01-24T11:11:40.821+0000",,,,,,11,24,11,january,40,40,thursday,2013,local,,,"prdfrb001.prd.betfair",,"production-bf-racecard-forms-results-api",19,,"Exception thrown:
    com.betfair.cougar.core.api.exception.CougarServiceException: Exception thrown by service method
         at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:54)
         at com.betfair.cougar.core.impl.ev.BaseExecutionVenue.execute(BaseExecutionVenue.java:103)
         at com.betfair.cougar.transport.impl.AbstractCommandProcessor$1.run(AbstractCommandProcessor.java:109)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: (Wrapped: Failed request execution for NonBackedUpDistributedCache service on Member(Id=3, Timestamp=2013-01-24 11:09:19.277, Address=10.40.178.30:8088, MachineId=36894, Location=site:prd.betfair,machine:prdfrb003,process:3482, Role=BetfairCougarCoreImplMain)) java.lang.ClassCastException: com.tangosol.util.Binary cannot be cast to com.betfair.formsandresults.cache.key.RunnerKey
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:286)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onGetRequest(DistributedCache.CDB:40)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$GetRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:130)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:37)
  • 4. Re: How does Coherence decide to send out com.tangosol.util.Binary ..... ?
    Angel.Ruiz Newbie
    Currently Being Moderated
    Hi,

    when is this exception thrown? is it thrown during a simple get operation from a remote client or a storage-disable member of the grid?
    Is it thrown within a backing map listener?

    Angel

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points