6 Replies Latest reply: Jan 7, 2013 8:45 PM by tmiddlet RSS

    Cache will be blank after RequestTimeoutException

    969508
      Hi,
      I am facing this below RequestTimeoutException issue with my Coherence*Extend-TCP client code and it is happening intermittently, not always. but I am more concerned about after receiving this exception my cache node will become blank and again I have to load my cache objects.

      com.tangosol.net.RequestTimeoutException: request timed out after 5000 millis. Even I tried with <connect-timeout>10s</connect-timeout> and upto 1 min.
      And I have two index keys in my coherence cache and it will always have more than 1 million cache objects, will this cause this RequestTimeoutException.

      Could someone please help me, Is there a way, so that I can make sure my cache objects will be safe within the cluster and how to overcome from this RequestTimeoutException.

      Thank in advance,
      Venki


      Oracle Coherence Version 3.7.1.0 Build 27797
      Grid Edition: Development mode
      Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

      2012-12-27 00:09:44.035/7.937 Oracle Coherence GE 3.7.1.0 <Info> (thread=Thread-9, member=n/a): Loaded cache configuration from "file:/nas/sdgshared/MyApp/devHarsha/config/MyApp-cache-config.xml"
      2012-12-27 00:09:44.258/8.160 Oracle Coherence GE 3.7.1.0 <D5> (thread=MyAppTcpInvocationService:TcpInitiator, member=n/a): Started: TcpInitiator{Name=MyAppTcpInvocationService:TcpInitiator, State=(SERVICE_STARTED), ThreadCount=0, Codec=Codec(Format=POF), Serializer=com.tangosol.io.DefaultSerializer, PingInterval=0, PingTimeout=5000, RequestTimeout=5000, ConnectTimeout=10000, SocketProvider=SystemSocketProvider, RemoteAddresses=[ctrel9012-12/10.176.4.104:9099], SocketOptions{LingerTimeout=0, KeepAliveEnabled=true, TcpDelayEnabled=false}}
      2012-12-27 00:09:44.261/8.163 Oracle Coherence GE 3.7.1.0 <D5> (thread=Thread-9, member=n/a): Connecting Socket to 10.176.4.104:9099
      2012-12-27 00:09:44.261/8.163 Oracle Coherence GE 3.7.1.0 <Info> (thread=Thread-9, member=n/a): Connected Socket to 10.176.4.104:9099
      2012-12-27 00:09:44.310/8.212 Oracle Coherence GE 3.7.1.0 <D6> (thread=MyAppTcpInvocationService:TcpInitiator, member=n/a): Opened: TcpConnection(Id=0x0000013BDAFB0D640AB00468787D2D3A03C34C7A7052322BCFDF1CD939E6A33C, Open=true, Member(Id=0, Timestamp=2012-12-27 00:09:44.273, Address=10.176.1.77:0, MachineId=0, Location=site:,machine:ctbr024-12,process:23132, Role=WestgroupMyAppProcessorManager), LocalAddress=10.176.1.77:54988, RemoteAddress=10.176.4.104:9099)
      2012-12-27 00:09:44.333/8.235 Oracle Coherence GE 3.7.1.0 <D6> (thread=MyAppTcpInvocationService:TcpInitiator, member=n/a): Opened: Channel(Id=1938366130, Open=true, Connection=0x0000013BDAFB0D640AB00468787D2D3A03C34C7A7052322BCFDF1CD939E6A33C)
      2012-12-27 00:09:44.387/8.289 Oracle Coherence GE 3.7.1.0 <D5> (thread=MyAppTcpCacheService:TcpInitiator, member=n/a): Started: TcpInitiator{Name=MyAppTcpCacheService:TcpInitiator, State=(SERVICE_STARTED), ThreadCount=0, Codec=Codec(Format=POF), Serializer=com.tangosol.io.DefaultSerializer, PingInterval=0, PingTimeout=5000, RequestTimeout=5000, ConnectTimeout=10000, SocketProvider=SystemSocketProvider, RemoteAddresses=[ctrel9012-12/10.176.4.104:9099], SocketOptions{LingerTimeout=0, KeepAliveEnabled=true, TcpDelayEnabled=false}}
      2012-12-27 00:09:44.388/8.290 Oracle Coherence GE 3.7.1.0 <D5> (thread=Thread-9, member=n/a): Connecting Socket to 10.176.4.104:9099
      2012-12-27 00:09:44.388/8.290 Oracle Coherence GE 3.7.1.0 <Info> (thread=Thread-9, member=n/a): Connected Socket to 10.176.4.104:9099
      2012-12-27 00:09:44.402/8.304 Oracle Coherence GE 3.7.1.0 <D6> (thread=MyAppTcpCacheService:TcpInitiator, member=n/a): Opened: TcpConnection(Id=0x0000013BDAFB0DC60AB00468731F416202178388101B3C4DCACF9DEB39E6A33D, Open=true, Member(Id=0, Timestamp=2012-12-27 00:09:44.273, Address=10.176.1.77:0, MachineId=0, Location=site:,machine:ctbr024-12,process:23132, Role=WestgroupMyAppProcessorManager), LocalAddress=10.176.1.77:54989, RemoteAddress=10.176.4.104:9099)
      2012-12-27 00:09:44.424/8.327 Oracle Coherence GE 3.7.1.0 <D6> (thread=MyAppTcpCacheService:TcpInitiator, member=n/a): Opened: Channel(Id=938856961, Open=true, Connection=0x0000013BDAFB0DC60AB00468731F416202178388101B3C4DCACF9DEB39E6A33D)
      2012-12-27 00:09:44.452/8.354 Oracle Coherence GE 3.7.1.0 <D6> (thread=MyAppTcpCacheService:TcpInitiator, member=n/a): Opened: Channel(Id=1243968140, Open=true, Connection=0x0000013BDAFB0DC60AB00468731F416202178388101B3C4DCACF9DEB39E6A33D)
      ERROR Thread-9 com.westgroup.MyApp.dao.lpa.EntityDAO - Unable to Connect to the LPA Cache
      com.tangosol.net.RequestTimeoutException: request timed out after 5000 millis
      at com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:58)
      at com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:1)
      at com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.query(RemoteNamedCache.CDB:29)
      at com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.entrySet(RemoteNamedCache.CDB:6)
      at com.tangosol.util.ConverterCollections$ConverterQueryMap.entrySet(ConverterCollections.java:2507)
      at com.tangosol.util.ConverterCollections$ConverterNamedCache.entrySet(ConverterCollections.java:2799)
      at com.tangosol.coherence.component.net.extend.RemoteNamedCache.entrySet(RemoteNamedCache.CDB:1)
      at com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)

      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      Cache Config file
      <?xml version="1.0"?>
      <cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
           xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config http://xmlns.oracle.com/coherence/coherence-cache-config/1.1/coherence-cache-config.xsd">
           
           <defaults>
                <serializer system-property="tangosol.coherence.serializer" />
                <socket-provider system-property="tangosol.coherence.socketprovider" />
           </defaults>

           <caching-scheme-mapping>
                <cache-mapping>
                     <cache-name>*</cache-name>
                     <scheme-name>MyAppCache</scheme-name>
                     <init-params>
                          <init-param>
                               <param-name>back-size-limit</param-name>
                               <param-value>250MB</param-value>
                          </init-param>
                     </init-params>
                </cache-mapping>

           </caching-scheme-mapping>

           <caching-schemes>
                <!-- Distributed caching scheme. -->
                <distributed-scheme>
                     <scheme-name>MyAppCache</scheme-name>
                     <service-name>HBaseMyAppCache</service-name>

                     <backing-map-scheme>
                          <read-write-backing-map-scheme>
                               <internal-cache-scheme>
                                    <class-scheme>
                                         <class-name>com.tangosol.util.ObservableHashMap</class-name>
                                    </class-scheme>
                               </internal-cache-scheme>
                               <cachestore-scheme>
                                    <class-scheme>
                                         <class-name>com.westgroup.coherence.profile.MyAppCacheProfile</class-name>                                   
                                    </class-scheme>
                               </cachestore-scheme>
                               <read-only>false</read-only>
                               <write-delay-seconds>0</write-delay-seconds>
                          </read-write-backing-map-scheme>
                     </backing-map-scheme>
                     <autostart>true</autostart>
                </distributed-scheme>

                <proxy-scheme>
                     <service-name>BermudaTCPProxyService</service-name>
                     <thread-count>5</thread-count>
                     <acceptor-config>
                          <tcp-acceptor>
                               <local-address>
                                    <address>localhost</address>
                                    <port>9099</port>
                               </local-address>                         
                          </tcp-acceptor>
                     </acceptor-config>
                     <autostart>true</autostart>
                </proxy-scheme>
                
           </caching-schemes>
      </cache-config>

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      Extended Client config

      <?xml version="1.0"?>
      <cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
           xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config http://xmlns.oracle.com/coherence/coherence-cache-config/1.1/coherence-cache-config.xsd">
           <caching-scheme-mapping>
                <cache-mapping>
                     <cache-name>*</cache-name>
                     <scheme-name>MyAppCache</scheme-name>
                </cache-mapping>
           </caching-scheme-mapping>

           <caching-schemes>

                <remote-cache-scheme>
                     <scheme-name>MyAppCache</scheme-name>
                     <service-name>MyAppTcpCacheService</service-name>
                     <initiator-config>
                          <tcp-initiator>
                               <remote-addresses>                              
                                    <socket-address>
                                         <address>ctrel9012-12</address>
                                         <port>9099</port>
                                    </socket-address>
                               </remote-addresses>
                               <connect-timeout>10s</connect-timeout>
                          </tcp-initiator>
                          <outgoing-message-handler>
                               <request-timeout>10s</request-timeout>
                          </outgoing-message-handler>
                     </initiator-config>
                </remote-cache-scheme>

                <remote-invocation-scheme>
                     <scheme-name>extend-invocation</scheme-name>
                     <service-name>MyAppTcpInvocationService</service-name>
                     <initiator-config>
                          <tcp-initiator>
                               <remote-addresses>
                                    
                                    <socket-address>
                                    <address>ctrel9012-12</address>
                                         <port>9099</port>
                                    </socket-address>
                               </remote-addresses>
                               <connect-timeout>10s</connect-timeout>
                          </tcp-initiator>
                          <outgoing-message-handler>
                               <request-timeout>10s</request-timeout>
                          </outgoing-message-handler>
                     </initiator-config>
                </remote-invocation-scheme>

           </caching-schemes>
      </cache-config>

      Edited by: 966505 on Dec 27, 2012 7:29 AM
        • 1. Re: Cache will be blank after RequestTimeoutException
          tmiddlet
          Hi

          A few questions on your post:

          1) When you say "receiving this exception my cache node will become blank and again I have to load my cache objects". Are you meaning that after the error the storage-enabled nodes are losing all their data and you have to reload the cache from your data-store via your read-write-backing maps?Or is this just an issue with the extend client accessing data?

          2) is there an exception raised in the proxy-server nodes? If so can you post this here?

          3) Are your proxy servers, storage-disabled? it is best practise to have storage disabled proxy servers.

          Thanks
          • 2. Re: Cache will be blank after RequestTimeoutException
            969508
            Hi ,

            Please find my comments below.

            1)     Yes all my storage-enabled nodes are losing all their data after encountering this exception.

            2)     Yes , sometime I receive this below exception , after reloading the cache it will be solved. Can you help me why this exception happens intermittently.
            java.lang.RuntimeException: Missing or inaccessible method: java.lang.String.getProfileKey()
                 com.tangosol.util.extractor.ReflectionExtractor.extract(ReflectionExtractor.java:113)
                 com.tangosol.util.extractor.AbstractExtractor.extractFromEntry(AbstractExtractor.java:115)
                 com.tangosol.util.InvocableMapHelper.extractFromEntry(InvocableMapHelper.java:315)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage$BinaryEntry.extract(PartitionedCache.CDB:27)
                 com.tangosol.util.filter.ExtractorFilter.evaluateEntry(ExtractorFilter.java:93)
                 com.tangosol.util.InvocableMapHelper.evaluateEntry(InvocableMapHelper.java:262)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.createQueryResult(PartitionedCache.CDB:84)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.query(PartitionedCache.CDB:72)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onQueryRequest(PartitionedCache.CDB:23)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$QueryRequest.run(PartitionedCache.CDB:1)
                 com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheRequest.onReceived(DistributedCacheRequest.CDB:12)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:34)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
                 com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
                 com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)

            3)     Yes I have used storage-disabled proxy servers.

            And I have one more question, most of the time I will end with below exception, Can you help what will be the root cause for this.

            com.tangosol.net.RequestTimeoutException: request timed out after 5000 millis
            at com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:58)
            at com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:1)
            at com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.query(RemoteNamedCache.CDB:29)
            at com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.entrySet(RemoteNamedCache.CDB:6)
            at com.tangosol.util.ConverterCollections$ConverterQueryMap.entrySet(ConverterCollections.java:2507)
            at com.tangosol.util.ConverterCollections$ConverterNamedCache.entrySet(ConverterCollections.java:2799)
            at com.tangosol.coherence.component.net.extend.RemoteNamedCache.entrySet(RemoteNamedCache.CDB:1)
            at com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)

            Thanks in advance.
            • 3. Re: Cache will be blank after RequestTimeoutException
              tmiddlet
              Seems the RequestTimeoutException is may be caused by the exception you are getting.
              The error is from the ReflectionExtractor.
              java.lang.RuntimeException: Missing or inaccessible method: java.lang.String.getProfileKey()
              It seems you are trying to run a RefectionExtractor on an object which is a String and not your expected POJO with a getProfileKey method.

              Do you have some code that uses new ReflectionExtractor("ProfileKey").

              Thanks
              • 4. Re: Cache will be blank after RequestTimeoutException
                969508
                Hi,

                Thanks for your response.

                Here is the piece of code which is added to update the index key.
                cache.addIndex(new ReflectionExtractor("getProfileKey"), true, null);

                POJO code.

                private String profileKey;

                public String getProfileKey()
                     {
                          return profileKey;
                     }

                     public void setProfileKey(String profileKey)
                     {
                          this.profileKey = profileKey;
                     }

                Here is the piece of code to get all the objects from cache using indexKey.
                public List<LPACacheProfile> getLPAProfile(String profileKey)
                     {
                          List<LPACacheProfile> localProfileList = new ArrayList<LPACacheProfile>();
                          Set localSet = this.cache.entrySet(new EqualsFilter(new ReflectionExtractor("getProfileKey"), profileKey));
                          Iterator iter = localSet.iterator();

                          while(iter.hasNext())
                          {
                               Map.Entry entry = (Map.Entry) iter.next();
                               localProfileList.add((LPACacheProfile)entry.getValue());               
                          }

                          return localProfileList;
                     }

                But this is happening intermittently and not always, sometimes when I query cache it returns me only the string(the key which I added while updating the cache) and sometimes it returns me the expected POJO.

                Does this help or need any other information.
                • 5. Re: Cache will be blank after RequestTimeoutException
                  user738616
                  Hi,
                  But this is happening intermittently and not always, sometimes when I query cache it returns me only the string(the key which I added while updating the cache) and sometimes it returns me the expected POJO.

                  Does this help or need any other information.
                  This is primarily because the POJO is not available in cache either because it is expired or evicted. Remember filters execute on the cached data and does not go to your cacheStore for fetching the data on cache miss.

                  HTH

                  Cheers,
                  NJ
                  • 6. Re: Cache will be blank after RequestTimeoutException
                    tmiddlet
                    The ReflectionExtractor works on the Value part of the Key,Value in the Cache.
                    So by getting that error your Value entry for a particular entry in the cace is a String object rather than a POJO (LPACacheProfile).

                    Are you sure some code is not someone adding a String Value somewhere in the cache instead of LPACacheProfile POJO?
                    You mentioned it is a read through cache and your cachestore-class is com.westgroup.coherence.profile.MyAppCacheProfile. Maybe just double check that this is correctly loading your POJO.

                    As mentioned, remember filters, entry processors, queries only run on cached data.

                    Maybe write some code to loop through all entries of your cache, if not too many, and check what type the entry.getValue() is.

                    Also, i'm assuming the profileKey is not the actually key of a LPACacheProfile entry?

                    Tim