4 Replies Latest reply: Nov 28, 2012 10:22 AM by 975076 RSS

    Aggregator - Cache Server Node high memory usage due to deserialization

    975076
      Hello All,

      We are currently on coherence version 3.5.3, and have the following setup:
      Cache Key : Long ID
      
      Cache Value :
      CacheData {
        Set<Long> refIds1;
        Set<Long> refIds2;
        Set<String> refIds3;
      }
      After loading the cache data, we are indexing the following:
      cache.addIndex( new ReflectionExtractor( "refIds1" ), false, null );
      cache.addIndex( new ReflectionExtractor( "refIds2" ), true, null );
      cache.addIndex( new ReflectionExtractor( "refIds3" ), false, null );
      We had a requirement to gather all refIds2 from cache for the following criteria:
      #1. Cache Key matches
      #2. Any refIds1 matches
      #3. Any refIds3 matches

      We have implemented the following Custom Aggregator:
      CustomAggregator extends AbstractComparableAggregator implements com.tangosol.io.pof.PortableObject {
      
        @Override
        public Object aggregate( Set values )
        {
          Set<Long> resultRefIds2 = new HashSet<Long>();
          for(Entry entry : (Set<InvocableMap.Entry>)values)
          {
            CacheData data = (ProductGroup)entry.getValue() ;
            resultRefIds2.addAll( data.getRefIds2() );
          }
          return resultRefIds2 ;
        }
        
        public Object aggregateResults( Collection results )
        {
          Set<Long> localRefIds2 = new HashSet<Long>();
          if( null != results && results.size() > 0 )
          {
           List<ImmutableArrayList> immutableProductGroupArrayList = new ArrayList<ImmutableArrayList>( results );
           for( int immutableArrayCnt = 0; immutableArrayCnt < immutableProductGroupArrayList.size(); immutableArrayCnt++ )
           {
             localRefIds2.addAll( immutableProductGroupArrayList.get( immutableArrayCnt ) );
           }
          } 
          return localRefIds2;
        }
        
      }
      Following code fetches the results:
      Filter[] filters = new Filter[3];
      filters[0] = new InFilter( new KeyExtractor( IdentityExtractor.INSTANCE ), inputKeys );
      filters[1] = new ContainsAnyFilter( "refIds1", inputRefIds1 );
      filters[2] = new ContainsAnyFilter( "refIds3", inputRefIds3 );
      Filter filter = new AllFilter( filters ) ;
      CustomAggregator aggregator = new CustomAggregator();
      Set<Long> refIds2 = (Set<Long>)cache.aggregate( filter, aggregator );
      While we are getting the results as expected, when the criteria returns all cached entries from the cache to the aggregator, we see a lot of memory usage due to deserialization of the cached data.

      Snippet from Aggregator:
          for(Entry entry : (Set<InvocableMap.Entry>)values)
          {
            CacheData data = (ProductGroup)entry.getValue() ;
            resultRefIds2.addAll( data.getRefIds2() );
          }
      #1. Is there a better way of achieving the same results, without having to deserialize the data in the cache.
      #2. Since we are already indexing the data we are aggregating, is there a way to use the indexed data while aggregating, instead of extracting from the cache.

      Thanks in Advance
      Avinash Gadapa

      Edited by: 972073 on Nov 19, 2012 12:43 PM