This discussion is archived
4 Replies Latest reply: Nov 28, 2012 8:22 AM by 975076 RSS

Aggregator - Cache Server Node high memory usage due to deserialization

975076 Newbie
Currently Being Moderated
Hello All,

We are currently on coherence version 3.5.3, and have the following setup:
Cache Key : Long ID

Cache Value :
CacheData {
  Set<Long> refIds1;
  Set<Long> refIds2;
  Set<String> refIds3;
}
After loading the cache data, we are indexing the following:
cache.addIndex( new ReflectionExtractor( "refIds1" ), false, null );
cache.addIndex( new ReflectionExtractor( "refIds2" ), true, null );
cache.addIndex( new ReflectionExtractor( "refIds3" ), false, null );
We had a requirement to gather all refIds2 from cache for the following criteria:
#1. Cache Key matches
#2. Any refIds1 matches
#3. Any refIds3 matches

We have implemented the following Custom Aggregator:
CustomAggregator extends AbstractComparableAggregator implements com.tangosol.io.pof.PortableObject {

  @Override
  public Object aggregate( Set values )
  {
    Set<Long> resultRefIds2 = new HashSet<Long>();
    for(Entry entry : (Set<InvocableMap.Entry>)values)
    {
      CacheData data = (ProductGroup)entry.getValue() ;
      resultRefIds2.addAll( data.getRefIds2() );
    }
    return resultRefIds2 ;
  }
  
  public Object aggregateResults( Collection results )
  {
    Set<Long> localRefIds2 = new HashSet<Long>();
    if( null != results && results.size() > 0 )
    {
     List<ImmutableArrayList> immutableProductGroupArrayList = new ArrayList<ImmutableArrayList>( results );
     for( int immutableArrayCnt = 0; immutableArrayCnt < immutableProductGroupArrayList.size(); immutableArrayCnt++ )
     {
       localRefIds2.addAll( immutableProductGroupArrayList.get( immutableArrayCnt ) );
     }
    } 
    return localRefIds2;
  }
  
}
Following code fetches the results:
Filter[] filters = new Filter[3];
filters[0] = new InFilter( new KeyExtractor( IdentityExtractor.INSTANCE ), inputKeys );
filters[1] = new ContainsAnyFilter( "refIds1", inputRefIds1 );
filters[2] = new ContainsAnyFilter( "refIds3", inputRefIds3 );
Filter filter = new AllFilter( filters ) ;
CustomAggregator aggregator = new CustomAggregator();
Set<Long> refIds2 = (Set<Long>)cache.aggregate( filter, aggregator );
While we are getting the results as expected, when the criteria returns all cached entries from the cache to the aggregator, we see a lot of memory usage due to deserialization of the cached data.

Snippet from Aggregator:
    for(Entry entry : (Set<InvocableMap.Entry>)values)
    {
      CacheData data = (ProductGroup)entry.getValue() ;
      resultRefIds2.addAll( data.getRefIds2() );
    }
#1. Is there a better way of achieving the same results, without having to deserialize the data in the cache.
#2. Since we are already indexing the data we are aggregating, is there a way to use the indexed data while aggregating, instead of extracting from the cache.

Thanks in Advance
Avinash Gadapa

Edited by: 972073 on Nov 19, 2012 12:43 PM

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points