7 Replies Latest reply on Jun 7, 2009 9:04 AM by robvarga

    Secondary index over head

    alexey.ragozin
    alexey.ragozin May 21, 2009 11:13 AM
      I'm analyzing using of memory by coherence caches in one of our projects.

      We have secondary index by string filed of object.
      Hallo,
      I single JVM we have 430k objects (about 20k unique values of indexed field)

      jmap shows following 
       num     #instances         #bytes  class name
----------------------------------------------
   1:        865293      110687424  [B
   2:        430791       27570624  com.tangosol.net.cache.LocalCache$Entry
   3:          5762       25606584  [I
   4:        881679       21160296  com.tangosol.util.SafeHashMap$Entry
   5:        861588       20678112  com.tangosol.util.Binary
   6:        436514       14216576  [C
   7:        436178       10468272  java.lang.String
   8:         19998        8863424  [Lcom.tangosol.util.SafeHashMap$Entry;
      +it is same case as in Using custom backing map for distributed scheme

      It looks like index is using two maps, first has about 20k (String -> Set) pairs second about 430k (Binary -> String) pairs.

      The thing bothering me is - while we have 20k distinct String in system, number of instances in heap ~ 430k. I clearly understand how it happened, but there is definitely room for improvement here.
      Also hash map and hash set implementations are wasting too much memory (looks at average size of com.tangosol.util.SafeHashMap$Entry[], and memory consumed by SafeHashMap$Entry).

      Is it possible to plug own index implementation to coherence cache somehow or at least use custom provided map and set classes for index?
      • 1491 Views
      • Tags:
        • 1. Re: Secondary index over head
          robvarga
          robvarga May 21, 2009 12:44 PM (in response to alexey.ragozin)
          Hi Alexey,

          if you internalized the String instances in the value extractor which is used when creating the index, then the number of String instances in the forward map (Binary->String) would be 20k. You woudn't need to change the filtering logic.

          Of course this comes at a small performance cost when updating entries and some memory overhead of maintaining the set of internalized String instances in the permanent generation space (and which can possibly cause OutOfMemoryError-s).

          Best regards,

          Robert
          • 2. Re: Secondary index over head
            alexey.ragozin
            alexey.ragozin May 21, 2009 1:13 PM (in response to robvarga)
            Hi Robert,

            Yes I'm aware of such technics. String.intern() is not an option due to performance reason (at least on Sun JVM, it has O(ln(n)) lookup time and O(n) insertion time ). Of cause I can use WeakHashMap to organize own string table.

            But I already have this table in first map (String -> Binary), in former project we have used handmade Map implementation with method internKey() to avoid duplication of data.
            I'm looking for a way to reuse our old and proven techniques with coherence caches.

            Thank you,
            Alexey
            • 3. Re: Secondary index over head
              robvarga
              robvarga May 21, 2009 1:44 PM (in response to alexey.ragozin)
              Hi Alexey,
              alexey.ragozin wrote:
              ...

              But I already have this table in first map (String -> Binary), in former project we have used handmade Map implementation with method internKey() to avoid duplication of data.
              I'm looking for a way to reuse our old and proven techniques with coherence caches.

              Thank you,
              Alexey
              Unfortunately there is no way the extractor could get hold of the already existing key reference in the reverse index. I agree that the extracted value reference in the reverse index entry (the key from that entry) should be reused by Coherence as the forward index value if the reverse index entry for the same extracted value exists, but apparently it is not.

              Try to submit an enhancement request for this (and please share the ticket number for it so we can also look for it in the release notes).

              Best regards,

              Robert
              1 person found this helpful
              • 4. Re: Secondary index over head
                339424
                339424 May 22, 2009 4:46 PM (in response to robvarga)
                Hi Alexey, Robert,

                We've filed COH-2350 to track this enhancement req.

                thanks,
                -Rob
                1 person found this helpful
                • 5. Re: Secondary index over head
                  robvarga
                  robvarga Jun 6, 2009 5:15 PM (in response to 339424)
                  Hi Rob,

                  could you also tell us which release is it scheduled for (even if only tentatively)?

                  Thanks and best regards,

                  Robert
                  • 6. Re: Secondary index over head
                    heneman
                    heneman Jun 7, 2009 12:49 AM (in response to robvarga)
                    Hi Robert,

                    You may want to review the [3.5 Pre-Release Notes|http://coherence.oracle.com/display/COH35UG/Coherence+3.5+Release+Notes].

                    Regards,

                    Harv
                    • 7. Re: Secondary index over head
                      robvarga
                      robvarga Jun 7, 2009 9:04 AM (in response to heneman)
                      Wow, that was fast :-)

                      Thanks and best regards,

                      Robert