10 Replies Latest reply: Jan 14, 2010 6:57 AM by jtahlborn RSS

    weak reference but still out of memory

    843798
      written a small program to understand the refrence api. I was thinking that the below code would never throw OOM but i get the OOM exception as shown below. Can someone pls exlpain why this might be happening. NOTE: same is happening when I use SoftReference.
      public static void weak(Map<WeakReference<Long>,Double> m) 
      {
         for(long i=0;;i++)
         {
           System.out.println(i);
           m.put(new WeakReference<Long>(new Long(i)), new Double(i));
          }
      }
      785338
      785339
      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
           at java.util.HashMap.resize(HashMap.java:508)
           at java.util.HashMap.addEntry(HashMap.java:799)
           at java.util.HashMap.put(HashMap.java:431)
           at OOMTest.weak(OOMTest.java:58)
           at OOMTest.main(OOMTest.java:26)
      785340
      785341
      785342
      785343
        • 1. Re: weak reference but still out of memory
          843798
          ok..after googlinr around...i thought the issue is not with soft/weak references but the way memory is re-allocated once the HasMap reaches its initial capacity. the OOM can be delayed (without increasing the jvm heap) by specifying the initial capacity (based on the requirements) and also controllng the load factor to do incremental resize(0.25) rather than re-allocating a new array with double the size
          Map<WeakReference<Long>,WeakReference<Double>> weak=new HashMap<WeakReference<Long>,WeakReference<Double>>(500000,1.0f);
          
          for(long l=0;l<=Long.MAX_VALUE;l+=100000)
          {
               weak(weak,100000);
          }
          
          public static void weak(Map<WeakReference<Long>,WeakReference<Double>> m , long limit) 
          {
               for(long i=0;;i++)
               {
                    m.put(new WeakReference<Long>(new Long(i)), new WeakReference<Double>(new Double(i)));
          
                    long heapFreeSize = Runtime.getRuntime().freeMemory(); 
                    if(i%100000==0)
                    {
                         System.out.println(i);
                         System.out.println(heapFreeSize/131072 +"MB");
                         System.out.println();
                    }
               }
          }
          0
          21MB

          100000
          10MB

          200000
          86MB

          300000
          144MB

          400000
          65MB

          500000
          189MB

          600000
          78MB

          700000
          57MB

          800000
          21MB

          Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
               at OOMTest.weak(OOMTest.java:73)
               at OOMTest.main(OOMTest.java:40)
          • 2. Re: weak reference but still out of memory
            jtahlborn
            because you have no code which removes expired references from your map. the references themselves (as well as the Double values) take up memory and will eventually cause OOME. if you look at the WeakHashMap impl in the jdk you will see that the weak references are created with a ReferenceQueue. the internal impl of WeakHashMap periodically polls this queue and removes entries from the map as they expire.
            • 3. Re: weak reference but still out of memory
              843798
              thx for the reply..i made changes as below based on ur input...however i never see WeakReference's expiring because the "poll" method is always returning null
              ReferenceQueue<Long> refQ = new ReferenceQueue<Long>();
              
              for (long l = 0; l <= Long.MAX_VALUE; l += 100000)
              {
                   weak(weak, refQ, 100000);
                   Reference<? extends Long> poll = refQ.poll();
                   if (poll != null)
                   {
                        System.out.println("removing " + poll.get());
                        poll.clear();
                   }
              }
              
              
              public static void weak(Map<WeakReference<Long>, WeakReference<Double>> m, ReferenceQueue<Long> refQ, long limit)
              {
                   for (long i = 0;; i++)
                   {
                        m.put(new WeakReference<Long>(new Long(i), refQ), new WeakReference<Double>(new Double(i)));
                        long heapFreeSize = Runtime.getRuntime().freeMemory();
              
                   }
              }
              • 4. Re: weak reference but still out of memory
                jtahlborn
                let me start over, i was not seeing the code clearly.

                the reference will always be null when you get the reference from the queue (already cleared).

                yet another rewrite of this comment. your weak() method will never return, so your polling code is never executed.

                Edited by: jtahlborn on Jan 13, 2010 3:58 PM
                • 5. Re: weak reference but still out of memory
                  843798
                  where do u think the problem might be...i thought this is simple ;-). From the code I expected all the key/value (Long, Double) along with the WeakReference's to be eligible for GC for every 100,000 records as I am coming out of the weak() method after iterating thru 100,000...but seems like my understanding is wrong
                  • 6. Re: weak reference but still out of memory
                    843798
                    k i corrected my for loop in the weak() method to run only till i<=100,000 so that it will return for every 100,000 iterations...and changed the polling code as below
                    for (long l = 0; l <= Long.MAX_VALUE; l += 100000)
                              {
                                   weak(weak, keyRefQ, valueRefQ, 100000);
                              System.out.println("Total PUTs so far = " + l);
                    
                                   System.out.print("Clearing KEY weak refs. total cleared = ");
                                   poll(keyRefQ);
                                   System.out.print("Clearing VALUE weak refs. total cleared = ");
                                   poll(valueRefQ);
                              }
                    
                    private static void poll(ReferenceQueue<?> keyOrValueRefQ)
                    {
                         Reference<?> poll = keyRefQ.poll();
                              int i = 0;
                              while (poll != null)
                              {
                                   // System.out.println(poll);
                                   poll.clear();
                                   poll = keyRefQ.poll();
                                   i++;
                              }
                              System.out.println(i);}
                    The log now is

                    amount of free memory within the heap in bytes = 1917280
                    100000
                    30MB

                    Total PUTs so far = 0
                    Clearing KEY weak refs. total cleared = 77965
                    Clearing VALUE weak refs. total cleared = 77963
                    100000
                    24MB

                    Total PUTs so far = 100000
                    Clearing KEY weak refs. total cleared = 56643
                    Clearing VALUE weak refs. total cleared = 56643
                    100000
                    53MB

                    Total PUTs so far = 200000
                    Clearing KEY weak refs. total cleared = 86876
                    Clearing VALUE weak refs. total cleared = 86877
                    100000
                    157MB

                    Total PUTs so far = 300000
                    Clearing KEY weak refs. total cleared = 145480
                    Clearing VALUE weak refs. total cleared = 145481
                    100000
                    77MB

                    Total PUTs so far = 400000
                    Clearing KEY weak refs. total cleared = 2
                    Clearing VALUE weak refs. total cleared = 0
                    100000
                    129MB

                    Total PUTs so far = 500000
                    Clearing KEY weak refs. total cleared = 166919
                    Clearing VALUE weak refs. total cleared = 166921
                    100000
                    50MB

                    Total PUTs so far = 600000
                    Clearing KEY weak refs. total cleared = 2
                    Clearing VALUE weak refs. total cleared = 0
                    100000
                    6MB

                    Total PUTs so far = 700000
                    Clearing KEY weak refs. total cleared = 241885
                    Clearing VALUE weak refs. total cleared = 241887
                    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
                         at OOMTest.weak(OOMTest.java:117)
                         at OOMTest.main(OOMTest.java:35)
                    • 7. Re: weak reference but still out of memory
                      jtahlborn
                      please reread my original reply, specifically the part which says +"and removes entries from the map as they expire"+.
                      • 8. Re: weak reference but still out of memory
                        843798
                        thx..now i am able to put pieces together..the advantage of reference api (if not using WeakHasMap) is that it tells u when the only refrence left is the weak/soft reference to an object and then we have to also remove the strong ref that we have to the weak-ref for it to be GC'ed
                        private static int poll(ReferenceQueue<?> keyRefQ, Map<WeakReference<Long>, WeakReference<Double>> weak)
                             {
                                  Reference<?> poll = keyRefQ.poll();
                                  int i = 0;
                                  while (poll != null)
                                  {
                                       if (poll.get() == null)
                                       {
                                            weak.remove(poll);
                                       }
                        
                                       poll = keyRefQ.poll();
                                       i++;
                                  }
                                  return i;
                        
                             }
                        • 9. Re: weak reference but still out of memory
                          843798
                          waiting to confirm my understanding
                          • 10. Re: weak reference but still out of memory
                            jtahlborn
                            yes, that code is essentially "correct". (i believe i already mentioned above that poll.get() will always be null at this point, not need to check it).

                            one further thing to note, if you generate refs fast enough, you may still manage to cause an OOME as they take longer to garbage collect than normal objects. as such, your artificial test may still manage to overwhelm the gc and cause an OOME.

                            Edited by: jtahlborn on Jan 14, 2010 7:56 AM