This discussion is archived
10 Replies Latest reply: Oct 25, 2012 1:38 AM by 970366 RSS

How to hide Objects form the garbage collector

970366 Newbie
Currently Being Moderated
Hello java folks,

I want to create a large tree to display some data. The problem I have is that during the creation of the tree the garbage collector runs wild. It tires to collect the newly created objects which will be used for a long period of time or are deleted all at the same time. A sample code which reproduces the problem is below:
import javax.swing.tree.DefaultMutableTreeNode;
import javax.swing.tree.TreeNode;

public class Test {
  public static void main(String[] args) {
    int size = 9675536;
    TreeNode[] nodes = new DefaultMutableTreeNode[size];
    for (int i = 0; i < nodes.length; ++i) {
      nodes[i] = new DefaultMutableTreeNode();
    }
  }
}
The output of the code code with the arguments -Xmx2048m -verbose:gc is the following:
[GC 57891K->57003K(77248K), 0.0828760 secs]
[Full GC 57003K->56898K(131520K), 0.3528290 secs]
[GC 76994K->76094K(137920K), 0.1042000 secs]
[GC 102590K->102558K(150656K), 0.1313760 secs]
[Full GC 102558K->102283K(233344K), 0.9590480 secs]
[GC 141515K->141291K(233344K), 0.1803370 secs]
[GC 181483K->181667K(234304K), 0.1970770 secs]
[Full GC 181667K->181300K(355648K), 1.6650550 secs]
[GC 221492K->221652K(418944K), 0.1781270 secs]
[GC 279124K->279300K(421440K), 0.4649640 secs]
[GC 336772K->336900K(459200K), 0.5072510 secs]
[Full GC 336900K->336432K(631744K), 2.5436460 secs]
During the program run all the processors are active with garbage collections and it takes for about 30 seconds to complete the run. So we have a lot of overhead from the garbage collection.
My question is now how I can tell the garbage collector to not collect the TreeNodes? (Google search has not helped and the Reference is the wrong direction.)

The option -Xms is also not a solution because that only delays the problem.

Thanks for any replies

Max

Edit:

I provided some additional Re: How to hide Objects form the garbage collector and statistics in response to [rp0428|https://forums.oracle.com/forums/message.jspa?messageID=10655274#10655274]

Edited by: Max 1338 on 25-Oct-2012 01:45
  • 1. Re: How to hide Objects form the garbage collector
    sabre150 Expert
    Currently Being Moderated
    Sorry but I don't see where the GC is involved. None of the 9675536 or so DefaultMutableTreeNode objects that you create become eligible for GC until the main() method exits so where is the GC involved ?
  • 2. Re: How to hide Objects form the garbage collector
    800268 Expert
    Currently Being Moderated
    967363 wrote:
    I want to create a large tree to display some data.
    It's unlikely your users want to look at ten million tree nodes at the same time (for comparison, my current screen resolution is just over 1.7 million pixels).

    -Xms is the solution for the initial load but as you say you will be reloading the nodes you'll simple hit the GC at some point.

    Re-examine why you want to load that data all upfront.
  • 3. Re: How to hide Objects form the garbage collector
    EJP Guru
    Currently Being Moderated
    It tries to collect the newly created objects which will be used for a long period of time or are deleted all at the same time.
    No it doesn't. It identifies and collects objects that aren't reachable via a chain of live references from a running thread. The objects in your program don't qualify.
    My question is now how I can tell the garbage collector to not collect the TreeNodes?
    It doesn't collect the TreeNodes. If it did your program would fail.

    Why exactly do you think you need a tree with 96 million elements?
  • 4. Re: How to hide Objects form the garbage collector
    970366 Newbie
    Currently Being Moderated
    The gc is involved because the starting heap space is used up after some time. It tries to free some space for the new objects but it doesn't find any so it increases the heapspace. Every garbage collection uses up more time and the full gc block the whole system.
  • 5. Re: How to hide Objects form the garbage collector
    970366 Newbie
    Currently Being Moderated
    I need all the elements to calculate some other values out of the data for each node. This calculated data is then displayed in the tree. Also the data layout of the files I have to parse would it make very difficult to read the nodes on the fly.

    I could write a new file format but that would defeat the purpose of the program and the initial problem that I once need to represent all the nodes in a tree which gives me the gc problems is still there.

    >
    No it doesn't. It identifies and collects objects that aren't reachable via a chain of live references from a running thread. The objects in your program don't qualify.
    >

    Yes you are right my objects don't qualify. But that doesn't change the fact that the gc tries to collect them, which takes a lot of time.
  • 6. Re: How to hide Objects form the garbage collector
    sabre150 Expert
    Currently Being Moderated
    967363 wrote:
    The gc is involved because the starting heap space is used up after some time. It tries to free some space for the new objects but it doesn't find any so it increases the heapspace. Every garbage collection uses up more time and the full gc block the whole system.
    I don't see this using 1.7.0_09 on Ubuntu 10.04. Your whole program takes under 12 seconds to run including compilation time! I have a very old (5 years) 32 bit desktop with 4 GByte of RAM and 2 processors.

    Edited by: sabre150 on Oct 24, 2012 12:59 PM

    Correction :
    Profiling the application in Netbeans shows some GC activity but no more than 10% of a single CPU is being used.
  • 7. Re: How to hide Objects form the garbage collector
    970366 Newbie
    Currently Being Moderated
    Thanks for the testing. Maybe I exagerated a little bit with the 30 seconds. I apologize for that. But if you change the data type to an integral type like int:
    public class Test {
      public static void main(String[] args) {
        int size = 9675536;
        int[] ints = new int[size];
        for (int i = 0; i < ints.length; ++i) {
          ints[i] = i;
        }
      }
    }
    You will see that this loop will execute in a few milliseconds.

    You will also notice that if you set the option -Xms2000m for the test case with the TreeNodes that this test case will actually also finish within 200 milliseconds. Compared to the 5 seconds without that option(On my computer). But that is only true for the test case. If I run my real application with the -Xms2000m option the gc runs will become less but the time of the runs will increase. So I won nothing. The other drawback of the Xms option is that the program uses a lot more mermory than it should.

    The ideal solution would be that I can create the TreeNodes in such a way that the gc ignores them. The rest of the program would run as usual.
  • 8. Re: How to hide Objects form the garbage collector
    aksarben Journeyer
    Currently Being Moderated
    No, your ideal solution is either:

    1. Run this on a mainframe which has enough memory to handle all you want to do, or

    2. Re-examine your basic design to reduce wasteful use of resources.
  • 9. Re: How to hide Objects form the garbage collector
    rp0428 Guru
    Currently Being Moderated
    >
    I want to create a large tree to display some data. The problem I have is that during the creation of the tree the garbage collector runs wild. It tires to collect the newly created objects which will be used for a long period of time or are deleted all at the same time. A sample code which reproduces the problem is below:
    import javax.swing.tree.DefaultMutableTreeNode;
    import javax.swing.tree.TreeNode;
     
    public class Test {
      public static void main(String[] args) {
        int size = 9675536;
        TreeNode[] nodes = new DefaultMutableTreeNode[size];
        for (int i = 0; i < nodes.length; ++i) {
          nodes[i] = new DefaultMutableTreeNode();
        }
      }
    }
    . . .
    During the program run all the processors are active with garbage collections and it takes for about 30 seconds to complete the run. So we have a lot of overhead from the garbage collection.
    My question is now how I can tell the garbage collector to not collect the TreeNodes? (Google search has not helped
    >
    No TreeNodes are being collected for that code since you are maintaining a reference to all of them in the array. So that code doesn't demonstrate a 'problem' at all.

    To the contrary, the gc output you show indicates that almost nothing is being collected since the 'after' size is not much different than the before 'size' shown.
    >
    An object is considered garbage when it can no longer be reached from any pointer in the running program. The most straightforward garbage collection algorithms simply iterate over every reachable object. Any objects left over are then considered garbage. The time this approach takes is proportional to the number of live objects, which is prohibitive for large applications maintaining lots of live data.
    >
    You can't disable the garbage collector but you can tune it to better match your use case.
    I didn't have any problem searching for information. Using 'java tuning garbage collection' as the search term the very first page produced links to Oracle articles and documents about VM garbage collection tuning.

    No specific help can be provided because you haven't provided

    1. your platform and version and number of processors
    2. your Java version
    3. the VM (client or server) you are using
    4. which garbage collector you are using
    5. the command line arguments that are related to memory and generation sizing and garbage collection.

    See the Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning doc at
    http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html

    There are three different collectors available (serial, parallel, concurrent) so maybe you should try using a different one.
    See section 5 'Available Collectors'.
    >
    Since collections occur when generations fill up, throughput is inversely proportional to the amount of memory available. Total available memory is the most important factor affecting garbage collection performance.
    >
    So your solution is to size the generations properly.
    Section 4 'Sizing the Generations' will give you the information you need to determine the best sizing for the eden, survivor and permanent generations to minimize the collections.

    And the proper choice of client vs. server VM and using the correct collector will let you minimize the collection activity.
  • 10. Re: How to hide Objects form the garbage collector
    970366 Newbie
    Currently Being Moderated
    I read the article and made some test I will present here but for all I can tell in my application the assumption that the young generation consists of many young dying objects and only a few tenured objects does not hold. This only confirms the tests I have done before opening this thread.

    My java version and machine is:
    Mac Book Pro 8Gb Ram Intel Core i7 Mac OS X 10.7.5
    
    java version "1.7.0_06"
    Java(TM) SE Runtime Environment (build 1.7.0_06-b24)
    Java HotSpot(TM) 64-Bit Server VM (build 23.2-b09, mixed mode)
    For the test application I tried the 3 different gc with the default options and the additional settings -Xmx2000m or -Xms2000m. The results are in milliseconds:
    java -verbose:gc -XX:+<GC> [-Xms2000m] [-Xmx2000m] Test
    
    (values in ms)      default        Xmx2000m         Xms2000m
    UseSerialGC           2207            2206               201
    UseParallelGC         4849            4780               194
    UseConcMarkSweepGC     924             973               199
    From the values one would expect that setting the -Xms option would solve the problem. That is only true for the testcase as I have mentioned earlier in this thread. In the real world application setting Xms only delays the gc which then takes very long. The second best option from the table is the concurrent gc. I tried that gc on my application. The result is that it performs better than the other gc's but it also introduces a large overhead. So the overall performance of the application is still very bad. The best under these bad results I got with the following options:
    java -verbose:gc -XX:+UseConcMarkSweepGC -Xmx2000m -XX:NewRatio=10
    The idea behind that was to minimise the size of the young generation so that the gc will move the objects to the tenured generation more quickly. This improved performance a little bit but overall it is still pretty bad.

    I have searched for other options to maybe tune the gc for my object lifespan but so far I have not found anything useful. I also believe that this problem can not be solved by gc tuning. Which is why I have formulated my question as it is.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points