This discussion is archived
6 Replies Latest reply: Jan 31, 2011 4:12 AM by 802316 RSS

Java memory management for Big Lists

817160 Newbie
Currently Being Moderated
Hello.

i have a big Linked List, this list is dynamic and can grow up very fast. I want to protect this from blowing up java heap space. Is it possbile to use:
Runtime.getRuntime().maxMemory() to get maximum java available memory and then Runtime.getRuntime().totalMemory() to get the actual used memory by my java program and if its at 90% of memory usage use a Semaphore to stop add more entrys. Is this a good solution ??
  • 1. Re: Java memory management for Big Lists
    796440 Guru
    Currently Being Moderated
    Silva wrote:
    Hello.

    i have a big Linked List, this list is dynamic and can grow up very fast. I want to protect this from blowing up java heap space. Is it possbile to use:
    Runtime.getRuntime().maxMemory() to get maximum java available memory and then Runtime.getRuntime().totalMemory() to get the actual used memory by my java program and if its at 90% of memory usage use a Semaphore to stop add more entrys.
    Try it and find out.
    Is this a good solution ??
    Impossible to say. I don't know your requirements. What are the consequences for your program if it wants to add something to the list but then is denied?

    A better approach might be to determine ahead of time a requirement for how many connections, sessions, intput items--whatever it is that makes the list grow--you need to be able to handle. Run tests that observe how memory usage grows as this input factor grows, extrapolate to how much memory you'll need to be able to handle that maximum requirement, and set your heap size somewhat larger--say, 20%-100% larger--at startup.
  • 2. Re: Java memory management for Big Lists
    817160 Newbie
    Currently Being Moderated
    If the list is full and any method tries to add to the list, i want it to wait for more memory.

    Im going to test the following:

    Do a addToList Method
    - checks for available memory - if its above 90% memory utilization a semaphore will block the add and waits

    Do a removeFromList Method
    - Removes a item from the list and if the semaphore is blocked (memory full ) it will unlock the semaphore


    The Runtime.getRuntime().maxMemory() gives me the maximum memory available to my JVM ? So it is the maximum memory my program can get? Is this dynamic can change over time ?
  • 3. Re: Java memory management for Big Lists
    EJP Guru
    Currently Being Moderated
    This doesn't make sense. It is very unlikely that actually adding to the list is going to run out of memory. It is many times as probable that allocating the object you are going to add to the list will do that. Assuming you are using a LinkedList, which you certainly should be in this kind of situation. Or maybe a database.
  • 4. Re: Java memory management for Big Lists
    817160 Newbie
    Currently Being Moderated
    But the thread responsbible to add the object to the list is going to stop, and runs again when more memory is available for the list. So no more object will be allocated.
    The List is going to be consumed by another thread. Its non finishing consumer thread.
    This way is possible to stop the producers thread to wait till more memory is available.
    Lets say, if 80% of the memory is full, producers stop working.

    Bad idea ?
  • 5. Re: Java memory management for Big Lists
    EJP Guru
    Currently Being Moderated
    My point remains valid. It won't be the list itself that blows memory, it will be the objects in the list. So I agree you have to stop the producers from producing those objects. It might make more sense just to bound the number of elements in the list, i.e. a bounded queue. Then when it gets too big the producers will st doing anything at all, including creating new objects, until it gets back below the limit.

    So see java.uti..concurrent and save yourself a lot of effort.
  • 6. Re: Java memory management for Big Lists
    802316 Pro
    Currently Being Moderated
    It is very rare that you gain anything by letting the producer get too far ahead of the consumer.

    e.g. Ideally the objects created will stay in eden space only. If the objects live too long they will be copied to survivor/tenured space and can add more overhead than they are worth.

    The simplest way to implement what you want is to use a bounded Queue with a modest maximum size (no where near your memory limit) e.g. ArrayBlockingQueue or LinkedBlockingQueue.

    The producer can offer entries to the queue and block or timeout if the queue is full.

    Try a maximum size of 1, 10, 100 etc and you may find that 1 to 100 gives you near optimal performance.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points