This discussion is archived
1 2 3 4 Previous Next 57 Replies Latest reply: Jan 13, 2006 1:56 PM by 807590 Go to original post RSS
  • 15. Re: How to catch OutOfMemoryError?
    796440 Guru
    Currently Being Moderated
    You can catch it just like you would any other
    exception,
    Throwable.
    exception, lowercase "e".
    My point was that it is not a subclass of Exception.
    But it appears we failed anyway...
    I know. But my point was that there's a difference between exception and Exception, and I was using the former. :-)
  • 16. Re: How to catch OutOfMemoryError?
    807590 Newbie
    Currently Being Moderated
    Is it possible there might be a method which could tell you how much memory is left available - this might help?
    Ross
  • 17. Re: How to catch OutOfMemoryError?
    807590 Newbie
    Currently Being Moderated
    Runtime.getRuntime().freeMemory() will give you the current free memory in the VM.

    http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Runtime.html
  • 18. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    The main problem with the current way to handle memory is that:

    1. In a threaded program, runtime.getFreeMemory() is almost meaningless since any other thread may create objects later on, so whatever your code assumes is free may no longer be free

    2. The OutOfMemoryError may be thrown elsewhere, so there is no practical way to catch it if your goal is to recover from it.

    There should be a way to tell the JVM to call a catcher before throwing an OOME and then retry. The catcher may take some action like liberate some buffer memory and/or give feedback to the rest of the program to use less memory.

    In most systems I've worked in, memory is used to speed things up, so a program may try to use as much memory as is available for buffers or queues, but that seems almost impossible to do properly in Java.

    Ideas?
  • 19. Re: How to catch OutOfMemoryError?
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    . In a threaded program, runtime.getFreeMemory() is almost meaningless since any other thread may create objects later on, so whatever your code assumes is free may no longer be free
    Might be meaningless to you but it isn't to me.

    If you don't know what your application is doing then it is pretty pointless to try to do anything to manage the memory.

    If you do know then understanding what can occur it part of the process of creating something different.
    2. The OutOfMemoryError may be thrown elsewhere, so there is no practical way to catch it if your goal is to recover from it.
    I can only suppose you are still talking about threads.

    Again you need to understand what your application is doing.
    There should be a way to tell the JVM to call a catcher before throwing an OOME and then retry.
    Huh? You can catch the OOME and then doing anything you like.
    The catcher may take some action like liberate some buffer memory and/or give feedback to the rest of the program to use less memory.
    Yes. Although using some of the classes in the Java API that already exist to support exactly this sort of functionality would seem like a better idea to me.
    In most systems I've worked in, memory is used to speed things up, so a program may try to use as much memory as is available for buffers or queues, but that seems almost impossible to do properly in Java.
    Hmmm...and yet I have done it in both java and C#.

    Although I have certainly never felt the need to try to use all of the remaining available memory. I can't suppose that that would ever work for anything that I would be working on.
    Ideas?
    1. Create a new thread rather than appending to an old one.
    2. Ask a question rather than assuming that no answer exists.
  • 20. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    jschell wrote:
    There should be a way to tell the JVM to call a catcher before throwing an OOME and then retry.
    Huh? You can catch the OOME and then doing anything you like.
    I think he was suggesting a user-defined piece of code that would be called on an OOME, where you would free up memory, but after execution, your program would resume without having disrupted the program stack. That's something you can't do with a try/catch block.
  • 21. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    endasil wrote:
    jschell wrote:
    There should be a way to tell the JVM to call a catcher before throwing an OOME and then retry.
    Huh? You can catch the OOME and then doing anything you like.
    I think he was suggesting a user-defined piece of code that would be called on an OOME, where you would free up memory, but after execution, your program would resume without having disrupted the program stack. That's something you can't do with a try/catch block.
    Yes, thanks, that is exactly what I meant. The point is that the available methods may not work when used in the context of multiple threads, because either you can't rely on the result of getFreeMemory() or by the time an OOME is thrown, the stack is lost and you can't resume your program.
  • 22. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    Again you need to understand what your application is doing.
    Understanding the application does not avoid the fact that another thread my be creating objects (because it has to) and therefore I can't rely on the result of getFreeMemory(). To make that value reliable I'd have to synchronize everything to the point of making the multi-threaded approach useless.
    In most systems I've worked in, memory is used to speed things up, so a program may try to use as much memory as is available for buffers or queues, but that seems almost impossible to do properly in Java.
    Hmmm...and yet I have done it in both java and C#.

    Although I have certainly never felt the need to try to use all of the remaining available memory. I can't suppose that that would ever work for anything that I would be working on.
    Anytime you can substitute slow disk access with fast memory access to get the most out of the server. For example sorting a very large file, where you sort in memory and then merge the resulting blocks. The more you can sort in memory, the faster. So how much memory do you give it? Being conservative is almost sure to avoid the OOME. That "almost" is what bothers me.

    Or a document processing server that needs to do some housekeeping every once in a while and uses a buffer to keep receiving documents in the meantime. There is no way to know how long the housekeeping will take or how many documents will arrive at the time, and the objective is to minimize the chance that the server has to reply "try later", So the queue has to grow dynamically or be as large as possible. There is no way to know when the queue is as large as will fit in memory without causing the rest of the system to throw OOME, and there is no way to catch that OOME before it breaks the application stack in order to issue the "try later" reply and wait until the queue is ready to resume receiving documents.
    Ideas?
    1. Create a new thread rather than appending to an old one.
    Sorry, didn't know there was a problem with continuing an old thread if the subject was still appropriate. Won't do it again, but I don't think it makes sense to switch now, it would create two threads on the same topic.
    2. Ask a question rather than assuming that no answer exists.
    Sorry again, I thought I was asking a question. So here is the question:

    What else is there besides getFreeMemory() and catch(OutOfMemoryError), and that has none of the problems these two approaches have? Not using threads is not an option. Synchronizing them to avoid memory fluctuations is not an option. Having the JVM crash is not an option. Catching the OOME after the fact and be left with a broken app is also not an option. Maybe some way to resume the app from where it was at the time of the OOME? Any ideas regarding that?
  • 23. Re: How to catch OutOfMemoryError?
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    Yes, thanks, that is exactly what I meant. The point is that the available methods may not work when used in the context of multiple threads, because either you can't rely on the result of getFreeMemory() or by the time an OOME is thrown, the stack is lost and you can't resume your program.
    Yes you can.

    As I said you need to understand your application. If you don't then you can't do it.

    The following certainly works
            while(...)
                  {
                     try
                         {
                         setup()
                         callSomething()
                         }
                      catch()
                         {
                         // Do something to make it more likely to work.
                         }
                  }
  • 24. Re: How to catch OutOfMemoryError?
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    Anytime you can substitute slow disk access with fast memory access to get the most out of the server.
    And that has nothing to do with using all of the REMAINING memory.

    As I said I really doubt that is ever a good idea.

    However by using a lot (which is not all) of the remaining memory or even assuming that a large block exists (by requiring a configuration value) on startup provides all sorts of opportunities.

    You might note that "fast" doesn't equate to "remaining" anyways in terms of generic access to memory in any language. You have no idea whether it is paging to the hard drive or not. On the other hand someone that it installing it to a specific box should have some idea of what the environment would support. And thus requiring configuration/tuning to the specific box is required.
    That "almost" is what bothers me.
    Any modern OS on a standard desktop/server is always going to use paging unless you specifically (again configuration) keep the app from exceeding the known limits of the box.
    Not using threads is not an option.
    And why not?

    Are you trying to build something that is fast or just "cool"?
    Maybe some way to resume the app from where it was at the time of the OOME? Any ideas regarding that?
    Nope. But excluding Basic I don't know of any language that does that. Not C, C++, C# nor perl.

    That doesn't mean that I can't do anything at all about it. It does require a solution for each specific application rather than thinking that there is some general solution that works for everything.

    Or just don't let it happen in the first place.....
            Workspace w = RetrieveWorkspaceFromList()
            // Workspace has all of the space needed 
            // so at this point space is assured.
            DoWork(w)
  • 25. Re: How to catch OutOfMemoryError?
    796254 Newbie
    Currently Being Moderated
    The original discussion is almost two years old, but perhaps still pertinent.

    Java's GC is a generational algorithm. You can end up with an OOME for a lot of reasons, and not all of them have to do with the max RAM available. For example, if the perm space is filled up you're screwed. So it's not so easy a question.

    My feeling is that there's little point in going on if you get an OOME, and your best bet is to get busy figuring out why it happened.

    %
  • 26. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    jschell wrote:
    while(...)
    {
    try
    {
    setup()
    callSomething()
    }
    catch()
    {
    // Do something to make it more likely to work.
    }
    }
    Yeah that still disrupts the stack though. What he's saying is completely different. I don't agree with the idea though, because the only thing it would let you clean up are static references (unless your "freeMemory()" method had access to the entire heap of objects), so basically it would only be able to gain memory through freeing globally referenced objects.

    Let's end this discussion with the Java API:
    *Java 1.5 API - Error*:
    An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. The ThreadDeath error, though a "normal" condition, is also a subclass of Error because most applications should not try to catch it.

    A method is not required to declare in its throws clause any subclasses of Error that might be thrown during the execution of the method but not caught, since these errors are abnormal conditions that should never occur.
  • 27. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    Anytime you can substitute slow disk access with fast memory access to get the most out of the server.
    And that has nothing to do with using all of the REMAINING memory.
    As I said I really doubt that is ever a good idea.
    Yes, you are correct, but I don't want to get all the remaining memory, just as much as possible without crashing the rest of the system.
    You might note that "fast" doesn't equate to "remaining" anyways in terms of generic access to memory in any language. You have no idea whether it is paging to the hard drive or not. On the other hand someone that it installing it to a specific box should have some idea of what the environment would support. And thus requiring configuration/tuning to the specific box is required.
    Again you are correct, and I do control the memory and process allocation on the server to make sure that it won't be paged, at least most of the time, so I do know that my memory buffers and queues stay in RAM, most of the time, which is what I need for the memory/speed tradeoff to work.
    Not using threads is not an option.
    And why not?
    Are you trying to build something that is fast or just "cool"?
    Are you really suggesting that threads serve no useful purpose?
    That doesn't mean that I can't do anything at all about it. It does require a solution for each specific application rather than thinking that there is some general solution that works for everything.

    Or just don't let it happen in the first place.....
    Workspace w = RetrieveWorkspaceFromList()
    // Workspace has all of the space needed 
    // so at this point space is assured.
    DoWork(w)
    Yes, I see that this would work, provided that the whole application, all classes and libraries and plugins and whatnot pre-allocated their memory. But how practical is that? Is it even possible?

    The thing is, most systems are dynamic in one way or another, and the memory needs vary from one moment to the next and as a response to external stimuli. Eventually two parts of the system will compete for the same memory. And there is no resolution mechanism. So each of these have to be extremely conservative to avoid an OOME hitting some random part of the system, because there is no practical way out of an OOME, and thus the system is not making the best possible use of the hardware.

    I know it looks like I'm just venting my frustration, and I am, but I came here in the hope that there is an answer, some technique I haven't heard of that helps keep the OOME in check. Sorry if I'm wasting your time.
  • 28. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    jhandl wrote:
    Not using threads is not an option.
    And why not?
    Are you trying to build something that is fast or just "cool"?
    Are you really suggesting that threads serve no useful purpose?
    No, he's suggesting that there are cases of actual programs that aren't multithreaded. This shouldn't come as a shock.
    I know it looks like I'm just venting my frustration, and I am, but I came here in the hope that there is an answer, some technique I haven't heard of that helps keep the OOME in check. Sorry if I'm wasting your time.
    Don't be sorry. This kind of debate is healthy when it's not simply trolling.
  • 29. Re: How to catch OutOfMemoryError?
    807592 Newbie
    Currently Being Moderated
    endasil wrote:
    jschell wrote:
    while(...)
    {
    try
    {
    setup()
    callSomething()
    }
    catch()
    {
    // Do something to make it more likely to work.
    }
    }
    Yeah that still disrupts the stack though.
    No, it would actually work but it requires that the whole application be instrumented in this way at each place an object is created, which is impractical for any number of reasons. I have to admit, though, that I remember reading about the need to check for null after a malloc in C and feeling guilty for not doing it (as all the rest of the programmers I know or have seen code of).
    What he's saying is completely different. I don't agree with the idea though, because the only thing it would let you clean up are static references (unless your "freeMemory()" method had access to the entire heap of objects), so basically it would only be able to gain memory through freeing globally referenced objects.
    That would be entirely possible. The handler could have a few KB or MB around just for the occation, free them as the first step to get some air, and then communicate with the critical parts of the system to let them know they have to cut down on RAM consumption and after some non-essential memory has been liberated, recapture the small buffer before relinquishing control and let the application (or the GC, more likely) resume.
    Let's end this discussion with the Java API:
    *Java 1.5 API - Error*:
    An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. The ThreadDeath error, though a "normal" condition, is also a subclass of Error because most applications should not try to catch it.

    A method is not required to declare in its throws clause any subclasses of Error that might be thrown during the execution of the method but not caught, since these errors are abnormal conditions that should never occur.
    Yeah, I wish the real world was so simple.