This discussion is archived
9 Replies Latest reply: Feb 5, 2013 6:04 AM by 870083 RSS

Ensuring software doesn't exceed memory available.

870083 Newbie
Currently Being Moderated
Hello,

I have been developing some software which carries out a large number of experiments and then exports alot of the information to html reports:
- Take user input
- Iterate through and create each experiment
- Iterate through and analyse each experiment
- Export results.

Ideally I want to do as many as I can but with my current design I am limited by physical memory because it retains all the information within the program until its finished and ready to export which I know isn't an ideal arrangement.
For development purposes I have just been using -Xmx because I have a computer with 24gb RAM at disposal but I know I willl have to change this for the final design.
I'm going to write the information from each stage in chunks (eg. every 100,000) and then read it back in when needed (eg. the next stage) but my question is how can I prevent the out of memory exception from happening and just leaving the user baffled (especially if they haven't run it from console)? Even with this approach I do not know what resources will be available on machines which will use the software so what are some acceptable ways of handling this problem?

I realise this is probably a very newbie question but I've never written anything before which has had any chance of exceeding a computer resources.

Thanks.
  • 1. Re: Ensuring software doesn't exceed memory available.
    Kayaman Guru
    Currently Being Moderated
    You still use -Xmx. You just set it to a reasonable level, if the defaults aren't good enough.

    Test your program to see the amount of memory it needs, then use that (+ a safety barrier). If the Xmx is too large for the computer trying to run it, then you'll know that at the time when you try to run it.
  • 2. Re: Ensuring software doesn't exceed memory available.
    870083 Newbie
    Currently Being Moderated
    Thanks the for the reply,

    I suppose I have to test it to a certain cut off level which I limit the software by and track the memory used, then ensure that memory usage is well within the heap size assigned. I suppose what you are saying is then is if I can't be sure of the possible outcomes and worse case scenarios available even within the set limits of the software, you can't guarantee that it will not occur. Is it sensible to at least catch it and display a dialog to inform the user or something?
  • 3. Re: Ensuring software doesn't exceed memory available.
    gimbal2 Guru
    Currently Being Moderated
    Kayaman wrote:
    You still use -Xmx. You just set it to a reasonable level, if the defaults aren't good enough.
    Indeed, but an extra step is needed. As you say a test should be performed, to figure out what a reasonable expectation in memory usage is going to be which will likely also involve some guess work. Then put the findings in system requirements that are attached to the software as a disclaimer.

    IMO: you can't really make reporting software perform well on limited hardware, you need memory to burn. So you shouldn't even try to handle it, make sure people know that they need the gear to run the stuff. If they then try to run it on a laptop with 512mb of memory, its their own bloody fault it doesn't work.
  • 4. Re: Ensuring software doesn't exceed memory available.
    870083 Newbie
    Currently Being Moderated
    haha ok then I'll accept that as my answer.

    I thought that might be the case but I was hoping there might be another solution. I know the software will be used on hardware ranging from 4 to 24gb of RAM so it would have been nice to utilise those extra resources and make the software more scalable without the user having to launch the program from the console with -Xmx because I know it could improve performance and reduce the number of file read/writes carried out.

    Thanks.
  • 5. Re: Ensuring software doesn't exceed memory available.
    gimbal2 Guru
    Currently Being Moderated
    Technical wrote:
    haha ok then I'll accept that as my answer.

    I thought that might be the case but I was hoping there might be another solution. I know the software will be used on hardware ranging from 4 to 24gb of RAM so it would have been nice to utilise those extra resources and make the software more scalable without the user having to launch the program from the console with -Xmx because I know it could improve performance and reduce the number of file read/writes carried out.
    Actually the server VM scales the maximum heap space according to available resources if you don't provide settings yourself. So you could achieve it if you provide a script to run the stuff that forces the server VM to be used.

    http://www.oracle.com/technetwork/java/javase/6u18-142093.html
  • 6. Re: Ensuring software doesn't exceed memory available.
    r035198x Pro
    Currently Being Moderated
    Consider selecting the chunk sizes dynamically based on the max memory available so the process is just slower for lower memory requirements. Also you don't have to guess program memory requirements, use jconsole to monitor the program and you'll have very close figures. After you have collected all these statistics and done lots of performance testing you'd even be in a position to tell the user how long the reports are going to take before running them.
  • 7. Re: Ensuring software doesn't exceed memory available.
    870083 Newbie
    Currently Being Moderated
    I like the sounds of that alot! I'll check it out and try it.

    How would I change the chunks dynamically? Is there some way of retrieving the max heap within the applicaion?
    I assume you mean you determine the max heap size and then from knowing roughly how much you can do per mb of memory determine how much each chunk should be?

    Thanks for the reply.
  • 8. Re: Ensuring software doesn't exceed memory available.
    Kayaman Guru
    Currently Being Moderated
    Technical wrote:
    I like the sounds of that alot! I'll check it out and try it.

    How would I change the chunks dynamically? Is there some way of retrieving the max heap within the applicaion?
    I assume you mean you determine the max heap size and then from knowing roughly how much you can do per mb of memory determine how much each chunk should be?
    Well, for one you can do
    Runtime.getRuntime().freeMemory(); // also totalMemory() and maxMemory()
    to see a bit about the available memory for your application.
  • 9. Re: Ensuring software doesn't exceed memory available.
    870083 Newbie
    Currently Being Moderated
    That's really useful, I never even thought to look for such a method. I'm sure I can do something with that.

    Thanks.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points