This discussion is archived
7 Replies Latest reply: Jul 30, 2011 3:19 AM by EJP RSS

Insufficient System Resources when merging large files

844979 Newbie
Currently Being Moderated
My client is running on Windows Server 2003, 64 bit. He has 30 gig of RAM and a large amount of file storage. The drives are NTFS.

I have a program that produces a number of text files from different processes, and then merges them when done. After running the code for many days (we're working with a lot of data), the merge process is failing with "Insufficient System Resources Exist to complete the requested process".


Insufficient system resources exist to complete the requested service
java.io.IOException: Insufficient system resources exist to complete the requested service
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(Unknown Source)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.io.OutputStreamWriter.write(Unknown Source)
at java.io.BufferedWriter.write(Unknown Source)
at functionality.ScenarioThreadedLoanProcessor.processLoans(ScenarioThreadedLoanProcessor.java:723)
at functionality.ScenarioThreadedLoanProcessor.construct(ScenarioThreadedLoanProcessor.java:227)
at utility.SwingWorker$2.run(SwingWorker.java:131)
at java.lang.Thread.run(Unknown Source)

I've investigated this problem in other places, and most of the answers seem to not apply to my case.

I've looked a this: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4938442

But I am not using file channels (I don't think...), and I am copying data in chunks of <32MB.

Here's the relevant code (I realize I don't need to be re-allocating the buffer, that's a legacy from an older version, but I don't think that's the cause.)

There are usually four threads, 1-4 reports, and 1 scenario, so this loop shouldn't be executing thousands and thousands of times.
for (int scenario = 0; scenario < scenarios.size(); scenario++)
               {
                    for (int report = 0; report < reportPages.size(); report++)
                    {
                         for (LoanThread ln : loanThreads)
                         {
                              BufferedReader br = new BufferedReader(new FileReader(new File(ln.reportPtr.get(
                                        (scenario * reportPages.size()) + report).getFileName())));
                              br.readLine();//header, throw away
                              BufferedWriter bw = new BufferedWriter(new FileWriter(formReport.get((scenario * reportPages.size()) + report)
                                        .getFileName(), true));//append
                              char[] bu = new char[1024 * 1024 * 16];
                              int charsRead = 0;
                              while ((charsRead = br.read(bu)) != -1)
                              {
                                   bw.write(bu, 0, charsRead);
                              }
                              bw.flush();
                              bw.close();
                              br.close();
                              File f = new File(ln.reportPtr.get((scenario * reportPages.size()) + report).getFileName());
                              f.delete();
                         }
                    }
               }

Any thoughts?
Edited by: LizardSF on Jul 29, 2011 8:11 AM

Edited by: sabre150 on 29-Jul-2011 09:02

Added [ code] tags to make the code (more) readable
  • 1. Re: Insufficient System Resources when merging large files
    sabre150 Expert
    Currently Being Moderated
    Some thoughts but I don't expect them to be the cause of you problem unless elsewhere you are ignoring some exceptions.

    1) You can allocate the buffer at the start of the outer loop to save the GC working overtime. You might even be able to move it out of the loops all together but I would need to see more code to be sure of that.
    2) bw.flush() is not needed since the close() contract ensures that anyway.
    3) the close() calls should be done in a finally clause making sure that if one fails the other is still called.

    P.S. Please use spaces instead of tabs.
  • 2. Re: Insufficient System Resources when merging large files
    jtahlborn Expert
    Currently Being Moderated
    well, for one thing, you should always close stream resources in finally blocks. that's probably not the cause of this issue, though, unless this code ends up throwing a lot of exceptions.
  • 3. Re: Insufficient System Resources when merging large files
    jtahlborn Expert
    Currently Being Moderated
    i don't think it's a java problem, i think it's a windows problem. googling for "windows server 2003 +"Insufficient system resources exist to complete the requested service"" led me to this article http://support.microsoft.com/kb/304101 .
  • 4. Re: Insufficient System Resources when merging large files
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    Presuming it is a windows problem then one solution would be to run as separate processes using Runtime.exec().

    So each piece gets spun up as a different VM and runs to completion.
  • 5. Re: Insufficient System Resources when merging large files
    EJP Guru
    Currently Being Moderated
    1) You can allocate the buffer at the start of the outer loop to save the GC working overtime. You might even be able to move it out of the loops all together but I would need to see more code to be sure of that.
    +1. Exactly. This is the most important change you must do.

    The other change I would make is to reduce the buffer size. The disk only works in units of 4-16k at a time anyway. You will be surprised how much you can reduce it without affecting performance at all. I would cut it down to no more than a megabyte.

    You could also speed it up probably by a factor of at least two by using InputStreams and OutputStreams and a byte[] buffer, instead of Readers and Writers and char[], as you're only copying the file anyway. Also, those BufferedReaders and Writers are contributing nothing much, in fact nothing after the readLine(), as you are already using a huge buffer. Finally, you should also investigate FileChannel.transferTo() to get even more performance, and no heap memory usage whatsoever. Note that like your copy loop above, you have to check the result it returns and loop until the copy is complete. There are also upper limits on the transferCount that are imposed by the operating system and will cause exceptions, so again don't try to set it too big. A megabyte is again sufficient.
  • 6. Re: Insufficient System Resources when merging large files
    802316 Pro
    Currently Being Moderated
    When you use a buffered reader/writer it copies the data into the buffer between your read/write and the OS. This works well when you have many small read/writes however when your read/write is larger than the buffer (the default is 8K chars) it can hurt performance.

    You might find that a buffer which is the same size as your primary cache (L1) is the optimal size to use. Say you have a 32 KB primary cache, try a new char[16*1024] or if you use bytes try new byte[32 * 1024]

    Is there a need to convert bytes into char and back into bytes except to remove the first line. This is a relatively expensive operation. You might consider rewriting this to use only bytes if you can safely assume that your new line will be \n.

    A further optimisation especially as you don't look at most of the data is to use direct ByteBuffers. These are particularly efficient when you don't actually read/write the data in Java space.

    BTW: close() calls flush()
  • 7. Re: Insufficient System Resources when merging large files
    EJP Guru
    Currently Being Moderated
    however when your read/write is larger than the buffer (the default is 8K chars) it can hurt performance.
    You might but in practice you won't. Reads and writes larger than the buffers go straight through without being buffered internally. Obviously there is a prior flush().
    A further optimisation especially as you don't look at most of the data is to use direct ByteBuffers. These are particularly efficient when you don't actually read/write the data in Java space.
    He does, he is reading and losing the first line.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points