This discussion is archived
8 Replies Latest reply: May 31, 2012 12:15 PM by DarrylBurke RSS

ImageIO leaks mmaps in jdk1.7.0_04 with certain JPEGs on linux.

456795 Newbie
Currently Being Moderated
Going from jdk1.7.0_03 to jdk1.7.0_04 I discovered a memory leak in the Linux 64bit implementation of jdk1.7.0_04. I haven't tested on other platforms, just Ubuntu. It took several days to track it down the cause. I was getting following errors in our logs:

java.lang.OutOfMemoryError.
Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
mmap failed for CEN and END part of zip file

From using the information discovered at http://blog.timstoop.nl/tag/java/ I was running out of Memory Map IO in jdk1.7.0_04. After a couple of hours of running our server we were exhausting the /proc/sys/vm/max_map_count which defaults to 65530. I increased it to 1,000,000. By running the following command I noticed that eventually I was going to approach that limit:

sudo cat /proc/$PID/maps | wc -l

I also noticed that there were a lot of lines that contained the text “(deleted)”

sudo cat /proc/$PID/maps | fgrep '(deleted)'
7f4c5d4d9000-7f4c5d4da000 rw-s 00000000 00:12 469934 /run/shm/sem.ZEIYcO (deleted)
7f4c5d4da000-7f4c5d4db000 rw-s 00000000 00:12 469933 /run/shm/sem.CpxrYf (deleted)

It looks like leaking semaphores to me. While searching for semaphore leaks, I found:

http://stackoverflow.com/questions/10441276/jdk-1-7-too-many-open-files-due-to-posix-semaphores
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7166379

I thought I had found a solution. After a few trials I realized that wasn't exactly the same problem, and I was still getting leaking semaphores, even with the workaround, with some images. What I finally discovered was that using ImageIO with certain JPEGs causes a the memory maps to grow unbounded in jdk1.7.0_04 but not in jdk1.7.0_03. Most JPEGs don't cause the memory leak.

Here is my test case. It is designed to run on Ubuntu and only causes problems on jdk1.7.0_04. I've tested it with no problems on jkd1.7.0_01 and jdk1.7.0_03. Take the image http://s9.postimage.org/h6mmgn3ij/ico_rx_sm.jpg or here http://postimage.org/image/h6mmgn3ij/ if the direct link changes. It's a small Rx image. Copy it to the same directory as the compiled class. If you increase “iterations” to 25,000 to max out the mmaps, it won't crash but will exit with:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

If it was a more complex application then any time the VM needed to mmap, it would fail.

I'm wondering if this “Memory leak” is related to:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7166379
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7167647

and if I should submit my own bug report.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.util.Iterator;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;


public class MMAPMemoryLeakTest {

     public static void main(String... args) throws IOException {
          final int iterations = Integer.parseInt(args[0]);
          final File file = new File(args[1]);

          String processId = ManagementFactory.getRuntimeMXBean().getName();
          processId = processId.substring(0, processId.indexOf('@'));
          System.out.println("Image is " + file);
          System.out.println("Process ID is " + processId);
          System.out.println("Iterating " + iterations + " times ");
          int height = 0;
          int width = 0;
          for (int i = 0; i < iterations; i++) {
               try (ImageInputStream iis = ImageIO.createImageInputStream(file)) {
                    Iterator<ImageReader> iter = ImageIO.getImageReaders(iis);
                    if (iter.hasNext()) {
                         ImageReader reader = null;
                         try {
                              reader = iter.next();
                              reader.setInput(iis);
                              height = reader.getHeight(0);
                              width = reader.getWidth(0);
                         } finally {
                              if (reader != null) {
                                   reader.dispose();
                              }
                         }
                    }
               }
          }
          String mmapFile = "/proc/" + processId + "/maps";
          System.out.println("Examining " + mmapFile + " for leaks.");
          try (BufferedReader in = new BufferedReader(new FileReader(mmapFile))) {
               int totalMMaps = 0;
               int markedAsDeleted = 0;
               String ln = in.readLine();
               while (ln != null) {
                    totalMMaps++;
                    int index = ln.indexOf("(deleted)");
                    if (index != -1) {
                         markedAsDeleted++;
                    }
                    ln = in.readLine();
               }
               System.out.println("Image height, width is " + height + ", " + width);
               System.out.println("Total mmaps is " + totalMMaps);
               System.out.println("(deleted) mmaps is " + markedAsDeleted);               
               System.out.println("difference is " + (totalMMaps - markedAsDeleted));               
          }
     }
}
Results:

jdk1.7.0_04/bin/java MMAPMemoryLeakTest 1000 ico_rx_sm.jpg

Process ID is 17357
Iterating 1000 times
Examining /proc/17357/maps for leaks.
Image height, width is 17, 18
Total mmaps is 3210
(deleted) mmaps is 3000
difference is 210


jdk1.7.0_04/bin/java MMAPMemoryLeakTest 25000 ico_rx_sm.jpg

Process ID is 17404
Iterating 25000 times
Examining /proc/17404/maps for leaks.
Image height, width is 17, 18
Total mmaps is 65532
(deleted) mmaps is 65321
difference is 211
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.


jdk1.7.0_03/bin/java MMAPMemoryLeakTest 1000 ico_rx_sm.jpg

Process ID is 17715
Iterating 1000 times
Examining /proc/17715/maps for leaks.
Image height, width is 17, 18
Total mmaps is 209
(deleted) mmaps is 0
difference is 209


jdk1.7.0_03/bin/java MMAPMemoryLeakTest 25000 ico_rx_sm.jpg

Process ID is 17499
Iterating 25000 times
Examining /proc/17499/maps for leaks.
Image height, width is 17, 18
Total mmaps is 211
(deleted) mmaps is 0
difference is 211


uname -a
Linux XXXX 3.2.0-24-generic #38-Ubuntu SMP Tue May 1 16:18:50 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.2.0-24-generic root=UUID=XXXXXXXX ro transparent_hugepage=always

I've tested on several machines here at work with the same results(Most with previous kernels). If there is anything else I can to do help, let me know.

Thanks.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points