3 Replies Latest reply: Apr 18, 2012 3:10 PM by greybird RSS

    running out of memory despite having set je.maxMemory to a moderate value

    931231
      I have set je.maxMemory to 20MB (je.maxMemory=20000000) and allowed a max heap size of 512MB (-Xms256M -Xmx512M).

      After two hours of running my web service, I'm running out of memory. After having profiled my service (using Yourkit Java Profiler 1.10.6), I can see the following:
      Name                                               Objects ShallowSize  RetainedSize
      byte[]                                               16711   124124880     124124880
      com.sleepycat.je.tree.BIN                              181       24616     116254200
      com.sleepycat.je.tree.Node[]                           187       98736     115743184
      com.sleepycat.je.tree.LN                              7092      226944     115253600
      java.util.concurrent.ConcurrentHashMap$HashEntry       554       17728      78328944
      java.util.concurrent.ConcurrentHashMap$HashEntry[]    1053       34728      77489632
      java.util.concurrent.ConcurrentHashMap                 117        5616      71812072
      java.util.concurrent.ConcurrentHashMap$Segment[]       118       10304      71807912
      java.util.concurrent.ConcurrentHashMap$Segment        1052       42080      71798808
      com.sleepycat.je.tree.IN                                 6         672      45592352
      java.lang.String                                    135888     4348416      14152664
      The memory profiler claims further, that com.sleepycat.je.tree.BIN is responsible for 71% of all heap memory.

      In any case, com.sleepycat.je.tree.BIN claims ~ 116MB of heap memory, which is by any goodwill, exceeded the limit of 20MB.

      How can this be?

      How is JE ensuring that the limit is not exceeded? Is there a timer (thread) running which once a while checks the memory used and then cleans up ; or is memory usage checked creating a com.sleepycat.je.tree.BIN object?

      My environment:
      BDB JE 4.0.92 - used as cache loader within Jboss Cache (3.2.7.GA), running on a JBOSS Application Server, Java 1.6 (IBM) on Linux. Further details are listed in the system properties below (except some deleted security items).


      System properties:
      (java.lang.String, int, java.lang.StringBuffer, int)=contains
      DestroyJavaVM helper thread=(java.lang.String, java.security.KeyStore$Entry, java.security.KeyStore$ProtectionParameter)
      base.collection.name=CD2JAVA
      bind.address=10.12.25.130
      catalina.base=/work/ocrgws_test/server0
      catalina.ext.dirs=/work/ocrgws_test/server0/lib
      catalina.home=/work/ocrgws_test/server0
      catalina.useNaming=false
      com.arjuna.ats.arjuna.objectstore.objectStoreDir=/work/ocrgws_test/server0/data/tx-object-store
      com.arjuna.ats.jta.lastResourceOptimisationInterface=org.jboss.tm.LastResource
      com.arjuna.ats.tsmx.agentimpl=com.arjuna.ats.internal.jbossatx.agent.LocalJBossAgentImpl
      com.arjuna.common.util.logger=log4j_releveler
      com.arjuna.common.util.logging.DebugLevel=0x00000000
      com.arjuna.common.util.logging.FacilityLevel=0xffffffff
      com.arjuna.common.util.logging.VisibilityLevel=0xffffffff
      com.ibm.cpu.endian=little
      com.ibm.jcl.checkClassPath=
      com.ibm.oti.configuration=scar
      com.ibm.oti.jcl.build=20100326_1904
      com.ibm.oti.shared.enabled=false
      com.ibm.oti.vm.bootstrap.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64
      com.ibm.oti.vm.library.version=24
      com.ibm.util.extralibs.properties=
      com.ibm.vm.bitmode=64
      common.loader=${catalina.home}/lib,${catalina.home}/lib/*.jar
      epo.jboss.deploymentscanner.extradirs=/work/ocrgws_test/app/
      external.cert.ldap.* = ***************
      file.encoding=UTF-8
      file.separator=/
      flipflop.activation.time=16:30
      hibernate.bytecode.provider=javassist
      ibm.signalhandling.rs=false
      ibm.signalhandling.sigchain=true
      ibm.signalhandling.sigint=true
      ibm.system.encoding=UTF-8
      jacorb.config.log.verbosity=0
      java.assistive=ON
      java.awt.fonts=
      java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
      java.awt.printerjob=sun.print.PSPrinterJob
      java.class.path=/work/ocrgws_test/config:/usr/local/jboss-eap-4.3-cp07/bin/run.jar:/opt/ibm/java-x86_64-60/lib/tools.jar
      java.class.version=50.0
      java.compiler=j9jit24
      java.endorsed.dirs=/usr/local/jboss-eap-4.3-cp07/lib/endorsed
      java.ext.dirs=/opt/ibm/java-x86_64-60/jre/lib/ext
      java.fullversion=JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr8-20100401_55940 (JIT enabled, AOT enabled)
      J9VM - 20100401_055940
      JIT - r9_20100401_15339
      GC - 20100308_AA_CMPRSS
      java.home=/opt/ibm/java-x86_64-60/jre
      java.io.tmpdir=/tmp
      java.jcl.version=20100408_01
      java.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64:/usr/lib64/mpi/gcc/openmpi/lib64:/usr/lib
      java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
      java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
      java.net.preferIPv4Stack=true
      java.protocol.handler.pkgs=org.jboss.net.protocol
      java.rmi.server.codebase=http://10.12.25.130:8083/
      java.rmi.server.hostname=10.12.25.130
      java.rmi.server.randomIDs=true
      java.runtime.name=Java(TM) SE Runtime Environment
      java.runtime.version=pxa6460sr8-20100409_01 (SR8)
      java.security.krb5.conf=/usr/local/jboss/etc/krb5.conf
      java.specification.name=Java Platform API Specification
      java.specification.vendor=Sun Microsystems Inc.
      java.specification.version=1.6
      java.util.prefs.PreferencesFactory=java.util.prefs.FileSystemPreferencesFactory
      java.vendor.url=http://www.ibm.com/
      java.vendor=IBM Corporation
      java.version=1.6.0
      java.vm.info=JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr8-20100401_55940 (JIT enabled, AOT enabled)
      J9VM - 20100401_055940
      JIT - r9_20100401_15339
      GC - 20100308_AA_CMPRSS
      java.vm.name=IBM J9 VM
      java.vm.specification.name=Java Virtual Machine Specification
      java.vm.specification.vendor=Sun Microsystems Inc.
      java.vm.specification.version=1.0
      java.vm.vendor=IBM Corporation
      java.vm.version=2.4
      javax.management.builder.initial=org.jboss.mx.server.MBeanServerBuilderImpl
      javax.net.ssl.trustStore=/usr/local/jboss/etc/ldap.truststore
      javax.net.ssl.trustStorePassword=password
      jboss.bind.address=10.12.25.130
      jboss.home.dir=/usr/local/jboss-eap-4.3-cp07
      jboss.home.url=file:/usr/local/jboss-eap-4.3-cp07/
      jboss.identity=30df88bc0a52e350x6e2ff59cx136c17794d5x-8000757
      jboss.lib.url=file:/usr/local/jboss-eap-4.3-cp07/lib/
      jboss.messaging.controlchanneludpaddress=239.1.200.4
      jboss.messaging.datachanneludpaddress=239.1.200.4
      jboss.partition.name=ocrgws_test_Partition
      jboss.partition.udpGroup=239.1.200.4
      jboss.remoting.domain=JBOSS
      jboss.remoting.instanceid=30df88bc0a52e350x6e2ff59cx136c17794d5x-8000757
      jboss.remoting.jmxid=luu002t.internal.epo.org_1334685694459
      jboss.remoting.version=22
      jboss.security.disable.secdomain.option=true
      jboss.server.config.url=file:/work/ocrgws_test/server0/conf/
      jboss.server.data.dir=/work/ocrgws_test/server0/data
      jboss.server.home.dir=/work/ocrgws_test/server0
      jboss.server.home.url=file:/work/ocrgws_test/server0/
      jboss.server.lib.url=file:/work/ocrgws_test/server0/lib/
      jboss.server.log.dir=/work/ocrgws_test/server0/log
      jboss.server.name=luu002t_ocrgws_test_server0
      jboss.server.temp.dir=/work/ocrgws_test/server0/tmp
      jboss.tomcat.udpGroup=239.1.200.4
      jbossmx.loader.repository.class=org.jboss.mx.loading.UnifiedLoaderRepository3
      je.maxMemory=20000000
      jgroups.bind_addr=10.12.25.130
      jmx.console.bindcredential=3bpwdmpc
      jmx.console.binddn=cn=jbossauth-ro,ou=accounts,ou=auth,dc=epo,dc=org
      jmx.console.rolesctxdn=ou=roles-test,ou=jboss,ou=applications,ou=internal,dc=epo,dc=org
      jndi.datasource.name=java:MainframeDS
      jnp.disableDiscovery=true
      jxe.current.romimage.version=15
      jxe.lowest.romimage.version=15
      line.separator=

      mainframelogin.password=720652a1e842fc7f
      mainframelogin.username=test_t
      org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger
      org.apache.tomcat.util.http.ServerCookie.VERSION_SWITCH=true
      org.epo.jboss.application.home=/work/ocrgws_test
      org.hyperic.sigar.path=/work/ocrgws_test/server0/./deploy/hyperic-hq.war/native-lib
      org.jboss.ORBSingletonDelegate=org.jacorb.orb.ORBSingleton
      org.omg.CORBA.ORBClass=org.jacorb.orb.ORB
      org.omg.CORBA.ORBSingletonClass=org.jboss.system.ORBSingleton
      org.w3c.dom.DOMImplementationSourceList=org.apache.xerces.dom.DOMXSImplementationSourceImpl
      os.arch=amd64
      os.name=Linux
      os.version=2.6.32.46-0.3-xen
      package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.,sun.beans.
      package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.
      path.separator=:
      poll.interval.milliseconds=300000
      program.name=run.sh
      server.loader=
      shared.loader=
      spnego.config=/usr/local/jboss/etc/spnego.properties
      sun.arch.data.model=64
      sun.boot.class.path=/usr/local/jboss-eap-4.3-cp07/lib/endorsed/xercesImpl.jar:/usr/local/jboss-eap-4.3-cp07/lib/endorsed/xalan.jar:/usr/local/jboss-eap-4.3-cp07/lib/endorsed/serializer.jar:/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs/jclSC160/vm.jar:/opt/ibm/java-x86_64-60/jre/lib/annotation.jar:/opt/ibm/java-x86_64-60/jre/lib/beans.jar:/opt/ibm/java-x86_64-60/jre/lib/java.util.jar:/opt/ibm/java-x86_64-60/jre/lib/jndi.jar:/opt/ibm/java-x86_64-60/jre/lib/logging.jar:/opt/ibm/java-x86_64-60/jre/lib/security.jar:/opt/ibm/java-x86_64-60/jre/lib/sql.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmorb.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmorbapi.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcfw.jar:/opt/ibm/java-x86_64-60/jre/lib/rt.jar:/opt/ibm/java-x86_64-60/jre/lib/charsets.jar:/opt/ibm/java-x86_64-60/jre/lib/resources.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmpkcs.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcertpathfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjgssfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjssefw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmsaslfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjcefw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjgssprovider.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjsseprovider2.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcertpathprovider.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmxmlcrypto.jar:/opt/ibm/java-x86_64-60/jre/lib/management-agent.jar:/opt/ibm/java-x86_64-60/jre/lib/xml.jar:/opt/ibm/java-x86_64-60/jre/lib/jlm.jar:/opt/ibm/java-x86_64-60/jre/lib/javascript.jar:/tmp/yjp201202191932.jar
      sun.boot.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64
      sun.io.unicode.encoding=UnicodeLittle
      sun.java.command=org.jboss.Main -b 10.12.25.130 -Djboss.server.home.dir=/work/ocrgws_test/server0 -Djboss.server.home.url=file:/work/ocrgws_test/server0 -Djboss.server.name=luu002t_ocrgws_test_server0 -Djboss.partition.name=ocrgws_test_Partition -Depo.jboss.deploymentscanner.extradirs=/work/ocrgws_test/app/ -Dorg.epo.jboss.application.home=/work/ocrgws_test
      sun.java.launcher.pid=17781
      sun.java.launcher=SUN_STANDARD
      sun.java2d.fontpath=
      sun.jnu.encoding=UTF-8
      sun.rmi.dgc.client.gcInterval=3685000
      sun.rmi.dgc.server.gcInterval=3685000
      system=java.io.ObjectStreamField
      tomcat.util.buf.StringCache.byte.enabled=true
      user.country=US
      user.dir=/work/ocrgws_test
      user.home=*****************
      user.language=en
      user.name=***********
      user.timezone=Europe/Berlin
      user.variant=
        • 1. Re: running out of memory despite having set je.maxMemory to a moderate value
          greybird
          The memory profiler claims further, that com.sleepycat.je.tree.BIN is responsible for 71% of all heap memory.
          In any case, com.sleepycat.je.tree.BIN claims ~ 116MB of heap memory, which is by any goodwill, exceeded the limit of 20MB. >

          I'm not sure whether the profiler is reporting live objects only (referenced) or all objects (including those not yet reclaimed). If the latter, it isn't telling you how much memory is actually referenced by the JE cache.

          Please look at the JE stats to see what the cache usage is, from JE's point of view.

          If you believe there is a bug in JE cache management, you'll need to write a small standalone test to demonstrate it and submit it to us, since we don't know of any such bug. Also note that we'll have difficulty supporting JE 4.0 (without a support contract anyway). Please use JE 5.0, or at least 4.1.

          Eviction occurs as objects are allocated, as well as in background threads. Eviction in background threads and concurrent eviction were greatly improved in JE 4.1.

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
          • 2. Re: running out of memory despite having set je.maxMemory to a moderate value
            931231
            I'm not sure whether the profiler is reporting live objects
            Live objects, i.e. only those that can't be garbage collected.
            Please look at the JE stats to see what the cache usage is, from JE's point of view.
            Not as easy as it sounds, will try. Is it possible to connect via jconsole (JMX) to get some live statistics??

            Anyway, I'm going to use latest 4.1.17, 5.x can't be used due to compatibility issues.

            //wolfgang

            P.S: thx for any support so far
            • 3. Re: running out of memory despite having set je.maxMemory to a moderate value
              greybird
              Yes, JMX can be used, see the docs.
              5.x can't be used due to compatibility issues
              What compatibility issues? Are you using Voldemort by chance?

              --mark