6 Replies Latest reply: Mar 15, 2012 1:50 AM by 915071 RSS

    Exception in thread "RMI

    915071
      Hello, I configure dbcache 90m for my data and heap size is 150m, but when I run multi-thread update on my store, the exception happened, and I can't restart the SN.
      by the way, I have followed the introduction for compute dbcache and heap size.


      03-09-12 11:33:19:81 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Exception in thread "RMI TCP Connection(idle)" Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      03-09-12 11:33:19:83 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Process exiting due to fault
      03-09-12 11:33:19:83 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Process exiting due to fault
      03-09-12 11:33:19:83 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: java.lang.OutOfMemoryError: Java heap space
      03-09-12 11:33:19:83 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: java.lang.OutOfMemoryError: Java heap space
      03-09-12 11:33:20:95 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: IOThread exiting
      03-09-12 11:33:20:95 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: exited, exit code: 200
      03-09-12 11:33:23:45 UTC+8 INFO [admin1] [sn1] rg1-rn1: Service status: ERROR_RESTARTING 03-09-12 11:33:20
      03-09-12 11:33:20:96 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: restartProcess called, totalRestarts is 2, restartCount is -1
      03-09-12 11:33:20:96 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: startProcess
      03-09-12 11:33:21:47 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 12).
      03-09-12 11:33:21:46 UTC+8 INFO [rg1-rn1] RepNodeStateUpdateThread started
      03-09-12 11:33:21:52 UTC+8 INFO [rg1-rn1] Partition db cache mode: EVICT_LN
      03-09-12 11:33:21:63 UTC+8 INFO [rg1-rn1] JVM Runtime maxMemory (bytes): 99811328
      03-09-12 11:33:21:64 UTC+8 INFO [rg1-rn1] Non-default JE properties for environment: {je.rep.vlsn.logCacheSize=128, je.rep.envUnknownStateTimeout=10 s, je.tree.maxDelta=10, je.log.faultReadSize=4096, je.rep.vlsn.distance=1000000, je.env.runEvictor=true, je.cleaner.minUtilization=40, je.rep.replicaAckTimeout=5 s, je.cleaner.threads=2, je.evictor.lruOnly=false, je.rep.replayMaxOpenDbHandles=100, je.log.writeQueueSize=2097152, je.evictor.criticalPercentage=105, je.log.iteratorReadSize=1048576, je.log.numBuffers=16, je.nodeMaxEntries=128, je.evictor.nodesPerScan=200, je.rep.preHeartbeatTimeoutMs=5000000000, je.checkpointer.bytesInterval=200000000, je.cleaner.lazyMigration=false, je.txn.durability=write_no_sync,write_no_sync,simple_majority, je.cleaner.readSize=1048576, je.rep.txnRollbackLimit=10, je.log.fileCacheSize=2000, je.log.fileMax=1073741824}
      03-09-12 11:33:21:69 UTC+8 INFO [rg1-rn1] Starting RepNodeService
      03-09-12 11:33:21:82 UTC+8 INFO [rg1-rn1] Starting MonitorAgent
      03-09-12 11:33:21:85 UTC+8 INFO [rg1-rn1] RepNodeAdmin registered
      03-09-12 11:33:22:24 UTC+8 INFO [rg1-rn1] JE recovery: FIND_END_OF_LOG
      03-09-12 11:33:26:97 UTC+8 INFO [rg1-rn1] JE recovery: FIND_LAST_CKPT
      03-09-12 11:33:26:98 UTC+8 INFO [rg1-rn1] JE recovery: READ_DBMAP_INFO
      03-09-12 11:33:41:49 UTC+8 INFO [rg1-rn1] JE recovery: UNDO_DBMAP_RECORDS
      03-09-12 11:33:43:51 UTC+8 INFO [admin1] [rg1-rn1] Service status: STARTING 03-09-12 11:33:43
      03-09-12 11:34:18:67 UTC+8 INFO [sn1] sn1: Shutdown starting, stopping services
      03-09-12 11:34:18:67 UTC+8 INFO [sn1] Service status changed from RUNNING to STOPPING
      03-09-12 11:34:23:44 UTC+8 INFO [admin1] [sn1] Service status: STOPPING 03-09-12 11:34:18
      03-09-12 11:35:04:24 UTC+8 SEVERE [sn1] Cannot get RepNodeAdmin handle from Registry: rg1-rn1: Error unmarshaling return; nested exception is:
      java.io.EOFException
      03-09-12 11:35:04:24 UTC+8 WARNING [sn1] rg1-rn1: Exception stopping RepNode: Cannot get RepNodeAdmin handle from Registry: rg1-rn1: Error unmarshaling return; nested exception is:
      java.io.EOFException; nested exception is:
      java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
      java.io.EOFException
      03-09-12 11:35:04:24 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: stopProcess
      03-09-12 11:35:05:66 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: IOThread exception: Stream closed
      03-09-12 11:35:05:66 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: IOThread exiting
      03-09-12 11:36:05:16 UTC+8 INFO [admin1] [admin1] rg1-rn1: Service status: UNREACHABLE 03-09-12 11:36:05
        • 1. Re: Exception in thread "RMI
          Charles Lamb
          It seems that you need to increase the amount of memory you are allocating.

          Charles Lamb
          • 2. Re: Exception in thread "RMI
            Linda Lee-Oracle
            Note that you also have this message in your log:

            03-09-12 11:33:21:47 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 12).

            which implies that something else might be wrong with the JVM running on that machine. While 90M is a pretty small cache for a NoSQL node, the node should still come up, unless (a) the node has been operating with a much larger cache in the past, and the node was not shut down cleanly before or (b) The JVM heap is not really able to access all its memory.
            • 3. Re: Exception in thread "RMI
              915071
              Hello,
              I assign 90M which is computed by "java -jar je.jar DbCacheSize -records 1000000 -key 10 -data 20", and 90M is more than the max Bytes for Cache Size.

              bty, I want to know what is the difference between "Database Cache Size" and "internal Node Usage by Btree Level" ? How do I set the value for both?

              Edited by: foreversunyao on 2012-3-14 上午12:55
              • 4. Re: Exception in thread "RMI
                Charles Lamb
                foreversunyao wrote:
                Hello,
                I assign 90M which is computed by "java -jar je.jar DbCacheSize -records 1000000 -key 10 -data 20", and 90M is more than the max Bytes for Cache Size.

                bty, I want to know what is the difference between "Database Cache Size" and "internal Node Usage by Btree Level" ? How do I set the value for both?
                Could you please send any policy parameters that you are setting in your configuration step?

                The Internal Node Usage by Btree Level is an estimate of how much space the Internal Nodes of the B-tree will take in the cache. It is not a parameter -- only an estimate given by DbCacheSize. The Database Cache Size can be set with

                set policy "cacheSize=nnnnnn"

                in the NoSQL Database configuration.

                Charles
                • 5. Re: Exception in thread "RMI
                  Charles Lamb
                  Also, do you still have this exception in your logs?

                  03-09-12 11:33:21:47 UTC+8 INFO [sn1] rg1-rn1: ProcessMonitor: Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 12).
                  • 6. Re: Exception in thread "RMI
                    915071
                    Hello,
                    My policy information is belowed:
                    kv-> show policy
                    collectEnvStats=false
                    maxTrackedLatency=1000 MILLISECONDS
                    serviceLogFileCount=20
                    adminLogFileCount=20
                    latencyCeiling=0
                    configProperties=
                    adminLogFileLimit=4000000
                    collectorPollPeriod=20 SECONDS
                    serviceLogFileLimit=2000000
                    throughputFloor=0
                    javaMiscParams=-server -d32 -Xms150m -Xmx200m
                    loggingConfigProps=
                    eventExpiryAge=30 DAYS
                    cacheSize=100000000
                    statsInterval=10 SECONDS