6 Replies Latest reply: Feb 9, 2012 8:58 PM by 915071 RSS

    Exception in thread "main" java.rmi.NotBoundException: commandService

    915071
      Hello,
      I want to deploy two SNs on one machine, and I chose different port(5030,5031), when I run "runadmin", the exceptions happened.

      [sunyao@datanode1:~/oracle_nosql/tools]$ jps -m
      25392 Jps -m
      25145 kvstore-1.2.123.jar start -root KVROOT_2
      23099 kvstore-1.2.123.jar start -root KVROOT
      23265 ManagedService -root KVROOT/mystore/sn2 -store mystore -class RepNode -service rg1-rn2
      25220 ManagedService -root KVROOT_2/mystore2/sn1 -store mystore2 -class Admin -service admin1
      [sunyao@datanode1:~/oracle_nosql/tools]$ java -jar kv-1.2.123/lib/kvstore-1.2.123.jar ping -port 5030 -host datanode1
      SNA at hostname: datanode1 registry port: 5030 has no available Admins or RNs registered.
      [sunyao@datanode1:~/oracle_nosql/tools]$ java -jar kv-1.2.123/lib/kvstore-1.2.123.jar runadmin -port 5030 -host datanode1
      Exception in thread "main" java.rmi.NotBoundException: commandService
      at sun.rmi.registry.RegistryImpl.lookup(RegistryImpl.java:106)
      at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
      at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:386)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:250)
      at sun.rmi.transport.Transport$1.run(Transport.java:159)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
      at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
      at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:359)
      at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
      at oracle.kv.impl.util.registry.RegistryUtils.getAdmin(RegistryUtils.java:197)
      at oracle.kv.impl.admin.client.KVAdmin.connect(KVAdmin.java:883)
      at oracle.kv.impl.admin.client.KVAdmin.main(KVAdmin.java:2131)
      at oracle.kv.impl.util.KVStoreMain$6.run(KVStoreMain.java:188)
      at oracle.kv.impl.util.KVStoreMain.main(KVStoreMain.java:319)
        • 1. Re: Exception in thread "main" java.rmi.NotBoundException: commandService
          guy.k.hillyer
          This error message suggests that while you have a StorageNode listening on port 5030, there is no Admin associated with that StorageNode. Guessing that the configuration in KVROOT uses port number 5030, and KVROOT_2 uses port number 5031, your Admin is probably available on port 5031. I can't tell much more without knowing the contents of the config.xml files.

          It seems unusual for the two storage nodes to be using consecutive port numbers, since you have to allocate several ports for each SN, but it's not impossible to have a configuration like that. I would double-check to be sure that the two storage nodes are using distinct sets of ports.
          • 2. Re: Exception in thread "main" java.rmi.NotBoundException: commandService
            Linda Lee
            In addition to Guy's comments, you may want to review the FAQ, at http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosqldb-faq-518364.html. In particular, the entries:

            - explaining the use of ports in NoSQL DB: http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosqldb-faq-518364.html#WhatarealltheseportsWhendoIusewhichone
            - how to create multiple SNs on one machine: http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosqldb-faq-518364.html#HowtocreatemultipleStorageNodesonthesamemachine

            may be useful.

            You may also want to try the java -jar kvstore.jar ping -host <hostname> -port <port> on various nodes to get a sense of what you've managed to deploy, and which services are up.
            • 3. Re: Exception in thread "main" java.rmi.NotBoundException: commandService
              915071
              hi ,

              I have started two admin service in one group:
              but one admin service can't access from 5001, and the log is below:
              [sunyao@datanode1:~/oracle_nosql/tools/KVROOT/mystore/log]$ cat admin2_0.log
              02-09-12 17:48:46:696 CST INFO [admin2] Starting service process: admin2, Java command line arguments: [-XX:+DisableExplicitGC, -Xms96M, -Xmx96M]
              02-09-12 17:48:46:720 CST INFO [admin2] Starting AdminService
              02-09-12 17:48:46:881 CST INFO [admin2] Initializing Admin for store: mystore
              02-09-12 17:48:46:883 CST INFO [admin2] Created new admin environment dir: KVROOT/mystore/sn2/admin2/env
              02-09-12 17:48:46:884 CST INFO [admin2] JVM Runtime maxMemory (bytes): 96468992
              02-09-12 17:48:46:889 CST INFO [admin2] Non-default JE properties for environment: {je.rep.vlsn.logCacheSize=128, je.rep.envUnknownStateTimeout=10 s, je.tree.maxDelta=10, je.log.faultReadSize=4096, je.rep.vlsn.distance=1000000, je.env.runEvictor=true, je.cleaner.minUtilization=40, je.rep.replicaAckTimeout=5 s, je.cleaner.threads=2, je.evictor.lruOnly=false, je.rep.replayMaxOpenDbHandles=100, je.log.writeQueueSize=2097152, je.evictor.criticalPercentage=105, je.log.iteratorReadSize=1048576, je.log.numBuffers=16, je.nodeMaxEntries=128, je.evictor.nodesPerScan=200, je.rep.preHeartbeatTimeoutMs=5000000000, je.checkpointer.bytesInterval=200000000, je.cleaner.lazyMigration=false, je.txn.durability=write_no_sync,write_no_sync,simple_majority, je.cleaner.readSize=1048576, je.rep.txnRollbackLimit=10, je.log.fileCacheSize=2000, je.log.fileMax=1073741824}
              02-09-12 17:48:48:390 CST INFO [admin2] State change event: Thu Feb 09 17:48:47 CST 2012, State: REPLICA, Master: 1
              02-09-12 17:48:48:852 CST INFO [admin2] Shutting down monitor collector
              02-09-12 17:48:48:853 CST INFO [admin2] Replicated environment handle established. Cache size: 57881395, State: REPLICA
              02-09-12 17:48:48:854 CST INFO [admin2] Starting Command service on rmi://datanode1:5000/commandService
              02-09-12 17:48:48:950 CST INFO [admin2] Starting Web service on port 5001
              02-09-12 17:48:49:183 CST INFO [admin2] jetty-7.4.0.v20110414
              02-09-12 17:48:49:350 CST INFO [admin2] Extract jar:file:/home/sunyao/oracle_nosql/tools/kv-1.2.123/lib/kvstore-1.2.123.jar!/war/KVAdminUI/ to /tmp/jetty-0.0.0.0-5001-KVAdminUI-_-any-/webapp
              02-09-12 17:48:50:65 CST INFO [admin2] NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
              02-09-12 17:48:50:120 CST INFO [admin2] started o.e.j.w.WebAppContext{,file:/tmp/jetty-0.0.0.0-5001-KVAdminUI-_-any-/webapp/},jar:file:/home/sunyao/oracle_nosql/tools/kv-1.2.123/lib/kvstore-1.2.123.jar!/war/KVAdminUI
              02-09-12 17:48:50:201 CST INFO [admin2] Started SelectChannelConnector@0.0.0.0:5001 STARTING
              02-09-12 17:48:50:203 CST INFO [admin2] Started AdminService
              02-09-12 17:49:04:408 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 17:49:24:427 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 17:49:28:432 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 17:52:16:974 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 17:52:29:844 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 17:54:03:343 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:00:48:854 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:02:10:25 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:02:55:400 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:02:59:708 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:03:02:127 CST INFO [admin2] Redirect to master at http://namenode:5001
              02-09-12 18:03:05:644 CST INFO [admin2] Redirect to master at http://namenode:5001

              namenode is the hostname of 192.168.7.200, and I can access the admin service from http://192.168.7.200:5001
              and the topology is
              Pinging components of store mystore based upon topology sequence #612
              mystore comprises 300 partitions and 3 Storage Nodes
              Storage Node [sn1] on namenode:5000 Datacenter: Boston [dc1] Status: RUNNING Ver: 11gR2.1.2.123
              Rep Node [rg1-rn1] Status: RUNNING,REPLICA at sequence number: 625 haPort: 5011
              Storage Node [sn2] on datanode1:5000 Datacenter: Boston [dc1] Status: RUNNING Ver: 11gR2.1.2.123
              Rep Node [rg1-rn2] Status: RUNNING,REPLICA at sequence number: 625 haPort: 5010
              Storage Node [sn3] on datanode2:5000 Datacenter: Boston [dc1] Status: RUNNING Ver: 11gR2.1.2.123
              Rep Node [rg1-rn3] Status: RUNNING,MASTER at sequence number: 625 haPort: 5010

              Edited by: 912068 on 2012-2-9 上午2:06


              and when I shutdown namenode SN and admin service , and access 185:5001, the exceptions are belowed:

              Error
              Dismiss
              The Admin instance is unable to service this request, as it is not the master Admin instance. Furthermore, its state is such that it does not know the master Admin's forwarding address.

              Edited by: 912068 on 2012-2-9 上午2:24
              • 4. Re: Exception in thread "main" java.rmi.NotBoundException: commandService
                guy.k.hillyer
                If I understand you correctly, what you have described is normal behavior. It seems you have two Admins running. The one on namenode is the master, while the one on datanode1 is a replica. The Admin on datanode1 cannot service requests, because it is not the master. When runadmin tries to connect to the Admin on datanode1, the connection is automatically forwarded to the master Admin on namenode. That is shown by the log messages that say "Redirect to master".

                When you shut down namenode, the Admin mastership cannot fail over to datanode1, because it can't form a majority. That is shown by the message about "The Admin instance is unable to service this request, as it is not the master Admin instance. Furthermore..."

                In the FAQ, we recommend that you deploy at least three Admin replicas. This is the reason for that recommendation. If you have three replicas, and the master goes down, the remaining two can form a majority and choose a new master. If there are only two Admins, and one of them shuts down, then the remaining one cannot form a majority.

                http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosqldb-faq-518364.html#HowmanyAdminservicesshouldIdeploy
                • 5. Re: Exception in thread "main" java.rmi.NotBoundException: commandService
                  Charles Lamb
                  Just to clarify, when Guy says you should have three replicas, he means 2 replicas and 1 master.

                  Charles Lamb