This discussion is archived
6 Replies Latest reply: Feb 7, 2012 12:27 PM by Charles Lamb RSS

java.net.ConnectException: Connection refused

Kristophe Newbie
Currently Being Moderated
Another one of these connection errors but I have no log file to look at.

I was doing well too, managing to go thru the admin setup steps including DCenter creation on the CENTOSBOX node, until I got a connection error trying to get a SNA node going on another connected box when I ran into an issue about localhost admin server vs fully qualified domain name, so I started from scratch and now can't get anything to work, including on CENTOSBOX where it worked before.

I am running ntp on all nodes and I started rmiregistry to no avail

There is a paucity of info to look at here but if anyone has any suggestios I'd appreciate it.
I have firewall disabled
Thanks


CENTOSBOX:/home/oracle/nosql/kv-1.2.123>java -jar lib/kvstore-1.2.123.jar runadmin -port 5000 -host centosbox.mydomain.local
Exception in thread "main" java.rmi.ConnectException: Connection refused to host: centosbox.mydomain.local; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at oracle.kv.impl.util.registry.RegistryUtils.getAdmin(RegistryUtils.java:197)
at oracle.kv.impl.admin.client.KVAdmin.connect(KVAdmin.java:883)
at oracle.kv.impl.admin.client.KVAdmin.main(KVAdmin.java:2131)
at oracle.kv.impl.util.KVStoreMain$6.run(KVStoreMain.java:188)
at oracle.kv.impl.util.KVStoreMain.main(KVStoreMain.java:319)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 9 more
CENTOSBOX:/home/oracle/nosql/kv-1.2.123>cat /etc/hosts
127.0.0.1 localhost localhost.localdomain
#127.0.0.1 centosbox.mydomain.local centosbox
::1 localhost6.localdomain6 localhost6
192.168.2.188 centosbox.mydomain.local centosbox
192.168.2.144 zareason.mydomain.local zareason
192.168.2.119 compaq.mydomain.local compaq
CENTOSBOX:/home/oracle/nosql/kv-1.2.123>
  • 1. Re: java.net.ConnectException: Connection refused
    Kristophe Newbie
    Currently Being Moderated
    I got around this by restoring my /etc/hosts to

    127.0.0.1 localhost localhost.localdomain
    *127.0.0.1 centosbox.mydomain.local centosbox**
    ::1 localhost6.localdomain6 localhost6
    192.168.2.188 centosbox.mydomain.local centosbox
    192.168.2.144 zareason.mydomain.local zareason
    192.168.2.119 compaq.mydomain.local compaq

    I had the *127.0.0.1 centosbox.mydomain.local centosbox** in there to satisfy 11g Fusion Middleware that I also have running on this box and I removed it because Nosql was complaining about it or so I thought.

    Anyway I'll report that error when it undoubtedly shows up
  • 2. Re: java.net.ConnectException: Connection refused
    Kristophe Newbie
    Currently Being Moderated
    Different error this time. I created an SNA and Admin service on Centosbox - worked fine
    Then trying to deploy a storage node on zareason.mydomain.local using the admin web interface I get this, whether rmiregistry is running or not (is this required?)


    in createPlan: Oracle NoSQL DB 11gR2.1.2.123 oracle.kv.impl.fault.OperationFaultException: Plan 6[Plan-5] finished in state ERROR. Problem during plan execution: snaService

    caused by:
    oracle.kv.impl.admin.AdminFaultException
    Oracle NoSQL DB 11gR2.1.2.123 oracle.kv.impl.fault.OperationFaultException: Plan 6[Plan-5] finished in state ERROR. Problem during plan execution: snaService
    Oracle NoSQL DB 11gR2.1.2.123 oracle.kv.impl.fault.OperationFaultException: Plan 6[Plan-5] finished in state ERROR. Problem during plan execution: snaService oracle.kv.impl.fault.OperationFaultException: Plan 6[Plan-5] finished in state ERROR. Problem during plan execution: snaService
         at sun.rmi.registry.RegistryImpl.lookup(RegistryImpl.java:114)
         at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
         at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:390)
         at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:248)
         at sun.rmi.transport.Transport$1.run(Transport.java:159)
         at java.security.AccessController.doPrivileged(Native Method)
         at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
         at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
         at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
         at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
         at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:359)
         at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
         at oracle.kv.impl.util.registry.RegistryUtils.getStorageNodeAgent(RegistryUtils.java:243)
         at oracle.kv.impl.admin.plan.task.DeploySN.call(DeploySN.java:94)
         at oracle.kv.impl.admin.plan.task.DeploySN.call(DeploySN.java:35)
         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: java.rmi.NotBoundException: snaService
         ... 25 more

         at oracle.kv.impl.admin.AdminServiceFaultHandler.getThrowException(AdminServiceFaultHandler.java:84)
         at oracle.kv.impl.fault.ProcessFaultHandler.rethrow(ProcessFaultHandler.java:205)
         at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:144)
         at oracle.kv.impl.admin.CommandServiceImpl.executePlan(CommandServiceImpl.java:234)
         at oracle.kv.impl.admin.CommandServiceAPI.executePlan(CommandServiceAPI.java:162)
         at oracle.kv.impl.admin.webapp.server.AdminUIServiceImpl.createPlan(AdminUIServiceImpl.java:554)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
         at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
         at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
         at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:538)
         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:476)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:517)
         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:934)
         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:404)
         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:869)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
         at org.eclipse.jetty.server.Server.handle(Server.java:341)
         at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:589)
         at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1065)
         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:823)
         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
         at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
         at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:515)
         at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
         at java.lang.Thread.run(Thread.java:662)
    (from Plans.PlanCallback)
  • 3. Re: java.net.ConnectException: Connection refused
    Linda Lee Journeyer
    Currently Being Moderated
    Chris,

    It seems likely that the SNA has not started up cleanly on the new node (zareason.mydomain.local). There is no need for you to invoke rmiregistry explicitly, as the NoSQL DB startup code should do that.

    I think the first thing to do is to look in any available log files on that machine to get a hint as to what could have gone wrong at startup. After the node is up, the administrative monitoring system will funnel problems and warnings to the administrative console, for easier perusal. But at SNA startup, since the node is not fully registered with the Admin service, some amount of information is only available on files local to that machine.

    Specifically, look in <KVROOT>/snaboot_*.log and <KVROOT>/<storename>/log/sn*.log on that node for trace messages when the SNA is starting up. You may want to compare the boot sequence to that of the node that has successfully come up. Hopefully that will provide a clue about any problems from the SNA startup.

    Using the ping and jps -m command described in http://docs.oracle.com/cd/NOSQL/html/AdminGuide/troubleshooting.html may also provide some useful clues.

    Linda
  • 4. Re: java.net.ConnectException: Connection refused
    Kristophe Newbie
    Currently Being Moderated
    Thanks Linda. I blew everything away and started again. Got as far as (apart from nodenames I'm using the demo names - BostonPool and so on) deploying the store, the final step when it blew up, due to the issue with the loopback adapter/localhost referred to earlier showed up

    Looks like I cannot coexist nosql with 11g fusion middleware which is probably not a problem in the real world because 11G fusion WLS wants a loopback 127.0.0.1 (fully qualified domain name)


    kv-> plan -execute -name "Deploy the store" deploy-store BostonPool 3 300
    Exception while executing command plan: Oracle NoSQL DB 11gR2.1.2.123 oracle.kv.impl.fault.OperationFaultException: Plan 6[Deploy the store] finished in state ERROR. DeployOneRepNode of rg1-rn2 on sn2/zareason.mydomain.local:5000 [RUNNING] failed.: Oracle NoSQL DB 11gR2.1.2.123 java.lang.IllegalStateException: Failed to attach to RepNodeService for rg1-rn2, see log, /home/oracle/nosql/data/KVROOT/mdistore/log/rg1-rn2*.log, on host zareason.mydomain.local for more information.



    bit of log is:-

    java.lang.IllegalArgumentException: zareason.mydomain.local/127.0.0.1:5010 the address associated with this node, is a loopback address. It conflicts with an existing use, by a different node of the address:centosbox.mydomain.local/192.168.2.188:5011 which is not a loopback address. Such mixing of addresses within a group is not allowed, since the nodes will not be able to communicate with each other.
    at com.sleepycat.je.rep.impl.node.RepNode.checkLoopbackAddresses(RepNode.java:1107)
    at com.sleepycat.je.rep.impl.node.RepNode.queryGroupForMaster(RepNode.java:1122)
    at com.sleepycat.je.rep.impl.node.RepNode.findMaster(RepNode.java:1054)
    at com.sleepycat.je.rep.impl.node.RepNode.startup(RepNode.java:723)
    at com.sleepycat.je.rep.impl.node.RepNode.joinGroup(RepNode.java:1542)
    at com.sleepycat.je.rep.impl.RepImpl.joinGroup(RepImpl.java:459)
    at com.sleepycat.je.rep.ReplicatedEnvironment.joinGroup(ReplicatedEnvironment.java:414)
    at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:467)
    at com.sleepycat.je.rep.ReplicatedEnvironment.<init>(ReplicatedEnvironment.java:332)
    at oracle.kv.impl.rep.RepEnvHandleManager.openEnv(RepEnvHandleManager.java:361)
    at oracle.kv.impl.rep.RepEnvHandleManager.renewRepEnv(RepEnvHandleManager.java:305)
    at oracle.kv.impl.rep.RepNode.startup(RepNode.java:509)
    at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:292)
    at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:250)
    at oracle.kv.impl.sna.ManagedRepNode$1.execute(ManagedRepNode.java:157)
    at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:127)
    at oracle.kv.impl.sna.ManagedRepNode.start(ManagedRepNode.java:152)
    at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:519)

    Edited by: ChrisEss on Feb 6, 2012 3:53 PM
  • 5. Re: java.net.ConnectException: Connection refused
    Kristophe Newbie
    Currently Being Moderated
    I figured out I only needed the loopback 127.0.0.1 pointing to the fully qualifed domain name for one box, the only one that needs the setting for Fusion Middleware, so I got rid of the loopback entry in /etc/hosts for 2 nodes, zareason and compaq, and started over.

    This time it failed earlier in the process with

    CENTOSBOX:/home/oracle/nosql/kv-1.2.123>java -jar lib/kvstore-1.2.123.jar runadmin -port 5000 -host centosbox.mydomain.local
    kv-> configure mdistore
    kv-> plan -execute -name "Deploy DC" deploy-datacenter "Boston" "Savvis"
    1
    kv-> plan -execute -name "Deploy n01" deploy-sn 1 centosbox.mydomain.local 5000 "comment"
    2
    kv-> plan -execute -name "Deploy admin" deploy-admin 1 5001
    3
    kv-> addpool BostonPool
    kv-> joinpool BostonPool 1
    AllStorageNodes: sn1
    BostonPool: sn1

    kv-> plan -execute -name "Deploy n02" deploy-sn 1 zareason.mydomain.local 5000
    Exception while executing command plan: Oracle NoSQL DB 11gR2.1.2.123 oracle.kv.impl.fault.OperationFaultException: Plan 4[Deploy n02] finished in state ERROR. Problem during plan execution: Oracle NoSQL DB 11gR2.1.2.123 java.lang.IllegalStateException: Register: Cannot mix loopback and non-loopback addresses in the same store. The store value is configured to use loopback addresses but storage node zareason.mydomain.local:5000 is not a loopback address.
    kv->
  • 6. Re: java.net.ConnectException: Connection refused
    Charles Lamb Pro
    Currently Being Moderated
    I'm not sure how to solve your problem -- we don't really have enough information and it seems that things are a moving target with your host names, ipaddrs, loopbacks, etc.

    Perhaps the easiest thing to do is to just start from scratch (i.e. rm kvroot, kill -9 all the relevant processes) and instead of using host names for each of your hosts, just use straight ipv4 ipaddrs (and not 127.0.0.1 loopback addresses!).

    Charles Lamb

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points