1 Reply Latest reply: Mar 8, 2013 8:29 AM by Tom B RSS

    weblogic.rjvm.PeerGoneException while executing JMS schedler thread(WLS8.1)

    993391
      I have a web application deployed on weblogic server 8.1 SP4 and scheduler jobs implemented with standalone jms client which communicates with weblogic server.
      Multiple threads get scheduled after every hour throughout the day.

      I get the following error in server log file, occasionally after say 20 days

      weblogic.rjvm.PeerGoneException: ; nested exception is:
           java.io.EOFException
           at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:108)
           at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:290)
           at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:247)
           at com.netechinc.rflow.webapp.ejbs.session.rfImportSessionEJB_fywk0g_EOImpl_814_WLStub.importFile(Unknown Source)
           at com.netechinc.rflow.webapp.transmission.Scheduler.run(Scheduler.java:140)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.EOFException
           at weblogic.rjvm.t3.T3JVMConnection.endOfStream(T3JVMConnection.java:874)
           at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:583)
           at weblogic.socket.SocketMuxer.deliverEndOfStream(SocketMuxer.java:533)
           at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:691)
           at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:638)
           at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:282)
           at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
           at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)

      <BEA-000513> <Failure in heartbeat trigger for RJVM: 7876994111581326024C:172.30.29.84R:-6676648650237697870S:ucabclg1pp:[7001,7001,-1,-1,7001,-1,-1,0,0]:rflowdomain:rflowserver
      java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@15eb24c - id: '7876994111581326024C:172.30.29.84R:-6676648650237697870S:ucabclg1pp:[7001,7001,-1,-1,7001,-1,-1,0,0]:rflowdomain:rflowserver' connect time: 'Mon Feb 18 02:15:02 MST 2013'' has already been shut down.
      java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@15eb24c - id: '7876994111581326024C:172.30.29.84R:-6676648650237697870S:ucabclg1pp:[7001,7001,-1,-1,7001,-1,-1,0,0]:rflowdomain:rflowserver' connect time: 'Mon Feb 18 02:15:02 MST 2013'' has already been shut down
           at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1589)
           at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1538)
           at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:555)
           at weblogic.rjvm.RJVMImpl$HeartbeatChecker.trigger(RJVMImpl.java:1496)
           at weblogic.time.common.internal.ScheduledTrigger.run(ScheduledTrigger.java:243)
           at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
           at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
           at weblogic.time.common.internal.ScheduledTrigger.executeLocally(ScheduledTrigger.java:229)
           at weblogic.time.common.internal.ScheduledTrigger.execute(ScheduledTrigger.java:223)
           at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
           at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

      Also get the following exception for some threads

      Could not establish a connection with -6676648650237697870S:ucabclg1pp:[7001,7001,-1,-1,7001,-1,-1,0,0]:rflowdomain:rflowserver, java.io.IOException: Destination unreachable; nested exception is:
           java.io.IOException: Empty server reply; No available router to destination
      java.rmi.ConnectException: Could not establish a connection with -6676648650237697870S:ucabclg1pp:[7001,7001,-1,-1,7001,-1,-1,0,0]:rflowdomain:rflowserver, java.io.IOException: Destination unreachable; nested exception is:
           java.io.IOException: Empty server reply; No available router to destination
           at weblogic.rjvm.RJVMImpl.getOutputStream(RJVMImpl.java:376)
           at weblogic.rjvm.RJVMImpl.getRequestStream(RJVMImpl.java:551)
           at weblogic.rjvm.RJVMImpl.getOutboundRequest(RJVMImpl.java:683)
           at weblogic.rmi.internal.BasicRemoteRef.getOutboundRequest(BasicRemoteRef.java:90)
           at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:282)
           at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:247)
           at com.netechinc.rflow.webapp.ejbs.session.rfExportSessionEJB_gkn6xe_EOImpl_814_WLStub.export(Unknown Source)
           at com.netechinc.rflow.webapp.transmission.Scheduler.run(Scheduler.java:127)
           at java.lang.Thread.run(Thread.java:595)

      All the scheduled threads throw the above exception and scheduled job get scheduled to next interval after the exception.

      I have checked if its network failure, but was nt.

      how to fix this issue.

      Please advise, thanx
        • 1. Re: weblogic.rjvm.PeerGoneException while executing JMS schedler thread(WLS8.1)
          Tom B
          I don't see any JMS in the stack. The "Destination" in the exception in this case is not a JMS destination, but an IP address that an EJB stub is attempting to use. The exception may be normal -- perhaps your EJB code is trying to connect to a WebLogic Server that is down. If you think all servers are up, then I think you have two basic things to look into that might help (1) closing your JNDI context and looking up your EJB again, or (2) checking if your cluster is is healthy (I think there are FAQs for cluster troubleshooting - at least in more recent versions of the doc -- 8.1 is many years old now...).

          If you need further help, I recommend trying to post to an EJB newsgroup or some-such.

          Hope this helps,

          Tom

          c