This discussion is archived
7 Replies Latest reply: Nov 8, 2012 12:38 AM by 902568 RSS

RJVM has already been shutdown error while looking upto Foreign Server

902568 Newbie
Currently Being Moderated
Hi,

I am trying to set up 2 new servers in a clustered environment. In this I have configured a Foreign Server which is looking up to queues in different server.

The structure is like this: 1 Foreign Server with 2 Queues and 1 Connection factory with exactly same name as they present on the host server.
While starting the server, I am getting an exception:

<Nov 5, 2012 7:39:58 PM EET> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: ClonedQueueAnonymousWaiter is unable to connect to the JMS destination: MDB_QUEUE_RESPONSE. The Error was:
javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
     java.io.IOException: RJVM has already been shutdown]
     at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
     at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:470)
     at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:426)
     at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
     at javax.naming.InitialContext.lookup(InitialContext.java:392)
     at weblogic.deployment.jms.ForeignOpaqueReference.getReferent(ForeignOpaqueReference.java:221)
     at weblogic.jndi.internal.WLNamingManager.getObjectInstance(WLNamingManager.java:96)
     at weblogic.jndi.internal.ServerNamingNode.resolveObject(ServerNamingNode.java:377)
     at weblogic.jndi.internal.BasicNamingNode.resolveObject(BasicNamingNode.java:856)
     at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:209)
     at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254)
     at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
     at javax.naming.InitialContext.lookup(InitialContext.java:392)
     at weblogic.jms.common.CDS$2.run(CDS.java:486)
     at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
     at weblogic.jms.common.CrossDomainSecurityManager.runAs(CrossDomainSecurityManager.java:131)
     at weblogic.jms.common.CDS.lookupDestination(CDS.java:480)
     at weblogic.jms.common.CDS.lookupDDAndCalloutListener(CDS.java:345)
     at weblogic.jms.common.CDS.access$100(CDS.java:41)
     at weblogic.jms.common.CDS$DDListenerRegistrationTimerListener.timerExpired(CDS.java:193)
     at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
     at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
Caused by: java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
     java.io.IOException: RJVM has already been shutdown
     at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:360)
     at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
     at weblogic.jndi.internal.ServerNamingNode_1034_WLStub.lookup(Unknown Source)
     at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:423)
     ... 21 more
Caused by: java.io.IOException: RJVM has already been shutdown
     at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
     at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
     ... 24 more
Caused by: java.io.IOException: RJVM has already been shutdown
     at weblogic.rjvm.RJVMImpl.addPendingResponse(RJVMImpl.java:513)
     at weblogic.rjvm.RJVMImpl.send(RJVMImpl.java:900)
     at weblogic.rjvm.MsgAbbrevOutputStream.flushAndSend(MsgAbbrevOutputStream.java:394)
     at weblogic.rjvm.MsgAbbrevOutputStream.sendRecv(MsgAbbrevOutputStream.java:404)
     at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
     at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
     ... 24 more
>

Weblogic is installed in Solaris machine and welogic's version is 10.3.4.0 with Java version 1.6.0 update 33. I have checked the firewall, /etc/hosts but there are no issues with that.

The JNDI Connection URL is configure like: t3://xx.xxx.x.xx,xx.xxx.x.xx:7001/
I have tried giving like this also: t3://xx.xxx.x.xx:7001,xx.xxx.x.xx:7001/ or t3://hostname1,hostname2:7001/ or t3://hostname1:7001,hostname2:7001/
but nothing worked.

The foreign servers are accessible from my Application Server when I do the telnet on the port. There is no cross domain security required. Funny thing is I have a same set-up in two other application server (in cluster again) and there on configuring the Foreign server to same IPs, I am not facing any issues at all.

Can anyone please help me with this. Searched all over the place but have no clue regarding this. Many forums mentioned about firewall and etc/hosts but that is also not case it seems.

Regards,
Priyam
  • 1. Re: RJVM has already been shutdown error while looking upto Foreign Server
    Mohammed Rayan-Oracle Journeyer
    Currently Being Moderated
    You can apply the below debug flags to check t he RJVM Communication in detail and figure out what's going wrong and do apply it at both the side


    -Dweblogic.kernel.debug=true
    -Dweblogic.debug.DebugConnection=true
    -Dweblogic.StdoutDebugEnabled=true
    -Dweblogic.log.LogSeverity=Debug
    -Dweblogic.log.LoggerSeverity=Debug
  • 2. Re: RJVM has already been shutdown error while looking upto Foreign Server
    902568 Newbie
    Currently Being Moderated
    809364 wrote:
    You can apply the below debug flags to check t he RJVM Communication in detail and figure out what's going wrong and do apply it at both the side


    -Dweblogic.kernel.debug=true
    -Dweblogic.debug.DebugConnection=true
    -Dweblogic.StdoutDebugEnabled=true
    -Dweblogic.log.LogSeverity=Debug
    -Dweblogic.log.LoggerSeverity=Debug
    I added the debug as you mentioned and I get the below information there:

    <Nov 6, 2012 11:41:16 AM EET> <Debug> <RJVM> <BEA-000500> <RJVMFinder trying to find rjvm for t3://moon1,moon2:7001->moon1,moon2:[7001, -1, -1, -1, -1, -1, -1]->[]
    Active RJVMs:
    {-8968250770921317063S:10.160.1.115:[7001,-1,-1,-1,-1,-1,-1]:fcdb_domain:AdminServer on [10.160.1.115:7001]=weblogic.rjvm.RJVMImpl@4320d68d - id: '-8968250770921317063S:10.160.1.115:[7001,-1,-1,-1,-1,-1,-1]:fcdb_domain:AdminServer' connect time: 'Tue Nov 06 11:40:26 EET 2012'}
    Synonyms:
    {}>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Bootstrapping connection to: 'moon1/10.160.1.40:7001' using: 't3'>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Looking for a new connection for channel Default[t3]:t3(t3):venus1:7001:null:-1 to remote server 0B:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1,-1,-1] active connections: {}>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <New connection for protocol t3, remote address moon1/10.160.1.40, remote port 7001, over channel Default[t3]>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Added: 'Default[t3]:t3(t3):venus1:7001:null:-1' connection: 'weblogic.rjvm.t3.MuxableSocketT3$T3MsgAbbrevJVMConnection@7582aacf'. Connections list size 1>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Bootstrapping using a: 't3', connection to: 'moon1/10.160.1.40' on port: '7001'>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Created RJVM for: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1 on [10.160.1.40:7001]', Active RJVMs:
    {-8968250770921317063S:10.160.1.115:[7001,-1,-1,-1,-1,-1,-1]:fcdb_domain:AdminServer on [10.160.1.115:7001]=weblogic.rjvm.RJVMImpl@4320d68d - id: '-8968250770921317063S:10.160.1.115:[7001,-1,-1,-1,-1,-1,-1]:fcdb_domain:AdminServer' connect time: 'Tue Nov 06 11:40:26 EET 2012', -3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1 on [10.160.1.40:7001]=weblogic.rjvm.RJVMImpl@7bb8a8c6 - id: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1' connect time: 'Tue Nov 06 11:41:17 EET 2012'}
    Synonyms:
    {}>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Remote heartbeat: '60000', local heartbeat: '60000'>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Setting heartbeat for RJVM: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1' to: '60000' milliseconds>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Added: 'Default[t3]:t3(t3):venus1:7001:null:-1' connection: 'weblogic.rjvm.t3.MuxableSocketT3$T3MsgAbbrevJVMConnection@7582aacf'. Connections list size 1>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000517> <Error on connection: 'weblogic.rjvm.t3.MuxableSocketT3$T3MsgAbbrevJVMConnection@7582aacf'
    java.io.EOFException
    java.io.EOFException
         at weblogic.rjvm.t3.MuxableSocketT3.endOfStream(MuxableSocketT3.java:345)
         at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:826)
         at weblogic.socket.SocketMuxer.deliverEndOfStream(SocketMuxer.java:760)
         at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:941)
         at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:898)
         at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:126)
         at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
    >
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Signaling peer: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1' gone - weblogic.rjvm.PeerGoneException: ; nested exception is:
         java.io.EOFException>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Closed and removed: 't3' connection to: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1'. Connections list size 0>
    <Nov 6, 2012 11:41:17 AM EET> <Debug> <RJVM> <BEA-000500> <Closing: '-3712802650201056847S:10.160.1.40:[7001,-1,-1,-1,-1,-1,-1]:fcj_domain:ManagedServer1'>
    <Nov 6, 2012 11:41:17 AM EET> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: ClonedQueueAnonymousWaiter is unable to connect to the JMS destination: MDB_QUEUE_RESPONSE. The Error was:
    javax.naming.CommunicationException [Root exception is weblogic.rjvm.PeerGoneException: ; nested exception is:
         java.io.EOFException]
         at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
         at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:470)
         at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:426)
         at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
         at javax.naming.InitialContext.lookup(InitialContext.java:392)
         at weblogic.deployment.jms.ForeignOpaqueReference.getReferent(ForeignOpaqueReference.java:221)
         at weblogic.jndi.internal.WLNamingManager.getObjectInstance(WLNamingManager.java:96)
         at weblogic.jndi.internal.ServerNamingNode.resolveObject(ServerNamingNode.java:377)
         at weblogic.jndi.internal.BasicNamingNode.resolveObject(BasicNamingNode.java:856)
         at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:209)
         at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254)
         at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
         at javax.naming.InitialContext.lookup(InitialContext.java:392)
         at weblogic.jms.common.CDS$2.run(CDS.java:486)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.jms.common.CrossDomainSecurityManager.runAs(CrossDomainSecurityManager.java:131)
         at weblogic.jms.common.CDS.lookupDestination(CDS.java:480)
         at weblogic.jms.common.CDS.lookupDDAndCalloutListener(CDS.java:345)
         at weblogic.jms.common.CDS.access$100(CDS.java:41)
         at weblogic.jms.common.CDS$DDListenerRegistrationTimerListener.timerExpired(CDS.java:193)
         at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
         at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
    Caused by: weblogic.rjvm.PeerGoneException: ; nested exception is:
         java.io.EOFException
         at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
         at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
         at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
         at weblogic.jndi.internal.ServerNamingNode_1034_WLStub.lookup(Unknown Source)
         at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:423)
         ... 21 more
    Caused by: java.io.EOFException
         at weblogic.rjvm.t3.MuxableSocketT3.endOfStream(MuxableSocketT3.java:345)
         at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:826)
         at weblogic.socket.SocketMuxer.deliverEndOfStream(SocketMuxer.java:760)
         at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:941)
         at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:898)
         at weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:126)
         at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
    >

    Seems like Peer gone exception is the root cause of this. Any idea for this ?

    Regards,
    Priyam

    Edited by: Priyam Srivastava on Nov 6, 2012 2:49 AM
  • 3. Re: RJVM has already been shutdown error while looking upto Foreign Server
    Mohammed Rayan-Oracle Journeyer
    Currently Being Moderated
    The root cause of PeerGone is due to the EOF exception here,as we can see it from the debug log.

    Caused by: weblogic.rjvm.PeerGoneException: ; nested exception is:
    java.io.EOFException
    at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
    at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)


    You can debug it more from the socket & messaging & perspective by adding the additional logging to the list of the debugs,which are already in place to verify if its not an issue with the DNS name resolution or other issues

    -Dweblogic.debug.DebugMessaging=true
    -Dweblogic.debug.DebugMuxer=true
    -Dweblogic.debug.DebugMuxerDetail=true
    -Dweblogic.debug.DebugMuxerTimeout=true
    -Dweblogic.debug.DebugMuxerConnection=true
    -Dweblogic.debug.DebugMuxerException=true
  • 4. Re: RJVM has already been shutdown error while looking upto Foreign Server
    Kalyan Pasupuleti-Oracle Expert
    Currently Being Moderated
    Hi,

    I would suggest you enable tunneling for this issue to avoid Peer gone exception.

    make sure to enable http tunneling on all the server which are exists in your machine along with Admin server.

    and check again whether you get similar exception or not.

    Regards,
    Kal
  • 5. Re: RJVM has already been shutdown error while looking upto Foreign Server
    902568 Newbie
    Currently Being Moderated
    Hi,

    Indeed it was DNS name resolution issue. Though I added the entry in /etc/hosts in my application server but the same needs to be added in the foreign server as it was not able to resolve the DNS.
    The other way around what I found (w/o any entry in /etc/hosts) was that in Weblogic Console-->Server-->Managed Server-->General Tab-->Listen Address field I had given the hostname of my managed server. I changed it to IP address of my Managed server and then was able to connect to the Foreign Server. The More Info* section of Listen Address field says few things about giving DNS names:

    To resolve a DNS name to an IP address, WebLogic Server must be able to contact an appropriate DNS server or obtain the IP address mapping locally. Therefore, if you specify a DNS name for the listen address, you must either leave a port open long enough for the WebLogic Server instance to connect to a DNS server and cache its mapping or you must specify the IP address mapping in a local file. If you specify an IP address for ListenAddress and then a client request specifies a DNS name, WebLogic Server will attempt to resolve the DNS name, but if it cannot access DNS name mapping, the request will fail.


    Thanks a lot for your help !!

    Regards,
    Priyam

    Edited by: Priyam Srivastava on Nov 7, 2012 2:47 AM
  • 6. Re: RJVM has already been shutdown error while looking upto Foreign Server
    Mohammed Rayan-Oracle Journeyer
    Currently Being Moderated
    Good to know that the issue is resolved and just curious to know if you had added the socket debugs mentioned earlier and came to know as DNS issue from the logs?
  • 7. Re: RJVM has already been shutdown error while looking upto Foreign Server
    902568 Newbie
    Currently Being Moderated
    I didn't saw your last message until I resolved the issue and also there was technical glitch in Oracle Forums yesterday. I added the debugs that yo mentioned in your first comment. As I said that I already have a similar set-up which is working and this new set-up was not working. So I added the debugs in both set-ups and compared the Managed Server logs. And that's how I came to know about the issue.

    Thanks again !!!

    Regards,
    Priyam

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points