8 Replies Latest reply on Feb 23, 2001 10:39 PM by 3004

    BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is not the secondary ]

    3004

                We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                
                we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                Thanx in advance
                Saurabh.
                this is the Exception we are getting :
                ========================================
                Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                at java.lang.Throwable.fillInStackTrace(Native Method)
                at java.lang.Throwable.fillInStackTrace(Compiled Code)
                at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                at weblogic.kernel.ExecuteThread.run(Compiled Code)
                --------------- nested within: ------------------
                weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                at java.lang.Throwable.fillInStackTrace(Native Method)
                at java.lang.Throwable.fillInStackTrace(Compiled Code)
                at java.lang.Throwable.<init>(Compiled Code)
                at java.lang.Error.<init>(Error.java:50)
                at weblogic.utils.NestedError.<init>(NestedError.java:23)
                at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                at weblogic.kernel.ExecuteThread.run(Compiled Code)
                
                
                
        • 1. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is not the secondary ]
          3004

                    "Saurabh" <SaurabhD@zeneb.com> wrote:
                    >
                    > We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                    > I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                    >
                    > we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem ..
                    
                    
                    No problems regarding this till NOW.
                    
                    
                    or there is something wrong going on with this exception with other people.
                    >i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                    >Thanx in advance
                    >Saurabh.
                    >this is the Exception we are getting :
                    >========================================
                    >Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                    >weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                    > at java.lang.Throwable.fillInStackTrace(Native Method)
                    > at java.lang.Throwable.fillInStackTrace(Compiled Code)
                    > at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                    > at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                    > at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                    > at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                    > at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                    > at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                    > at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                    > at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                    > at weblogic.kernel.ExecuteThread.run(Compiled Code)
                    >--------------- nested within: ------------------
                    >weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                    >[weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                    > at java.lang.Throwable.fillInStackTrace(Native Method)
                    > at java.lang.Throwable.fillInStackTrace(Compiled Code)
                    > at java.lang.Throwable.<init>(Compiled Code)
                    > at java.lang.Error.<init>(Error.java:50)
                    > at weblogic.utils.NestedError.<init>(NestedError.java:23)
                    > at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                    > at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                    > at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                    > at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                    > at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                    > at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                    > at weblogic.kernel.ExecuteThread.run(Compiled Code)
                    >
                    >
                    
                    
          • 2. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is not the secondary ]
            3004
            The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                      
                      Typically you see this under load if your heap size is not set right. If you turn on cluster debugging, you will see a Timed Out Server message and after that you will start seeing BSE.
                      Also turning on verbose gc will help too.
                      
                      Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                      
                      Hope you are using latest service pack.
                      
                      - Prasad
                      
                      Saurabh wrote:
                      
                      > We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                      > I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                      >
                      > we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                      > i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                      > Thanx in advance
                      > Saurabh.
                      > this is the Exception we are getting :
                      > ========================================
                      > Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                      > weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                      > at java.lang.Throwable.fillInStackTrace(Native Method)
                      > at java.lang.Throwable.fillInStackTrace(Compiled Code)
                      > at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                      > at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                      > at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                      > at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                      > at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                      > at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                      > at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                      > at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                      > at weblogic.kernel.ExecuteThread.run(Compiled Code)
                      > --------------- nested within: ------------------
                      > weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                      > [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                      > at java.lang.Throwable.fillInStackTrace(Native Method)
                      > at java.lang.Throwable.fillInStackTrace(Compiled Code)
                      > at java.lang.Throwable.<init>(Compiled Code)
                      > at java.lang.Error.<init>(Error.java:50)
                      > at weblogic.utils.NestedError.<init>(NestedError.java:23)
                      > at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                      > at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                      > at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                      > at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                      > at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                      > at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                      > at weblogic.kernel.ExecuteThread.run(Compiled Code)
                      >
                      >
                      
                      --
                      Cheers
                      
                      - Prasad
                      
                      
                      
            • 3. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
              3004

                        today i found that i am also having the same problem:)))
                        
                        Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                        >The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                        >
                        >Typically you see this under load if your heap size is not set right.
                        
                        could you be more detailed regarding RIGHT?
                        
                        If you turn on cluster debugging
                        
                        what properties i have to set to what?
                        
                        , you will see a Timed Out Server message and after that you will start seeing BSE.
                        
                        Whats BSE?
                        
                        >Also turning on verbose gc will help too.
                        >
                        >Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                        >
                        >Hope you are using latest service pack.
                        
                        i am using WLS5.1 sp6
                        
                        >
                        >- Prasad
                        >
                        >Saurabh wrote:
                        >
                        >> We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                        >> I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                        >>
                        >> we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                        >> i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                        >> Thanx in advance
                        >> Saurabh.
                        >> this is the Exception we are getting :
                        >> ========================================
                        >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                        >> weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                        >> at java.lang.Throwable.fillInStackTrace(Native Method)
                        >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                        >> at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                        >> at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                        >> at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                        >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                        >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                        >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                        >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                        >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                        >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                        >> --------------- nested within: ------------------
                        >> weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                        >> [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                        >> at java.lang.Throwable.fillInStackTrace(Native Method)
                        >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                        >> at java.lang.Throwable.<init>(Compiled Code)
                        >> at java.lang.Error.<init>(Error.java:50)
                        >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                        >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                        >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                        >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                        >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                        >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                        >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                        >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                        >>
                        >>
                        >
                        >--
                        >Cheers
                        >
                        >- Prasad
                        >
                        >
                        
                        
              • 4. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
                3004
                shivu wrote:
                          
                          > today i found that i am also having the same problem:)))
                          >
                          > Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                          > >The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                          > >
                          > >Typically you see this under load if your heap size is not set right.
                          >
                          > could you be more detailed regarding RIGHT?
                          
                          Too much or too little could cause problems. You have experiment, to set the right heap size to handle load on the servers. You have to use tool like load runner to figure out the thread count, heap size to over come this problem.
                          
                          
                          >
                          > If you turn on cluster debugging
                          >
                          > what properties i have to set to what?
                          
                          weblogic.debug.cluster=1
                          
                          >
                          >
                          > , you will see a Timed Out Server message and after that you will start seeing BSE.
                          >
                          > Whats BSE?
                          >
                          
                          BadStatusException.
                          
                          
                          
                          
                          >
                          > >Also turning on verbose gc will help too.
                          > >
                          > >Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                          > >
                          > >Hope you are using latest service pack.
                          >
                          > i am using WLS5.1 sp6
                          >
                          > >
                          > >- Prasad
                          > >
                          > >Saurabh wrote:
                          > >
                          > >> We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                          > >> I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                          > >>
                          > >> we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                          > >> i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                          > >> Thanx in advance
                          > >> Saurabh.
                          > >> this is the Exception we are getting :
                          > >> ========================================
                          > >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                          > >> weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                          > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                          > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                          > >> at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                          > >> at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                          > >> at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                          > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                          > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                          > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                          > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                          > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                          > >> --------------- nested within: ------------------
                          > >> weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                          > >> [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                          > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                          > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                          > >> at java.lang.Throwable.<init>(Compiled Code)
                          > >> at java.lang.Error.<init>(Error.java:50)
                          > >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                          > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                          > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                          > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                          > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                          > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                          > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                          > >>
                          > >>
                          > >
                          > >--
                          > >Cheers
                          > >
                          > >- Prasad
                          > >
                          > >
                          
                          
                • 5. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
                  3004

                            thanks a lot...
                            
                            will check with the debugs/new values...
                            
                            >Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                            >shivu wrote:
                            >
                            >> today i found that i am also having the same problem:)))
                            >>
                            >> Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                            >> >The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                            >> >
                            >> >Typically you see this under load if your heap size is not set right.
                            >>
                            >> could you be more detailed regarding RIGHT?
                            >
                            >Too much or too little could cause problems. You have experiment, to set the right heap size to handle load on the servers. You have to use tool like load runner to figure out the thread count, heap size to over come this problem.
                            >
                            >
                            >>
                            >> If you turn on cluster debugging
                            >>
                            >> what properties i have to set to what?
                            >
                            >weblogic.debug.cluster=1
                            >
                            >>
                            >>
                            >> , you will see a Timed Out Server message and after that you will start seeing BSE.
                            >>
                            >> Whats BSE?
                            >>
                            >
                            >BadStatusException.
                            >
                            >
                            >
                            >
                            >>
                            >> >Also turning on verbose gc will help too.
                            >> >
                            >> >Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                            >> >
                            >> >Hope you are using latest service pack.
                            >>
                            >> i am using WLS5.1 sp6
                            >>
                            >> >
                            >> >- Prasad
                            >> >
                            >> >Saurabh wrote:
                            >> >
                            >> >> We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                            >> >> I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                            >> >>
                            >> >> we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                            >> >> i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                            >> >> Thanx in advance
                            >> >> Saurabh.
                            >> >> this is the Exception we are getting :
                            >> >> ========================================
                            >> >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                            >> >> weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                            >> >> at java.lang.Throwable.fillInStackTrace(Native Method)
                            >> >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                            >> >> at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                            >> >> at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                            >> >> at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                            >> >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                            >> >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                            >> >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                            >> >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                            >> >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                            >> >> --------------- nested within: ------------------
                            >> >> weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                            >> >> [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                            >> >> at java.lang.Throwable.fillInStackTrace(Native Method)
                            >> >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                            >> >> at java.lang.Throwable.<init>(Compiled Code)
                            >> >> at java.lang.Error.<init>(Error.java:50)
                            >> >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                            >> >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                            >> >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                            >> >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                            >> >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                            >> >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                            >> >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                            >> >>
                            >> >>
                            >> >
                            >> >--
                            >> >Cheers
                            >> >
                            >> >- Prasad
                            >> >
                            >> >
                            >
                            
                            
                  • 6. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
                    3004

                              
                              Thanx Shivu & Prasad
                              I have put the comments below :
                              Prasad Peddada wrote:
                              
                              > shivu wrote:
                              >
                              > > today i found that i am also having the same problem:)))
                              > >
                              > > Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                              > > >The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                              
                              ### timing out means unable to send responce or it is indicating some Exception which could crash the server.
                              
                              > > >
                              > > >Typically you see this under load if your heap size is not set right.
                              > >
                              > > could you be more detailed regarding RIGHT?
                              >
                              > Too much or too little could cause problems. You have experiment, to set the right heap size to handle load on the servers. You have to use tool like load runner to figure out the thread count, heap size to over come this problem.
                              
                              > ## I noticed some some suggestion which says that if the GC will take more some specified period of time then this could cause this problem. This obviosly pointing not to have very large heapsize.
                              
                              on the contray using small size means that GC will be dome quite frequently. Can you give an idea what % of available memory on average should be used for heap in this case for weblogic server.
                              
                              > >
                              > > If you turn on cluster debugging
                              > >
                              > > what properties i have to set to what?
                              > weblogic.debug.cluster=1
                              > >
                              > >
                              > > , you will see a Timed Out Server message and after that you will start seeing BSE.
                              > >
                              >
                              
                              ## what specific message comes for this. does it notify when it removes a WLS from cluster. also any inputs on what could cause the running server to crash. Is Application has any restriction on normal coding. (likewhat will happen if do not clustering
                              specific propertise for stateless EJBs).
                              
                              > > Whats BSE?
                              > >
                              >
                              > BadStatusException.
                              >
                              > >
                              > > >Also turning on verbose gc will help too.
                              > > >
                              > > >Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                              > > >
                              
                              ## we see this problem on QA environment which i believe couldn't said to have heavy load and traffic.
                              
                              > > >Hope you are using latest service pack.
                              > >
                              > > i am using WLS5.1 sp6
                              > >
                              > > >
                              > > >- Prasad
                              > > >
                              
                              ## Is installation of web server and WLS on the same m/c could make any impact on here (as we are doing that). if yes please let me know why ???thanks,
                              Saurabh.
                              
                              > > >Saurabh wrote:
                              > > >
                              > > >> We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                              > > >> I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                              > > >>
                              > > >> we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                              > > >> i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                              > > >> Thanx in advance
                              > > >> Saurabh.
                              > > >> this is the Exception we are getting :
                              > > >> ========================================
                              > > >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                              > > >> weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                              > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                              > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                              > > >> at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                              > > >> at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                              > > >> at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                              > > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                              > > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                              > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                              > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                              > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                              > > >> --------------- nested within: ------------------
                              > > >> weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                              > > >> [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                              > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                              > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                              > > >> at java.lang.Throwable.<init>(Compiled Code)
                              > > >> at java.lang.Error.<init>(Error.java:50)
                              > > >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                              > > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                              > > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                              > > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                              > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                              > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                              > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                              > > >>
                              > > >>
                              > > >
                              > > >--
                              > > >Cheers
                              > > >
                              > > >- Prasad
                              > > >
                              > > >
                              
                              
                              
                              
                              
                              
                    • 7. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
                      3004
                      Saurabh Dixit wrote:
                                
                                > Thanx Shivu & Prasad
                                > I have put the comments below :
                                > Prasad Peddada wrote:
                                >
                                > > shivu wrote:
                                > >
                                > > > today i found that i am also having the same problem:)))
                                > > >
                                > > > Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                                > > > >The reason you see this problem is because of servers in a cluster timing out. Meaning, they break out of the cluster and join the cluster again in a short duration causing confusion.
                                >
                                > ### timing out means unable to send responce or it is indicating some Exception which could crash the server.
                                
                                Servers in a cluster use heart beats to communicate. If you don't receive heart beats in 30 seconds, you assume that the servers is dead.
                                
                                
                                > > > >
                                > > > >Typically you see this under load if your heap size is not set right.
                                > > >
                                > > > could you be more detailed regarding RIGHT?
                                > >
                                > > Too much or too little could cause problems. You have experiment, to set the right heap size to handle load on the servers. You have to use tool like load runner to figure out the thread count, heap size to over come this problem.
                                >
                                > > ## I noticed some some suggestion which says that if the GC will take more some specified period of time then this could cause this problem. This obviosly pointing not to have very large heapsize.
                                
                                This totally depends on the VM. Different version behave in different manner. Currently if you use heap of 1GB, you might not get good performance. You shouldnt' me needing more than 512MB. Also make sure that you are not swapping.
                                
                                
                                > on the contray using small size means that GC will be dome quite frequently. Can you give an idea what % of available memory on average should be used for heap in this case for weblogic server.
                                >
                                
                                It all depends on application. There is no thumb rule.
                                
                                
                                >
                                > > >
                                > > > If you turn on cluster debugging
                                > > >
                                > > > what properties i have to set to what?
                                > > weblogic.debug.cluster=1
                                > > >
                                > > >
                                > > > , you will see a Timed Out Server message and after that you will start seeing BSE.
                                > > >
                                > >
                                >
                                > ## what specific message comes for this. does it notify when it removes a WLS from cluster. also any inputs on what could cause the running server to crash. Is Application has any restriction on normal coding. (likewhat will happen if do not clustering
                                > specific propertise for stateless EJBs).
                                >
                                
                                Timed Out Server and tells you which server left the cluster. All these are transparent to the application developers.
                                
                                
                                >
                                > > > Whats BSE?
                                > > >
                                > >
                                > > BadStatusException.
                                > >
                                > > >
                                > > > >Also turning on verbose gc will help too.
                                > > > >
                                > > > >Tuning your server should help you over come the problem. This has nothing to do with NFS or non NFS.
                                > > > >
                                >
                                > ## we see this problem on QA environment which i believe couldn't said to have heavy load and traffic.
                                
                                You have to try and figure out why the servers are timing out. Turn on verbose gc and see if gc is taking more than 30 secs.
                                
                                >
                                > > > >Hope you are using latest service pack.
                                > > >
                                > > > i am using WLS5.1 sp6
                                > > >
                                > > > >
                                > > > >- Prasad
                                > > > >
                                >
                                > ## Is installation of web server and WLS on the same m/c could make any impact on here (as we are doing that). if yes please let me know why ???thanks,
                                > Saurabh.
                                
                                If you don't have enough memory and the o/s swaps to disk, yes it will cause problems.
                                
                                - Prasad
                                
                                >
                                > > > >Saurabh wrote:
                                > > > >
                                > > > >> We are still getting the "BadStatus Exception" when we are using In Memory Replication using netScape Plugin on ironPlanet 4.0 on SOLARIS platform.
                                > > > >> I understand the cause of the problem given in previos replies, but let me know if there is any fix for this problem.
                                > > > >>
                                > > > >> we c'd not see any configuration issues, only point is that we are not using the Shared File System. Pl. confirm not using a shared file system could cause this problem .. or there is something wrong going on with this exception with other people.
                                > > > >> i will really appriciate if some body will give us a point wise check point to narrow the focus of the problem.
                                > > > >> Thanx in advance
                                > > > >> Saurabh.
                                > > > >> this is the Exception we are getting :
                                > > > >> ========================================
                                > > > >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet request terminiated with Error
                                > > > >> weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary
                                > > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                                > > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                                > > > >> at weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                                > > > >> at weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationManager_WLStub.java:181)
                                > > > >> at weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationManager.java:664)
                                > > > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                                > > > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                                > > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                                > > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                                > > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                                > > > >> --------------- nested within: ------------------
                                > > > >> weblogic.utils.NestedError: Tried to update secondary, but it thought it was the primary - with nested exception:
                                > > > >> [weblogic.cluster.replication.BadStatusException: update found 2183773591869100580 but it is not the secondary]
                                > > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                                > > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                                > > > >> at java.lang.Throwable.<init>(Compiled Code)
                                > > > >> at java.lang.Error.<init>(Error.java:50)
                                > > > >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                                > > > >> at weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                                > > > >> at weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                > > > >> at weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                                > > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled Code)
                                > > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                                > > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                                > > > >>
                                > > > >>
                                > > > >
                                > > > >--
                                > > > >Cheers
                                > > > >
                                > > > >- Prasad
                                > > > >
                                > > > >
                                
                                --
                                Cheers
                                
                                - Prasad
                                
                                
                                
                      • 8. Re: BadStatus Exception..( WLS 5.1 + SP5 )..[update found .... but it is
                        3004
                        Whoa! Just letting people know that there are two people with the same name
                                  on this newsgroup. But these are indeed two different people.
                                  
                                  "Saurabh Dixit" <SaurabhD@zeneb.com> wrote in message
                                  news:39EFD862.5691FD96@zeneb.com...
                                  >
                                  >
                                  > Thanx Shivu & Prasad
                                  > I have put the comments below :
                                  > Prasad Peddada wrote:
                                  >
                                  > > shivu wrote:
                                  > >
                                  > > > today i found that i am also having the same problem:)))
                                  > > >
                                  > > > Prasad Peddada <prasadp@bgnet.bgsu.edu> wrote:
                                  > > > >The reason you see this problem is because of servers in a cluster
                                  timing out. Meaning, they break out of the cluster and join the cluster
                                  again in a short duration causing confusion.
                                  >
                                  > ### timing out means unable to send responce or it is indicating some
                                  Exception which could crash the server.
                                  >
                                  > > > >
                                  > > > >Typically you see this under load if your heap size is not set right.
                                  > > >
                                  > > > could you be more detailed regarding RIGHT?
                                  > >
                                  > > Too much or too little could cause problems. You have experiment, to set
                                  the right heap size to handle load on the servers. You have to use tool like
                                  load runner to figure out the thread count, heap size to over come this
                                  problem.
                                  >
                                  > > ## I noticed some some suggestion which says that if the GC will take
                                  more some specified period of time then this could cause this problem. This
                                  obviosly pointing not to have very large heapsize.
                                  >
                                  > on the contray using small size means that GC will be dome
                                  quite frequently. Can you give an idea what % of available memory on average
                                  should be used for heap in this case for weblogic server.
                                  >
                                  > > >
                                  > > > If you turn on cluster debugging
                                  > > >
                                  > > > what properties i have to set to what?
                                  > > weblogic.debug.cluster=1
                                  > > >
                                  > > >
                                  > > > , you will see a Timed Out Server message and after that you will
                                  start seeing BSE.
                                  > > >
                                  > >
                                  >
                                  > ## what specific message comes for this. does it notify when it removes a
                                  WLS from cluster. also any inputs on what could cause the running server to
                                  crash. Is Application has any restriction on normal coding. (likewhat will
                                  happen if do not clustering
                                  > specific propertise for stateless EJBs).
                                  >
                                  > > > Whats BSE?
                                  > > >
                                  > >
                                  > > BadStatusException.
                                  > >
                                  > > >
                                  > > > >Also turning on verbose gc will help too.
                                  > > > >
                                  > > > >Tuning your server should help you over come the problem. This has
                                  nothing to do with NFS or non NFS.
                                  > > > >
                                  >
                                  > ## we see this problem on QA environment which i believe couldn't said to
                                  have heavy load and traffic.
                                  >
                                  > > > >Hope you are using latest service pack.
                                  > > >
                                  > > > i am using WLS5.1 sp6
                                  > > >
                                  > > > >
                                  > > > >- Prasad
                                  > > > >
                                  >
                                  > ## Is installation of web server and WLS on the same m/c could make any
                                  impact on here (as we are doing that). if yes please let me know why
                                  ???thanks,
                                  > Saurabh.
                                  >
                                  > > > >Saurabh wrote:
                                  > > > >
                                  > > > >> We are still getting the "BadStatus Exception" when we are
                                  using In Memory Replication using netScape Plugin on ironPlanet 4.0 on
                                  SOLARIS platform.
                                  > > > >> I understand the cause of the problem given in previos replies,
                                  but let me know if there is any fix for this problem.
                                  > > > >>
                                  > > > >> we c'd not see any configuration issues, only point is that we
                                  are not using the Shared File System. Pl. confirm not using a shared file
                                  system could cause this problem .. or there is something wrong going on with
                                  this exception with other people.
                                  > > > >> i will really appriciate if some body will give us a point wise
                                  check point to narrow the focus of the problem.
                                  > > > >> Thanx in advance
                                  > > > >> Saurabh.
                                  > > > >> this is the Exception we are getting :
                                  > > > >> ========================================
                                  > > > >> Wed Oct 11 08:51:56 PDT 2000:<E> <ServletContextManager> Servlet
                                  request terminiated with Error
                                  > > > >> weblogic.cluster.replication.BadStatusException: update found
                                  2183773591869100580 but it is not the secondary
                                  > > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                                  > > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                                  > > > >> at
                                  weblogic.rmi.extensions.AbstractRequest.sendReceive(Compiled Code)
                                  > > > >> at
                                  weblogic.cluster.replication.ReplicationManager_WLStub.update(ReplicationMan
                                  ager_WLStub.java:181)
                                  > > > >> at
                                  weblogic.cluster.replication.ReplicationManager.updateSecondary(ReplicationM
                                  anager.java:664)
                                  > > > >> at
                                  weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled
                                  Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                                  > > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled
                                  Code)
                                  > > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                                  > > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                                  > > > >> --------------- nested within: ------------------
                                  > > > >> weblogic.utils.NestedError: Tried to update secondary, but it
                                  thought it was the primary - with nested exception:
                                  > > > >> [weblogic.cluster.replication.BadStatusException: update found
                                  2183773591869100580 but it is not the secondary]
                                  > > > >> at java.lang.Throwable.fillInStackTrace(Native Method)
                                  > > > >> at java.lang.Throwable.fillInStackTrace(Compiled Code)
                                  > > > >> at java.lang.Throwable.<init>(Compiled Code)
                                  > > > >> at java.lang.Error.<init>(Error.java:50)
                                  > > > >> at weblogic.utils.NestedError.<init>(NestedError.java:23)
                                  > > > >> at
                                  weblogic.servlet.internal.session.ReplicatedSession.sync(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.session.ReplicatedSessionContext.sync(Compiled
                                  Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletRequestImpl.syncSession(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextImpl.invokeServlet(Compiled Code)
                                  > > > >> at
                                  weblogic.servlet.internal.ServletContextManager.invokeServlet(Compiled Code)
                                  > > > >> at weblogic.socket.MuxableSocketHTTP.invokeServlet(Compiled
                                  Code)
                                  > > > >> at weblogic.socket.MuxableSocketHTTP.execute(Compiled Code)
                                  > > > >> at weblogic.kernel.ExecuteThread.run(Compiled Code)
                                  > > > >>
                                  > > > >>
                                  > > > >
                                  > > > >--
                                  > > > >Cheers
                                  > > > >
                                  > > > >- Prasad
                                  > > > >
                                  > > > >
                                  >
                                  >
                                  >
                                  >
                                  >