5 Replies Latest reply on Oct 2, 2019 4:10 AM by Harshitajain-Oracle

    OVM Jobs never gets completed

    3589191

      Hi Everybody,

       

      Hope you are doing fine.

       

      On OVM 3.4.6 when y trow some Jobs these never gets completed, I see this happening when I try to start or stop a VM or when I try to import a virtual appliance through OVM Manager.

       

      Does anybody have the same problem?

       

      If I try to start a VM the Job status says "Starting" even when the VM is already RUNNING, if I abort this Job the VM status change to RUNNING... seems OVMM is never updated with the real state of the Job.

       

      If I get the status from CLI it says the job is running but it never gets completed:

       

      OVM> show job id=1569622305176

      Command: show job id=1569622305176

      Status: Success

      Time: 2019-09-27 16:52:22,952 CST

      Data:

        Run State = Running

        Summary State = Running

        Done = No

        Summary Done = No

        Job Group = No

        Username = admin

        Creation Time = Sep 26, 2019 1:11:45 pm

        Start Time = Sep 26, 2019 1:11:45 pm

        Id = 1569622305176  [Import Virtual Appliance to Repository: NODE_SYS_REPO_X8. URL: ftp://?uname?:?pwd?@192.168.1.9:21/../Dev.ova]

        Name = Import Virtual Appliance to Repository: NODE_SYS_REPO_X8. URL: ftp://?uname?:?pwd?@192.168.1.9:21/../Dev.ova

        Description = Import Virtual Appliance to Repository: NODE_SYS_REPO_X8. URL: ftp://?uname?:?pwd?@192.168.1.9:21/../Dev.ova

        Locked = false

        • 1. Re: OVM Jobs never gets completed
          Harshitajain-Oracle

          Hi,

           

          Can you try restarting ovs-agent and ovmm service and check?

           

          On OVM MANAGER

           

          # service ovmm stop

          #service ovmm start

           

          ON Server

           

          #service ovs-agent stop

          #service ovs-agent start

           

          Can you now try and see,

           

          --> Also for me it looks like DB corruption. Do you see any "cluster is NULL"

          Or any kind of below errors:

           

          No such object (level 1):

          ObjectNotFoundException: No such object (level 1):

          odof.exception.ObjectNotFoundException: No such object (level 1):

          com.oracle.odof.exception.ObjectNotFoundException: No such object (level 1)

          Additional  messages in the AdminServer.log

          java.io.IOException: Broken pipe

          Example of a java.lang.RuntimeException error

          java.lang.RuntimeException:  and java.lang.NullPointerException

           

          In AdminServer.log? If yes likely to be database corruption.

           

          Hope this helps

           

          Regards,

          Harshita.

          1 person found this helpful
          • 2. Re: OVM Jobs never gets completed
            3589191

            Hi,

             

            Thanks for your response, I tried restarting ovs and ovm services but that did not work.

             

            I search for those types of errors you suggested but did not find any of those in AdminServer log, just errors like these:

             

            ####<2019-09-30T11:02:32.936-0600> <Info> <WorkManager> <ovmmanager> <AdminServer> <OvmTimerThread> <<anonymous>> <> <62f68613-b208-40a9-9e39-8b06995e27b5-0000000d> <1569862952936> <BEA-002936> <maximum thread constraint WatchManagerEvents is reached>

            ####<2019-09-30T11:02:32.937-0600> <Error> <com.oracle.ovm.mgr.ws.WsSessionData> <ovmmanager> <AdminServer> <OvmTimerThread> <<anonymous>> <> <62f68613-b208-40a9-9e39-8b06995e27b5-0000000d> <1569862952937> <BEA-000000> <    java.net.SocketInputStream.socketRead0(Native Method)

                    java.net.SocketInputStream.read(SocketInputStream.java:153)

                    java.net.SocketInputStream.read(SocketInputStream.java:122)

                    java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

                    java.io.BufferedInputStream.read(BufferedInputStream.java:254)

                    java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2642)

                    java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2949)

                    java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2959)

                    java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1543)

                    java.io.ObjectInputStream.readObject(ObjectInputStream.java:434)

                    com.oracle.odof.io.AbstractSocket.receive(AbstractSocket.java:221)

                    com.oracle.odof.io.AbstractSocket.receive(AbstractSocket.java:173)

                    com.oracle.odof.OdofExchange.send(OdofExchange.java:493)

                    com.oracle.odof.OdofExchange.send(OdofExchange.java:440)

                    com.oracle.odof.OdofExchange.invokeMethod(OdofExchange.java:1016)

                    com.oracle.ovm.mgr.api.manager.EventManagerProxy.postNotificationWithData(Unknown Source)

                    com.oracle.ovm.mgr.ws.internal.mapper.ServerOnlyMethods.sendEvents(ServerOnlyMethods.java:82)

                    com.oracle.ovm.mgr.ws.internal.rest.ServerOnlyRs.serverSendEvents(ServerOnlyRs.java:132)

                    sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)

                    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                    java.lang.reflect.Method.invoke(Method.java:606)

                    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

                    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$JResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:236)

            com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

                    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

                    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

                    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

                    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

                    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

                    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)

                    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)

                    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)

                    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)

                    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)

                    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)

                    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)

                    weblogic.jaxrs.server.portable.servlet.ServletContainer.service(ServletContainer.java:219)

                    javax.servlet.http.HttpServlet.service(HttpServlet.java:844)

                    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)

                    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)

                    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)

                    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)

                    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)

                    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)

                    com.oracle.ovm.mgr.ws.WsApiDebugFilter.doFilter(WsApiDebugFilter.java:56)

                    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)

                    oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:137)

                    java.security.AccessController.doPrivileged(Native Method)

                    oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)

                    oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)

                    oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:120)

                    oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:217)

                    oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:81)

                    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)

                    oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:220)

            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)

                    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3456)

                    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3422)

                    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:323)

                    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

                    weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)

                    weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)

                    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)

                    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)

                    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1632)

                    weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:256)

                    weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)

                    weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

            >

            ####<2019-09-30T11:02:32.937-0600> <Error> <com.oracle.ovm.mgr.ws.WsSessionData> <ovmmanager> <AdminServer> <OvmTimerThread> <<anonymous>> <> <62f68613-b208-40a9-9e39-8b06995e27b5-0000000d> <1569862952937> <BEA-000000> <The contents of the stack at the acquisition time of the manager connection follows.>

            ####<2019-09-30T11:02:32.938-0600> <Error> <com.oracle.ovm.mgr.ws.WsSessionData> <ovmmanager> <AdminServer> <OvmTimerThread> <<anonymous>> <> <62f68613-b208-40a9-9e39-8b06995e27b5-0000000d> <1569862952938> <BEA-000000> <    java.lang.Thread.getStackTrace(Thread.java:1589)

                    com.oracle.ovm.mgr.ws.WsSessionData.acquireConnection(WsSessionData.java:216)

                    com.oracle.ovm.mgr.ws.rest.RestBase.getOvmManager(RestBase.java:156)

                    com.oracle.ovm.mgr.ws.rest.RestBase.getOvmManager(RestBase.java:83)

                    com.oracle.ovm.mgr.ws.internal.rest.ServerOnlyRs.serverSendEvents(ServerOnlyRs.java:131)

                    sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)

                    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                    java.lang.reflect.Method.invoke(Method.java:606)

                    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

                    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$JResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:236)

                    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

                    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

                    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

             

             

            What I see is that backups of MySQL db are not performed:

             

            • 3. Re: OVM Jobs never gets completed
              Harshitajain-Oracle

              Hi,

               

              Can you check file OVM_STATISTICS file?

               

              # du -hs /u01/app/oracle/mysql/data/ovs/OVM_STATISTIC*

               

              There you can check for the size of OVM_STATISTICS.ibd file. If its large please truncate the same and check if daily bacups are occuring now?

               

              To truncate:

               

              # mysql ovs -u ovs -p -S /u01/app/oracle/mysql/data/mysqld.sock

              mysql> truncate table OVM_STATISTIC;

               

              Reference - Oracle VM Manager Issues with Huge OVM_STATISTIC Table (Doc ID 2215428.1)

               

              Regards,

              Harshita.

              • 4. Re: OVM Jobs never gets completed
                3589191

                Hi Harshita,

                 

                Preior truncate table I see OVM_STATISTIC table has no rows:

                 

                • 5. Re: OVM Jobs never gets completed
                  Harshitajain-Oracle

                  Hi,

                   

                  Okay then it looks like a DB corruption for me.

                   

                  Can you please regenerate DB and check?

                   

                  You can refer to the below document:

                   

                  Oracle VM: How To Regenerate The Oracle VM Manager 3.3.x/3.4.x DB (Doc ID 2038168.1)

                   

                  Regards,

                  Harshita