8 Replies Latest reply: Jan 25, 2013 8:33 AM by 980874 RSS

    Adding an OVM server back into OVM Manager after a reload

    980874
      We have created a sandbox environment to test Oracle virtualization and had 2 physical servers setup. One running Oracle OVM and the other Oracle Linux. The Oracle Linux box had OVM Manager installed and was using local storage for the repository.

      After about a month of running the Oracle Linux box was reloaded without saving any of the OVM Manager configuration information. After reloading the Oracle Linux box and installing OVM Manager my first issue was adding the OVM server back into a pool. We were getting the error that it was already being managed by another host. After a quick search of the Oracle forum I found a post suggesting that we stop the ovs-agent on the OVM box and remove a file.

      These were the commands:
      # service ovs-agent stop
      # rm /etc/ovs-agent/db/server
      # service ovs-agent start

      After doing this I was able to discover and add the server into a pool. The next issue was when I went to look at the virtual machines hosted on the OVM machine. All 6 of the virtual machines that were created now show as "orphan_004fb0000060000...". In digging around I noticed that the repository for the local storage was not setup so when I go to try add the local disk back as a repository and I select "create a data repository: select physical disk" it comes back as blank.

      If I click the storage tab and look under local file systems I see the disk and it has location repository specified and server listed under presented servers.

      I am thinking the mapping is somehow messed up after adding the host back to a new OVM Manager and I need to relink the storage to the repository but am a little lost as to how to accomplish this.

      I am trying to avoid reloading the entire OVM server so I don't lose the work already done with the existing virtual machines.
        • 1. Re: Adding an OVM server back into OVM Manager after a reload
          user12273962
          1. Right click the rep under "local file systems" and run a refresh. See if that helps.

          2. Did you have unclustered server pool? If NOT, then where is your clustered server pool mount point?
          • 2. Re: Adding an OVM server back into OVM Manager after a reload
            980874
            To answer your second question first, It is an unclustered server pool.

            When I run the refresh I get an error. Below are the details:

            Job Construction Phase
            ----------------------
            begin()
            Appended operation 'FileSystem refresh' to object '0004fb0000050000f6e5bfcdec6df114'.
            commit()
            Completed Step: COMMIT

            Objects and Operations
            ----------------------
            Object (IN_USE): [Server] 44:45:4c:4c:31:00:10:51:80:50:b1:c0:4f:4e:4d:31 (fnt1gvs001.syncreon.local)
            Object (IN_USE): [LocalFileServer] 0004fb00000900008975e0e107a39557 (Local FS fnt1gvs001.syncreon.local)
            Object (IN_USE): [LocalFileSystem] 0004fb0000050000f6e5bfcdec6df114
            Operation: FileSystem refresh

            Job Running Phase at 15:03 on Wed, Dec 19, 2012
            ----------------------------------------------
            Job Participants: [44:45:4c:4c:31:00:10:51:80:50:b1:c0:4f:4e:4d:31 (fnt1gvs001.syncreon.local)]


            Actioner
            --------
            Starting operation 'FileSystem refresh' on object '0004fb0000050000f6e5bfcdec6df114'
            Job Internal Error (Indeterminate)com.oracle.ovm.mgr.api.exception.RuleException: OVMRU_002017E Cannot perform operation. This repository: 36842b2b01874c700182161460904cc01, is not owned by this manager
            Wed Dec 19 15:03:54 EST 2012
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.throwErrorIfNotOwnedByThisManager(RepositoryRules.java:378)
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.setSimpleNamePre(RepositoryRules.java:197)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:662)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:631)
            at com.oracle.ovm.mgr.rules.RulesEngine.runRules(RulesEngine.java:190)
            at com.oracle.ovm.mgr.rules.RulesEngine.preProcess(RulesEngine.java:142)
            at com.oracle.ovm.mgr.model.ModelEngine.preValidate(ModelEngine.java:547)
            at com.oracle.ovm.mgr.model.ModelEngine.access$200(ModelEngine.java:65)
            at com.oracle.ovm.mgr.model.ModelEngine$3.notify(ModelEngine.java:352)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:326)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.virtual.RepositoryProxy.setSimpleName(Unknown Source)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.processRepoMetadata(FileSystemRefresh.java:338)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.discoverOcfs2FileSystem(FileSystemRefresh.java:188)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.action(FileSystemRefresh.java:84)
            at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.physical.storage.LocalFileSystemProxy.executeCurrentJobOperationAction(Unknown Source)
            at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
            at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)


            FailedOperationCleanup
            ----------
            Starting failed operation 'FileSystem refresh' cleanup on object '0004fb0000050000f6e5bfcdec6df114'
            Complete rollback operation 'FileSystem refresh' completed with direction=0004fb0000050000f6e5bfcdec6df114

            Rollbacker
            ----------
            Executing rollback operation 'FileSystem refresh' on object '0004fb0000050000f6e5bfcdec6df114'
            Complete rollback operation 'FileSystem refresh' completed with direction=DONE

            Objects To Be Rolled Back
            -------------------------
            Object (IN_USE): [Server] 44:45:4c:4c:31:00:10:51:80:50:b1:c0:4f:4e:4d:31 (fnt1gvs001.syncreon.local)
            Object (IN_USE): [Repository] 0004fb0000030000eac6a78cc3aa8b91 (36842b2b01874c700182161460904cc01)
            Object (IN_USE): [LocalFileServer] 0004fb00000900008975e0e107a39557 (Local FS fnt1gvs001.syncreon.local)
            Object (IN_USE): [LocalFileSystem] 0004fb0000050000f6e5bfcdec6df114


            Write Methods Invoked
            -------------------
            Class=InternalJobDbImpl vessel_id=1569 method=addTransactionIdentifier accessLevel=6
            Class=LocalFileSystemDbImpl vessel_id=930 method=refresh accessLevel=6
            Class=ServerDbImpl vessel_id=544 method=lock accessLevel=6
            Class=LocalFileServerDbImpl vessel_id=897 method=lock accessLevel=6
            Class=InternalJobDbImpl vessel_id=1569 method=setCompletedStep accessLevel=6
            Class=InternalJobDbImpl vessel_id=1569 method=setAssociatedHandles accessLevel=6
            Class=RepositoryDbImpl vessel_id=1111 method=setSimpleName accessLevel=6
            Class=LocalFileSystemDbImpl vessel_id=930 method=nextJobOperation accessLevel=6
            Class=InternalJobDbImpl vessel_id=1569 method=setFailedOperation accessLevel=6
            Class=ServerDbImpl vessel_id=544 method=nextJobOperation accessLevel=6
            Class=LocalFileServerDbImpl vessel_id=897 method=nextJobOperation accessLevel=6
            Class=LocalFileSystemDbImpl vessel_id=930 method=nextJobOperation accessLevel=6
            Class=LocalFileSystemDbImpl vessel_id=930 method=nextJobOperation accessLevel=6
            Completed Step: ROLLBACK
            Job failed commit (internal) due to com.oracle.ovm.mgr.api.exception.RuleException: OVMRU_002017E Cannot perform operation. This repository: 36842b2b01874c700182161460904cc01, is not owned by this manager
            Wed Dec 19 15:03:54 EST 2012
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.throwErrorIfNotOwnedByThisManager(RepositoryRules.java:378)
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.setSimpleNamePre(RepositoryRules.java:197)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:662)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:631)
            at com.oracle.ovm.mgr.rules.RulesEngine.runRules(RulesEngine.java:190)
            at com.oracle.ovm.mgr.rules.RulesEngine.preProcess(RulesEngine.java:142)
            at com.oracle.ovm.mgr.model.ModelEngine.preValidate(ModelEngine.java:547)
            at com.oracle.ovm.mgr.model.ModelEngine.access$200(ModelEngine.java:65)
            at com.oracle.ovm.mgr.model.ModelEngine$3.notify(ModelEngine.java:352)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:326)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.virtual.RepositoryProxy.setSimpleName(Unknown Source)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.processRepoMetadata(FileSystemRefresh.java:338)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.discoverOcfs2FileSystem(FileSystemRefresh.java:188)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.action(FileSystemRefresh.java:84)
            at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.physical.storage.LocalFileSystemProxy.executeCurrentJobOperationAction(Unknown Source)
            at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
            at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)
            ...
            Wed Dec 19 15:03:54 EST 2012
            com.oracle.ovm.mgr.api.exception.FailedOperationException: com.oracle.ovm.mgr.api.exception.RuleException: OVMRU_002017E Cannot perform operation. This repository: 36842b2b01874c700182161460904cc01, is not owned by this manager
            Wed Dec 19 15:03:54 EST 2012
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.throwErrorIfNotOwnedByThisManager(RepositoryRules.java:378)
            at com.oracle.ovm.mgr.rules.modules.api.virtual.RepositoryRules.setSimpleNamePre(RepositoryRules.java:197)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:662)
            at com.oracle.ovm.mgr.api.job.JobEngine.invokeMethod(JobEngine.java:631)
            at com.oracle.ovm.mgr.rules.RulesEngine.runRules(RulesEngine.java:190)
            at com.oracle.ovm.mgr.rules.RulesEngine.preProcess(RulesEngine.java:142)
            at com.oracle.ovm.mgr.model.ModelEngine.preValidate(ModelEngine.java:547)
            at com.oracle.ovm.mgr.model.ModelEngine.access$200(ModelEngine.java:65)
            at com.oracle.ovm.mgr.model.ModelEngine$3.notify(ModelEngine.java:352)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:326)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.virtual.RepositoryProxy.setSimpleName(Unknown Source)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.processRepoMetadata(FileSystemRefresh.java:338)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.discoverOcfs2FileSystem(FileSystemRefresh.java:188)
            at com.oracle.ovm.mgr.op.physical.storage.FileSystemRefresh.action(FileSystemRefresh.java:84)
            at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.physical.storage.LocalFileSystemProxy.executeCurrentJobOperationAction(Unknown Source)
            at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
            at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)
            ...
            Wed Dec 19 15:03:54 EST 2012
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1201)
            at sun.reflect.GeneratedMethodAccessor1396.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)


            ----------
            End of Job
            ----------
            • 3. Re: Adding an OVM server back into OVM Manager after a reload
              user12273962
              The only solution I know of is to reinstall the OVM Manager using the old VM managers UUID

              http://docs.oracle.com/cd/E27300_01/E27308/html/vmiug-manager-restoring.html

              I suggest you open a SR with oracle about this if you don't have the old UUID. I don't know of a work around. Maybe someone else does. The repo connection is looking for the old UUID of the manager.
              • 4. Re: Adding an OVM server back into OVM Manager after a reload
                980874
                Thanks for the info. This is something I was afraid of and was hoping that I could forcefully break the link and reconnect it to the new OVM Manager. I will hold out hope that someone may know another solution.
                • 5. Re: Adding an OVM server back into OVM Manager after a reload
                  934842
                  This happened to me not to long ago. I had an issue when deploying 3.2.1 beta and it ended up corrupting my db. I blew away my OVM manager and completely redeployed clean...I did not capture the UUID. The majority of what you need is in the OCFS2 server pool filesystem of the host.

                  In the storage tab you said you can see the storage disk. In the repository tab you also stated the storage disks were presented to the pool (or server) correct?

                  Do you have the OVM CLI tools on the manager? If so can you confirm the environment with the following:

                  - list repository
                  - show repository name=<name of repo from above command> (if no name is there change name= to id= and us the id)
                  - list server
                  - show server name=<name of server from list server command>

                  Does it show presented?
                  • 6. Re: Adding an OVM server back into OVM Manager after a reload
                    945503
                    Yes. The simple answer is that you have to reinstall the OVM server node.
                    Any legacy configuration that the server had before you rediscovered it in your manager will still be there, and likely to give you grief.

                    The more complicated answer is that you will have to manually hack the server under the hood and undo the previous configuration before you have the manager discover it.
                    That's great and all if you know what configuration file in linux that the manager had changed before you dropped the server out of the manager.

                    It would be really nice if there was an option to RESET and roll back to configuration on the OVM server node back to a fresh install configuration without having to get media out and reinstall the box.
                    Unfortunately no such feature exists yet. (one can dream though)

                    Edited by: 942500 on Dec 20, 2012 5:53 PM
                    • 7. Re: Adding an OVM server back into OVM Manager after a reload
                      980874
                      We are currently running version 3.1.1.305 and from my understanding if I am reading correctly I need 3.1.1.365 for the CLI. I do not currently have a support contract with Oracle so I am unable to download patch ID 14227416.

                      I am trying to get some assistance on this issue from our Oracle sales rep.
                      • 8. Re: Adding an OVM server back into OVM Manager after a reload
                        980874
                        Ok I was able to find this through some digging. It appears that the UUID for your OVM Manager is also stored in the ".ovsrepo" file. The OVS_REPO_MGR_UUID is the Oracle VM Manager UUID.

                        The Oracle VM Manager UUID is listed in the “.config ” file on the Oracle VM Manager host in the /u01/app/oracle/ovm-manager-3/ directory AS WELL as in each server pool' .ovsrepo file in the pool file system.

                        Once I found this I reinstalled OVM Manager with the correct UUID and was able to reattach the storage repositories without losing any data or having to reload the host machine.