9 Replies Latest reply on Aug 8, 2012 12:30 PM by ovmnewbee

    Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error

    ovmnewbee
      Hi

      I have a OVM 3.0.3 setup. The OVM manager and the Oracle 11g database are running as 2 seperate VM's on a X6270 M2 Blade which is on VMware ESXi.

      I created 2 LUN's one 25GB for the Clusterfilesystem and another 300GB as a storage repository for the OVM server on a NetApp 3020 filer. I am able to map the LUN onto the OVM manager but when I try to create a server pool and select the 25GB LUN as the repository for the cluster file system I keep getting the following error.

      tamp:     Aug 03, 2012 2:18:38 pm
      Status:     Failed
      Description:     Create Server Pool clustpool
      Created By:     admin
      Duration:     93ms
      Start Time:     Aug 03, 2012 2:30:24 pm
      End Time:      Aug 03, 2012 2:30:24 pm
      Message:     (08/03/2012 02:30:24:400 PM) OVMAPI_4010E Attempt to send command: create_pool_filesystem to server: englabbld06 failed. OVMAPI_4004E Server Failed Command: create_pool_filesystem nfs 206.236.134.21:/u01/clustrepo 620c03e59354719c 0004fb00000500000655780207887c30 920c26a9-dfdf-4977-8987-32bf39b47e25 0004fb000001000063c25c719da3e856 0004fb0000020000620c03e59354719c, Status: java.lang.InterruptedException Fri Aug 03 14:30:24 IST 2012 Fri Aug 03 14:30:24 IST 2012


      I repeated the same exercise using NFS but the same error keeps popping up.

      I have thoroughly checked the configuration of OVM servers as well as the OVM manager including, host file resolution, DNS entry, firewall, ntp etc...but nothing seems to work, I have re installed the complete infrastructure once again but does not seem to solve the problem.
        • 1. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
          Pioneer
          Try this:

          You should hide your .snapshot directory, otherwise oracle VM sees that the share is not empty and will fail.

          Go to Oncomand System manager or Filerview and disable the visibility of the .snapshot directory. Please let me know if this helps.
          • 2. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
            861130
            Also remember that since you're using NFS, if you've got a Netapp clustered with dual heads you need to increase the OCFS timeout on the pool repository based on the time it takes to do a head 'handover'. We've seen our Netapp hold the IO for over 2 minutes before during a handover, and this causes all the vm's to panic..
            • 3. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
              ovmnewbee
              Hi Pioneer
              thanks for your inputs...but it does not seem to be working...recreated the LUN's assigned it to the OVM manager but server pool creation fails...the issue here is that the server pool creation process gets locked for more than 15-20 mnts and then fails...

              BTW I was going through the following thread mentioned below and its seems to be a bug in OVM..which is resolved by installing the patch...please advise if I need to do the same. I will anyway once again recheck all the parameters

              Can't create repository as iSCSI physical disk

              Please find the log files below

              b Construction Phase
              ----------------------
              begin()
              Appended operation 'Pool File System Construct' to object '0004fb0000050000feccace26da5f020 (Pool filesystem for clustrepo)'.
              commit()
              Completed Step: COMMIT

              Objects and Operations
              ----------------------
              Object (CREATED): [Cluster] 8b88de52d58a611b
              Object (CREATED): [ServerPool] 0004fb00000200008b88de52d58a611b (clustrepo)
              Object (CREATED): [PoolFileSystem] 0004fb0000050000feccace26da5f020 (Pool filesystem for clustrepo)
              Operation: Pool File System Construct
              Object (IN_USE): [StorageElement] 0004fb000018000049d49e4aca3a430b (clustrepo)
              Object (CREATED): [BalancerControl] balancer_0004fb00000200008b88de52d58a611b

              Job Running Phase at 10:10 on Tue, Aug 7, 2012
              ----------------------------------------------
              Job Participants: [ff:20:00:08:ff:ff:ff:ff:ff:ff:da:48:de:28:21:00 (englabbld06)]


              Actioner
              --------
              Starting operation 'Pool File System Construct' on object '0004fb0000050000feccace26da5f020 (Pool filesystem for clustrepo)'
              Job: 1344314405855, aborted post-commit by user: admin

              Write Methods Invoked
              -------------------
              Class=InternalJobDbImpl vessel_id=2580 method=addTransactionIdentifier accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setFoundryContext accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=onPersistableCreate accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setName accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setFoundryContext accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=onPersistableCreate accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setLifecycleState accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setRollbackLifecycleState accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setServerPool accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setName accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setFoundryContext accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=onPersistableCreate accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setLifecycleState accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setRollbackLifecycleState accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setServerPool accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setLifecycleState accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setRollbackLifecycleState accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setSimpleName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setSimpleName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setVirtualIP accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setKeymapName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setVmMigrateUsingSsl accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=createPoolFileSystem accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setName accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setFoundryContext accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=onPersistableCreate accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setLifecycleState accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setRollbackLifecycleState accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setSimpleName accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setStorageDevice accessLevel=6
              Class=StorageElementDbImpl vessel_id=2448 method=addLayeredFileSystem accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setUuid accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setManagerUuid accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setServerPool accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=addJobOperation accessLevel=6
              Class=InternalJobDbImpl vessel_id=2580 method=setCompletedStep accessLevel=6
              Class=InternalJobDbImpl vessel_id=2580 method=setAssociatedHandles accessLevel=6
              Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: create_pool_filesystem to server: englabbld06 failed. OVMAPI_4004E Server Failed Command: create_pool_filesystem lun /dev/mapper/360a98000433465644e346c654c51642d 8b88de52d58a611b 0004fb0000050000feccace26da5f020 0004fb000001000063c25c719da3e856 0004fb00000200008b88de52d58a611b, Status: java.lang.InterruptedException
              Tue Aug 07 10:13:53 IST 2012
              Tue Aug 07 10:13:53 IST 2012
              at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
              at com.oracle.ovm.mgr.action.ActionEngine.sendUndispatchedServerCommand(ActionEngine.java:427)
              at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:369)
              at com.oracle.ovm.mgr.action.ClusterAction.createPoolFileSystem(ClusterAction.java:47)
              at com.oracle.ovm.mgr.op.physical.storage.PoolFileSystemConstruct.constructAction(PoolFileSystemConstruct.java:101)
              at com.oracle.ovm.mgr.op.physical.storage.PoolFileSystemConstruct.action(PoolFileSystemConstruct.java:50)
              at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
              at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
              at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
              at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
              at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
              at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
              at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
              at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
              at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:751)
              at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
              at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
              at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
              at java.lang.Thread.run(Thread.java:662)
              Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: create_pool_filesystem lun /dev/mapper/360a98000433465644e346c654c51642d 8b88de52d58a611b 0004fb0000050000feccace26da5f020 0004fb000001000063c25c719da3e856 0004fb00000200008b88de52d58a611b, Status: java.lang.InterruptedException
              Tue Aug 07 10:13:53 IST 2012
              at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:753)
              at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:471)
              ... 21 more


              FailedOperationCleanup
              ----------
              Starting failed operation 'Pool File System Construct' cleanup on object 'Pool filesystem for clustrepo'
              Complete rollback operation 'Pool File System Construct' completed with direction=Pool filesystem for clustrepo

              Rollbacker
              ----------

              Objects To Be Rolled Back
              -------------------------
              Object (CREATED): [Cluster] 8b88de52d58a611b
              Object (CREATED): [ServerPool] 0004fb00000200008b88de52d58a611b (clustrepo)
              Object (CREATED): [PoolFileSystem] 0004fb0000050000feccace26da5f020 (Pool filesystem for clustrepo)
              Object (IN_USE): [StorageElement] 0004fb000018000049d49e4aca3a430b (clustrepo)
              Object (CREATED): [BalancerControl] balancer_0004fb00000200008b88de52d58a611b


              Write Methods Invoked
              -------------------
              Class=InternalJobDbImpl vessel_id=2580 method=addTransactionIdentifier accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setFoundryContext accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=onPersistableCreate accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setName accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setFoundryContext accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=onPersistableCreate accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setLifecycleState accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setRollbackLifecycleState accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=setServerPool accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setName accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setFoundryContext accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=onPersistableCreate accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setLifecycleState accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setRollbackLifecycleState accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=setServerPool accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setLifecycleState accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setRollbackLifecycleState accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setSimpleName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setSimpleName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setVirtualIP accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setKeymapName accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=setVmMigrateUsingSsl accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=createPoolFileSystem accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setName accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setFoundryContext accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=onPersistableCreate accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setLifecycleState accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setRollbackLifecycleState accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setSimpleName accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setStorageDevice accessLevel=6
              Class=StorageElementDbImpl vessel_id=2448 method=addLayeredFileSystem accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setUuid accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setManagerUuid accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=setServerPool accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=addJobOperation accessLevel=6
              Class=InternalJobDbImpl vessel_id=2580 method=setCompletedStep accessLevel=6
              Class=InternalJobDbImpl vessel_id=2580 method=setAssociatedHandles accessLevel=6
              Class=InternalJobDbImpl vessel_id=2580 method=setFailedOperation accessLevel=6
              Class=ClusterDbImpl vessel_id=2595 method=nextJobOperation accessLevel=6
              Class=ServerPoolDbImpl vessel_id=2589 method=nextJobOperation accessLevel=6
              Class=PoolFileSystemDbImpl vessel_id=2607 method=nextJobOperation accessLevel=6
              Class=StorageElementDbImpl vessel_id=2448 method=nextJobOperation accessLevel=6
              Class=BalancerControlDbImpl vessel_id=2601 method=nextJobOperation accessLevel=6
              Completed Step: ROLLBACK
              Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: create_pool_filesystem to server: englabbld06 failed. OVMAPI_4004E Server Failed Command: create_pool_filesystem lun /dev/mapper/360a98000433465644e346c654c51642d 8b88de52d58a611b 0004fb0000050000feccace26da5f020 0004fb000001000063c25c719da3e856 0004fb00000200008b88de52d58a611b, Status: java.lang.InterruptedException
              Tue Aug 07 10:13:53 IST 2012
              Tue Aug 07 10:13:53 IST 2012
              com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: create_pool_filesystem to server: englabbld06 failed. OVMAPI_4004E Server Failed Command: create_pool_filesystem lun /dev/mapper/360a98000433465644e346c654c51642d 8b88de52d58a611b 0004fb0000050000feccace26da5f020 0004fb000001000063c25c719da3e856 0004fb00000200008b88de52d58a611b, Status: java.lang.InterruptedException
              Tue Aug 07 10:13:53 IST 2012
              Tue Aug 07 10:13:53 IST 2012
              at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
              at com.oracle.ovm.mgr.action.ActionEngine.sendUndispatchedServerCommand(ActionEngine.java:427)
              at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:369)
              at com.oracle.ovm.mgr.action.ClusterAction.createPoolFileSystem(ClusterAction.java:47)
              at com.oracle.ovm.mgr.op.physical.storage.PoolFileSystemConstruct.constructAction(PoolFileSystemConstruct.java:101)
              at com.oracle.ovm.mgr.op.physical.storage.PoolFileSystemConstruct.action(PoolFileSystemConstruct.java:50)
              at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
              at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
              at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
              at sun.reflect.GeneratedMethodAccessor1167.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
              at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
              at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
              at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
              at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
              • 4. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                ovmnewbee
                hi Dave

                We have stand alone NetApp filer and no clusters. I am new to OVM :) would greatly apprecicate if you could let me know how to increase the OCFS timeout for NFS..I am stuck in creating a server pool itself..therefore VM creation is still not in the picture
                • 5. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                  Pioneer
                  Thats strange... You already found a very interesting thread. --> Can't create repository as iSCSI physical disk Unfortunately I can't come up with what could be the problem.

                  Did you enable the 'storage' capability on the network card from OVM-manager and can you verify that you are able to connect (ping) to the storage from any of the VM-servers?
                  Does it work with NFS (and hidden snapshot dir)?

                  The thread mentioned seems to be about a timeout for iscsi storage initialization. It seems that you need that patch if you have slow storage or use a very large isci lun. Neither seems the case I think. You can always try to use the OVM-utils they mention, these also bypass the timeout according to this blog: https://blogs.oracle.com/wim/entry/using_ovm_utils
                  • 6. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                    ovmnewbee
                    Hi Pioneer

                    Yes I have enabled the storage capability on my network card that was the first configuration that I checked when I faced the issue...I am able to ping the storage from both the VM serverwithout any issues... please let me know if I have missed out on anything else...or probably I need to get help from Oracle directly in regard to this issue :(

                    i will go through the blogs on the ovm_utils and see how that helps

                    I will work on the NFS and get back with the feedback
                    • 7. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                      Liao
                      just map your storage to the Oracle VM Server not the Oracle VM Manager.
                      and then ,check your storage support shared-partition&&two controlers or not.
                      i hope this advise can help you
                      • 8. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                        861130
                        Hi,
                        If you've only got one netapp head there is nothing to worry about - other than you only having one netapp head.

                        the ocfs heartbeat timeouts are in /etc/sysconfig/o2cb:

                        O2CB_HEARTBEAT_THRESHOLD=31 (seconds).
                        • 9. Re: Unable to create server pool cluster using ISCSI/NFS... OVMAPI_4010E error
                          ovmnewbee
                          Hi

                          I am getting the same error in NFS also :(...any pointers on how to resolve this would be of great help to me!!