7 Replies Latest reply: Jun 24, 2012 11:01 PM by 944494 RSS

    Can't create repository as iSCSI physical disk

    944494
      Using OVM 3.1.1

      I am using a ZFS Storage Appliance (simulated under virtualbox for testing) as a SAN.
      I created two iSCSI LUN devices on the appliance in the same ZFS pool:- a LUN for a server pool, and a LUN for a repository.
      After creating an Access Group that two of my OVM servers could use to see this pool, I was able to create a pool using the LUN I made for the server pool.
      Of course I have already installed the ZFS-SA pugin on my two OVM nodes, so this all works.

      When I tried to create the repository, I use the other iSCSI LUN and I get the message box with spinning timer icon telling me that it is creating the repository.
      However, the process times out and fails. The details of the failure I cut and pasted here.

      The interesting this is that instead of an iSCSI LUN, I can create an NFS share on the ZFS-SA, mount that, and use that as a repository and have that work.
      That's not what I want, however.
      What's going on? The detailed log output gives me no clue whatsoever as to what is wrong. Looks like it's clashing with OCFS2 or something.


      Job Construction Phase
      ----------------------
      begin()
      Appended operation 'File System Construct' to object '0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)'.
      Appended operation 'Cluster File System Present' to object 'ec686c238f27311b'.
      Appended operation 'Repository Construct' to object '0004fb000003000027ca3c09e0f30673 (SUN (2))'.
      commit()
      Completed Step: COMMIT

      Objects and Operations
      ----------------------
      Object (IN_USE): [Cluster] ec686c238f27311b
      Operation: Cluster File System Present
      Object (CREATED): [LocalFileSystem] 0004fb0000050000e580a3d171ecf6c1 (fs_repo01)
      Object (IN_USE): [LocalFileServer] 0004fb00000900008e232246f9e4b224 (Local FS ovm-dev-02)
      Object (CREATED): [Repository] 0004fb000003000027ca3c09e0f30673 (repo01)
      Operation: Repository Construct
      Object (IN_USE): [LocalFileServer] 0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)
      Operation: File System Construct
      Object (IN_USE): [StorageElement] 0004fb00001800009299c1a46c0e3979 (SUN (2))

      Job Running Phase at 00:39 on Wed, Jun 20, 2012
      ----------------------------------------------
      Job Participants: [34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)]


      Actioner
      --------
      Starting operation 'Cluster File System Present' on object '0004fb0000050000e580a3d171ecf6c1 (fs_repo01)'
      Completed operation 'Cluster File System Present' completed with direction ==> DONE
      Starting operation 'Repository Construct' on object '0004fb000003000027ca3c09e0f30673 (repo01)'
      Completed operation 'Repository Construct' completed with direction ==> LATER
      Starting operation 'File System Construct' on object '0004fb0000050000e580a3d171ecf6c1 (fs_repo01)'
      Job: 1340118585250, aborted post-commit by user: admin

      Write Methods Invoked
      -------------------
      Class=InternalJobDbImpl vessel_id=1450 method=addTransactionIdentifier accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=675 method=createFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setName accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setFoundryContext accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=onPersistableCreate accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setLifecycleState accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setRollbackLifecycleState accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setRefreshed accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setBackingDevices accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setUuid accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setPath accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setSimpleName accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addFileServer accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setStorageDevice accessLevel=6
      Class=StorageElementDbImpl vessel_id=1273 method=addLayeredFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setSimpleName accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=921 method=addFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addFileServer accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=addLocalFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setCluster accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setAsset accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=createRepository accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setFoundryContext accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=onPersistableCreate accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setLifecycleState accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setRollbackLifecycleState accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setRefreshed accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setDom0Uuid accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSharePath accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSimpleName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addRepository accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setManagerUuid accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setVersion accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=addJobOperation accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSimpleName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setDescription accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setCompletedStep accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setAssociatedHandles accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=setCurrentJobOperationComplete accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setTuringMachineFlag accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setTuringMachineFlag accessLevel=6
      Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
      at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:894)
      at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.createFileSystem(FileSystemConstruct.java:57)
      at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.action(FileSystemConstruct.java:49)
      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
      at com.oracle.ovm.mgr.api.physical.storage.LocalFileServerProxy.executeCurrentJobOperationAction(Unknown Source)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
      at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:890)
      ... 27 more
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
      ... 30 more


      FailedOperationCleanup
      ----------
      Starting failed operation 'File System Construct' cleanup on object 'fs_repo01'
      Complete rollback operation 'File System Construct' completed with direction=fs_repo01

      Rollbacker
      ----------
      Executing rollback operation 'Cluster File System Present' on object '0004fb0000050000e580a3d171ecf6c1 (fs_repo01)'
      Complete rollback operation 'Cluster File System Present' completed with direction=DONE
      Executing rollback operation 'File System Construct' on object '0004fb0000050000e580a3d171ecf6c1 (fs_repo01)'
      Complete rollback operation 'File System Construct' completed with direction=DONE

      Objects To Be Rolled Back
      -------------------------
      Object (IN_USE): [Cluster] ec686c238f27311b
      Object (CREATED): [LocalFileSystem] 0004fb0000050000e580a3d171ecf6c1 (fs_repo01)
      Object (IN_USE): [LocalFileServer] 0004fb00000900008e232246f9e4b224 (Local FS ovm-dev-02)
      Object (CREATED): [Repository] 0004fb000003000027ca3c09e0f30673 (repo01)
      Object (IN_USE): [LocalFileServer] 0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)
      Object (IN_USE): [StorageElement] 0004fb00001800009299c1a46c0e3979 (SUN (2))


      Write Methods Invoked
      -------------------
      Class=InternalJobDbImpl vessel_id=1450 method=addTransactionIdentifier accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=675 method=createFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setName accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setFoundryContext accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=onPersistableCreate accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setLifecycleState accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setRollbackLifecycleState accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setRefreshed accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setBackingDevices accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setUuid accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setPath accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setSimpleName accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addFileServer accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setStorageDevice accessLevel=6
      Class=StorageElementDbImpl vessel_id=1273 method=addLayeredFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setSimpleName accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=921 method=addFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addFileServer accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=addLocalFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setCluster accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=setAsset accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=createRepository accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setFoundryContext accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=onPersistableCreate accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setLifecycleState accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setRollbackLifecycleState accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setRefreshed accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setDom0Uuid accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSharePath accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSimpleName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setFileSystem accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=addRepository accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setManagerUuid accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setVersion accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=addJobOperation accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setSimpleName accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setDescription accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setCompletedStep accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setAssociatedHandles accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=setCurrentJobOperationComplete accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setTuringMachineFlag accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=setCurrentOperationToLater accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setTuringMachineFlag accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
      Class=InternalJobDbImpl vessel_id=1450 method=setFailedOperation accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
      Class=LocalFileSystemDbImpl vessel_id=1459 method=nextJobOperation accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=921 method=nextJobOperation accessLevel=6
      Class=RepositoryDbImpl vessel_id=1464 method=nextJobOperation accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
      Class=StorageElementDbImpl vessel_id=1273 method=nextJobOperation accessLevel=6
      Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
      Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
      Completed Step: ROLLBACK
      Job failed commit (internal) due to OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
      at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:894)
      at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.createFileSystem(FileSystemConstruct.java:57)
      at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.action(FileSystemConstruct.java:49)
      at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
      at sun.reflect.GeneratedMethodAccessor728.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
      at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
      at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
      at com.oracle.ovm.mgr.api.physical.storage.LocalFileServerProxy.executeCurrentJobOperationAction(Unknown Source)
      at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
      at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
      at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
      at sun.reflect.GeneratedMethodAccessor1229.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
      at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
      at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
      at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
      at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
      at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
      at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
      at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
      at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
      at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
      at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
      at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:890)
      ... 27 more
      Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000e580a3d171ecf6c1 /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: java.lang.InterruptedException
      Wed Jun 20 00:41:47 CST 2012
      at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
      at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
      ... 30 more


      ----------
      End of Job
      ----------

      Edited by: 941491 on Jun 19, 2012 5:54 PM

      Edited by: 941491 on Jun 19, 2012 5:55 PM

      Edited by: 941491 on Jun 19, 2012 5:56 PM
        • 1. Re: Can't create repository as iSCSI physical disk
          944494
          I conducted another test.

          I am only using one network /24 as management and server pool.
          I went to the "Networking" tab and checked the "Virtual Machine" and "Storage" boxes on my management network, so that all boxes were checked.

          This time, after trying to add the repository iSCSI LUN as a physical disk, I get the following debugging:-

          Job Construction Phase
          ----------------------
          begin()
          Appended operation 'File System Construct' to object '0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)'.
          Appended operation 'Cluster File System Present' to object 'ec686c238f27311b'.
          Appended operation 'Repository Construct' to object '0004fb0000030000b527d6f7e28a0052 (SUN (2))'.
          commit()
          Completed Step: COMMIT

          Objects and Operations
          ----------------------
          Object (IN_USE): [Cluster] ec686c238f27311b
          Operation: Cluster File System Present
          Object (IN_USE): [LocalFileServer] 0004fb00000900008e232246f9e4b224 (Local FS ovm-dev-02)
          Object (CREATED): [LocalFileSystem] 0004fb0000050000f00e264fcad6a2cb (fs_repo01)
          Object (CREATED): [Repository] 0004fb0000030000b527d6f7e28a0052 (repo01)
          Operation: Repository Construct
          Object (IN_USE): [LocalFileServer] 0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)
          Operation: File System Construct
          Object (IN_USE): [StorageElement] 0004fb00001800009299c1a46c0e3979 (SUN (2))

          Job Running Phase at 01:01 on Wed, Jun 20, 2012
          ----------------------------------------------
          Job Participants: [34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01), 34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)]


          Actioner
          --------
          Starting operation 'Cluster File System Present' on object '0004fb0000050000f00e264fcad6a2cb (fs_repo01)'
          Completed operation 'Cluster File System Present' completed with direction ==> DONE
          Starting operation 'Repository Construct' on object '0004fb0000030000b527d6f7e28a0052 (repo01)'
          Completed operation 'Repository Construct' completed with direction ==> LATER
          Starting operation 'File System Construct' on object '0004fb0000050000f00e264fcad6a2cb (fs_repo01)'
          Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
          at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:894)
          at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.createFileSystem(FileSystemConstruct.java:57)
          at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.action(FileSystemConstruct.java:49)
          at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
          at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
          at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
          at com.oracle.ovm.mgr.api.physical.storage.LocalFileServerProxy.executeCurrentJobOperationAction(Unknown Source)
          at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
          at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
          at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
          at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
          at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
          at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
          at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
          at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
          at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
          at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
          at java.lang.Thread.run(Thread.java:662)
          Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
          at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
          at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
          at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:890)
          ... 27 more
          Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
          at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
          ... 30 more


          FailedOperationCleanup
          ----------
          Starting failed operation 'File System Construct' cleanup on object 'fs_repo01'
          Complete rollback operation 'File System Construct' completed with direction=fs_repo01

          Rollbacker
          ----------
          Executing rollback operation 'Cluster File System Present' on object '0004fb0000050000f00e264fcad6a2cb (fs_repo01)'
          Complete rollback operation 'Cluster File System Present' completed with direction=DONE
          Executing rollback operation 'File System Construct' on object '0004fb0000050000f00e264fcad6a2cb (fs_repo01)'
          Complete rollback operation 'File System Construct' completed with direction=DONE

          Objects To Be Rolled Back
          -------------------------
          Object (IN_USE): [Cluster] ec686c238f27311b
          Object (IN_USE): [LocalFileServer] 0004fb00000900008e232246f9e4b224 (Local FS ovm-dev-02)
          Object (CREATED): [LocalFileSystem] 0004fb0000050000f00e264fcad6a2cb (fs_repo01)
          Object (CREATED): [Repository] 0004fb0000030000b527d6f7e28a0052 (repo01)
          Object (IN_USE): [LocalFileServer] 0004fb0000090000de3c84de0325cbb2 (Local FS ovm-dev-01)
          Object (IN_USE): [StorageElement] 0004fb00001800009299c1a46c0e3979 (SUN (2))


          Write Methods Invoked
          -------------------
          Class=InternalJobDbImpl vessel_id=1607 method=addTransactionIdentifier accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=675 method=createFileSystem accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setName accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setFoundryContext accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=onPersistableCreate accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setLifecycleState accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setRollbackLifecycleState accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setRefreshed accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setBackingDevices accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setUuid accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setPath accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setSimpleName accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=addFileServer accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setStorageDevice accessLevel=6
          Class=StorageElementDbImpl vessel_id=1273 method=addLayeredFileSystem accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setSimpleName accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=921 method=addFileSystem accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=addFileServer accessLevel=6
          Class=ClusterDbImpl vessel_id=1374 method=addLocalFileSystem accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setCluster accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=setAsset accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=createRepository accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setName accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setFoundryContext accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=onPersistableCreate accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setLifecycleState accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setRollbackLifecycleState accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setRefreshed accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setDom0Uuid accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setSharePath accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setSimpleName accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setFileSystem accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=addRepository accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setManagerUuid accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setVersion accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=addJobOperation accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setSimpleName accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setDescription accessLevel=6
          Class=InternalJobDbImpl vessel_id=1607 method=setCompletedStep accessLevel=6
          Class=InternalJobDbImpl vessel_id=1607 method=setAssociatedHandles accessLevel=6
          Class=ClusterDbImpl vessel_id=1374 method=setCurrentJobOperationComplete accessLevel=6
          Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
          Class=InternalJobDbImpl vessel_id=1607 method=setTuringMachineFlag accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=setCurrentOperationToLater accessLevel=6
          Class=InternalJobDbImpl vessel_id=1607 method=setTuringMachineFlag accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
          Class=InternalJobDbImpl vessel_id=1607 method=setFailedOperation accessLevel=6
          Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=921 method=nextJobOperation accessLevel=6
          Class=LocalFileSystemDbImpl vessel_id=1616 method=nextJobOperation accessLevel=6
          Class=RepositoryDbImpl vessel_id=1621 method=nextJobOperation accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
          Class=StorageElementDbImpl vessel_id=1273 method=nextJobOperation accessLevel=6
          Class=ClusterDbImpl vessel_id=1374 method=nextJobOperation accessLevel=6
          Class=LocalFileServerDbImpl vessel_id=675 method=nextJobOperation accessLevel=6
          Completed Step: ROLLBACK
          Job failed commit (internal) due to OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_createFileSystem] failed for storage server [0004fb0000090000de3c84de0325cbb2] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
          at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:894)
          at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.createFileSystem(FileSystemConstruct.java:57)
          at com.oracle.ovm.mgr.op.physical.storage.FileSystemConstruct.action(FileSystemConstruct.java:49)
          at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
          at sun.reflect.GeneratedMethodAccessor728.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
          at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
          at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
          at com.oracle.ovm.mgr.api.physical.storage.LocalFileServerProxy.executeCurrentJobOperationAction(Unknown Source)
          at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
          at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
          at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
          at sun.reflect.GeneratedMethodAccessor1229.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
          at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
          at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
          at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
          at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
          at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
          at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
          at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
          at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
          at java.lang.Thread.run(Thread.java:662)
          Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
          at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
          at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
          at com.oracle.ovm.mgr.action.StoragePluginAction.createFileSystem(StoragePluginAction.java:890)
          ... 27 more
          Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_createFileSystem oracle.ocfs2.OCFS2.OCFS2Plugin 0004fb0000050000f00e264fcad6a2cb /dev/mapper/3600144f0c17a765000004fe11a280004 0, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.FileSystemBusyEx:'An ocfs2 filesystem already exists on /dev/mapper/3600144f0c17a765000004fe11a280004'
          Wed Jun 20 01:01:50 CST 2012
          at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
          at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
          ... 30 more


          ----------
          End of Job
          ----------
          • 2. Re: Can't create repository as iSCSI physical disk
            944494
            To frustrate me further, trying to remove my repository LUN from the access group hasn't worked.
            I begin by editing the storage access group and swapping the repository LUN I created back out and onto the left panel (where it is to be unused I imagine).
            What's ridiculous is that the LUN isn't mounted or used by anything (it didn't work, remember?). If it was easy to add to the group... why so hard to remove?
            ...to my surprise, I get this:-

            Perhaps it's because I am attempting to do something before something else has been properly cleaned up.
            It would be nice to be told of that, rather than have error messages like this thrust at you.

            Job Construction Phase
            ----------------------
            begin()
            Appended operation 'Storage Element Teardown' to object '34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)'.
            Appended operation 'Storage Element Teardown' to object '34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)'.
            Appended operation 'Storage Element UnPresent' to object '0004fb00001800000e2fa7b367ddf334 (repo01)'.
            commit()
            Completed Step: COMMIT

            Objects and Operations
            ----------------------
            Object (IN_USE): [Server] 34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)
            Operation: Storage Element Teardown
            Object (DELETING): [IscsiStoragePath] iqn.1988-12.com.oracle:fe7bba90add3 : iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643 (3600144f0c17a765000004fe138810007)
            Object (IN_USE): [Server] 34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)
            Operation: Storage Element Teardown
            Object (IN_USE): [AccessGroup] group01 @ 0004fb00000900003b330fb87739bbfe (group01)
            Object (IN_USE): [IscsiStorageTarget] iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643
            Object (IN_USE): [StorageElement] 0004fb00001800000e2fa7b367ddf334 (repo01)
            Operation: Storage Element UnPresent
            Object (IN_USE): [IscsiStorageInitiator] iqn.1988-12.com.oracle:fe7bba90add3
            Object (DELETING): [IscsiStoragePath] iqn.1988-12.com.oracle:43e520f2e5f : iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643 (3600144f0c17a765000004fe138810007)
            Object (IN_USE): [IscsiStorageInitiator] iqn.1988-12.com.oracle:43e520f2e5f

            Job Running Phase at 02:56 on Wed, Jun 20, 2012
            ----------------------------------------------
            Job Participants: [34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)]


            Actioner
            --------
            Starting operation 'Storage Element Teardown' on object '34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)'
            Sending storage element teardown command to server [ovm-dev-02] for element whose page 83 id is [3600144f0c17a765000004fe138810007]
            Completed operation 'Storage Element Teardown' completed with direction ==> DONE
            Starting operation 'Storage Element Teardown' on object '34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)'
            Sending storage element teardown command to server [ovm-dev-01] for element whose page 83 id is [3600144f0c17a765000004fe138810007]
            Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [teardown] failed for storage server [{access_grps=[{grp_name=default, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f], grp_modes=[]}, {grp_name=group01, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f, iqn.1988-12.com.oracle:fe7bba90add3], grp_modes=[]}], passwd=null, admin_passwd=W1a,1bT7, storage_id=[iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643], chap=false, access_host=zfs-app.icesa.catholic.edu.au, storage_server_id=2b34e1ce-5465-ecd1-9a10-c17a7650ba08, vol_groups=[{vol_alloc_sz=0, vol_free_sz=0, vol_used_sz=0, vol_name=ovmpool/local/default, vol_total_sz=0, vol_desc=}], username=null, name=0004fb00000900003b330fb87739bbfe, admin_user=cesa_ovm, uuid=0004fb00000900003b330fb87739bbfe, extra_info=OVM-iSCSI,OVM-iSCSI-Target, access_port=3260, storage_type=iSCSI, admin_host=zfs-app.icesa.catholic.edu.au}] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012...
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.op.physical.storage.StorageElementTeardown.action(StorageElementTeardown.java:75)
            at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
            at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
            at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)
            Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
            at com.oracle.ovm.mgr.action.StoragePluginAction.teardownStorageElement(StoragePluginAction.java:821)
            at com.oracle.ovm.mgr.op.physical.storage.StorageElementTeardown.action(StorageElementTeardown.java:71)
            ... 25 more
            Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
            at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
            at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
            at com.oracle.ovm.mgr.action.StoragePluginAction.teardownStorageElement(StoragePluginAction.java:817)
            ... 26 more
            Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
            at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
            ... 29 more


            FailedOperationCleanup
            ----------
            Starting failed operation 'Storage Element Teardown' cleanup on object 'ovm-dev-01'
            Complete rollback operation 'Storage Element Teardown' completed with direction=ovm-dev-01

            Rollbacker
            ----------
            Executing rollback operation 'Storage Element Teardown' on object '34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)'
            Complete rollback operation 'Storage Element Teardown' completed with direction=DONE
            Executing rollback operation 'Storage Element Teardown' on object '34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)'
            Complete rollback operation 'Storage Element Teardown' completed with direction=DONE

            Objects To Be Rolled Back
            -------------------------
            Object (IN_USE): [Server] 34:35:33:33:33:30:43:4e:37:37:34:37:30:33:42:42 (ovm-dev-02)
            Object (DELETING): [IscsiStoragePath] iqn.1988-12.com.oracle:fe7bba90add3 : iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643 (3600144f0c17a765000004fe138810007)
            Object (IN_USE): [Server] 34:35:33:33:33:30:43:4e:37:37:34:37:30:32:53:35 (ovm-dev-01)
            Object (IN_USE): [AccessGroup] group01 @ 0004fb00000900003b330fb87739bbfe (group01)
            Object (IN_USE): [IscsiStorageTarget] iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643
            Object (IN_USE): [StorageElement] 0004fb00001800000e2fa7b367ddf334 (repo01)
            Object (IN_USE): [IscsiStorageInitiator] iqn.1988-12.com.oracle:fe7bba90add3
            Object (DELETING): [IscsiStoragePath] iqn.1988-12.com.oracle:43e520f2e5f : iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643 (3600144f0c17a765000004fe138810007)
            Object (IN_USE): [IscsiStorageInitiator] iqn.1988-12.com.oracle:43e520f2e5f


            Write Methods Invoked
            -------------------
            Class=InternalJobDbImpl vessel_id=2295 method=addTransactionIdentifier accessLevel=6
            Class=StorageElementDbImpl vessel_id=1923 method=unpresent accessLevel=6
            Class=ServerDbImpl vessel_id=728 method=teardownStorageElements accessLevel=6
            Class=ServerDbImpl vessel_id=1686 method=teardownStorageElements accessLevel=6
            Class=AccessGroupDbImpl vessel_id=1941 method=removeStorageElement accessLevel=6
            Class=IscsiStorageInitiatorDbImpl vessel_id=845 method=deleteStoragePath accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2070 method=setLifecycleState accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2070 method=setRollbackLifecycleState accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2070 method=onPersistableClean accessLevel=6
            Class=StorageElementDbImpl vessel_id=1923 method=removeStoragePath accessLevel=6
            Class=IscsiStorageTargetDbImpl vessel_id=580 method=removeStoragePath accessLevel=6
            Class=IscsiStorageInitiatorDbImpl vessel_id=1803 method=deleteStoragePath accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2132 method=setLifecycleState accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2132 method=setRollbackLifecycleState accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2132 method=onPersistableClean accessLevel=6
            Class=StorageElementDbImpl vessel_id=1923 method=removeStoragePath accessLevel=6
            Class=IscsiStorageTargetDbImpl vessel_id=580 method=removeStoragePath accessLevel=6
            Class=InternalJobDbImpl vessel_id=2295 method=setCompletedStep accessLevel=6
            Class=InternalJobDbImpl vessel_id=2295 method=setAssociatedHandles accessLevel=6
            Class=ServerDbImpl vessel_id=728 method=setCurrentJobOperationComplete accessLevel=6
            Class=ServerDbImpl vessel_id=728 method=nextJobOperation accessLevel=6
            Class=ServerDbImpl vessel_id=1686 method=nextJobOperation accessLevel=6
            Class=InternalJobDbImpl vessel_id=2295 method=setFailedOperation accessLevel=6
            Class=ServerDbImpl vessel_id=728 method=nextJobOperation accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2132 method=nextJobOperation accessLevel=6
            Class=ServerDbImpl vessel_id=1686 method=nextJobOperation accessLevel=6
            Class=AccessGroupDbImpl vessel_id=1941 method=nextJobOperation accessLevel=6
            Class=IscsiStorageTargetDbImpl vessel_id=580 method=nextJobOperation accessLevel=6
            Class=StorageElementDbImpl vessel_id=1923 method=nextJobOperation accessLevel=6
            Class=IscsiStorageInitiatorDbImpl vessel_id=1803 method=nextJobOperation accessLevel=6
            Class=IscsiStoragePathDbImpl vessel_id=2070 method=nextJobOperation accessLevel=6
            Class=IscsiStorageInitiatorDbImpl vessel_id=845 method=nextJobOperation accessLevel=6
            Class=ServerDbImpl vessel_id=728 method=nextJobOperation accessLevel=6
            Class=ServerDbImpl vessel_id=1686 method=nextJobOperation accessLevel=6
            Completed Step: ROLLBACK
            Job failed commit (internal) due to OVMAPI_B000E Storage plugin command [teardown] failed for storage server [{access_grps=[{grp_name=default, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f], grp_modes=[]}, {grp_name=group01, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f, iqn.1988-12.com.oracle:fe7bba90add3], grp_modes=[]}], passwd=null, admin_passwd=W1a,1bT7, storage_id=[iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643], chap=false, access_host=zfs-app.icesa.catholic.edu.au, storage_server_id=2b34e1ce-5465-ecd1-9a10-c17a7650ba08, vol_groups=[{vol_alloc_sz=0, vol_free_sz=0, vol_used_sz=0, vol_name=ovmpool/local/default, vol_total_sz=0, vol_desc=}], username=null, name=0004fb00000900003b330fb87739bbfe, admin_user=cesa_ovm, uuid=0004fb00000900003b330fb87739bbfe, extra_info=OVM-iSCSI,OVM-iSCSI-Target, access_port=3260, storage_type=iSCSI, admin_host=zfs-app.icesa.catholic.edu.au}] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012...
            Wed Jun 20 02:56:08 CST 2012
            com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [teardown] failed for storage server [{access_grps=[{grp_name=default, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f], grp_modes=[]}, {grp_name=group01, grp_entries=[iqn.1988-12.com.oracle:43e520f2e5f, iqn.1988-12.com.oracle:fe7bba90add3], grp_modes=[]}], passwd=null, admin_passwd=W1a,1bT7, storage_id=[iqn.1986-03.com.sun:02:a0386184-4a49-eaeb-f2f0-b3b9070f3643], chap=false, access_host=zfs-app.icesa.catholic.edu.au, storage_server_id=2b34e1ce-5465-ecd1-9a10-c17a7650ba08, vol_groups=[{vol_alloc_sz=0, vol_free_sz=0, vol_used_sz=0, vol_name=ovmpool/local/default, vol_total_sz=0, vol_desc=}], username=null, name=0004fb00000900003b330fb87739bbfe, admin_user=cesa_ovm, uuid=0004fb00000900003b330fb87739bbfe, extra_info=OVM-iSCSI,OVM-iSCSI-Target, access_port=3260, storage_type=iSCSI, admin_host=zfs-app.icesa.catholic.edu.au}] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012...
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.op.physical.storage.StorageElementTeardown.action(StorageElementTeardown.java:75)
            at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
            at sun.reflect.GeneratedMethodAccessor835.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
            at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
            at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
            at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
            at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
            at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
            at sun.reflect.GeneratedMethodAccessor1118.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
            at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
            at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
            at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
            at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
            at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
            at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
            at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
            at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
            at java.lang.Thread.run(Thread.java:662)
            Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_B000E Storage plugin command [storage_plugin_deviceTeardown] failed for storage server [0004fb00000900003b330fb87739bbfe] failed with [com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012] OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.StoragePluginAction.processException(StoragePluginAction.java:1371)
            at com.oracle.ovm.mgr.action.StoragePluginAction.teardownStorageElement(StoragePluginAction.java:821)
            at com.oracle.ovm.mgr.op.physical.storage.StorageElementTeardown.action(StorageElementTeardown.java:71)
            ... 25 more
            Caused by: com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovm-dev-01 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
            at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
            at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
            at com.oracle.ovm.mgr.action.StoragePluginAction.teardownStorageElement(StoragePluginAction.java:817)
            ... 26 more
            Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@10.2.32.62:8899/api/2 storage_plugin_deviceTeardown oracle.s7k.SCSIPlugin.SCSIPlugin, Status: org.apache.xmlrpc.XmlRpcException: OSCPlugin.OperationFailedEx:'Unable to tear down multipath device /dev/mapper/3600144f0c17a765000004fe138810007: device-mapper: remove ioctl failed: Device or resource busy\nCommand failed\n'
            Wed Jun 20 02:56:08 CST 2012
            at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
            at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
            ... 29 more


            ----------
            End of Job
            ----------

            Edited by: 941491 on Jun 19, 2012 7:59 PM

            Edited by: 941491 on Jun 19, 2012 8:03 PM

            Edited by: 941491 on Jun 19, 2012 8:06 PM
            • 3. Re: Can't create repository as iSCSI physical disk
              budachst
              Well, I am currently in the same boat - except the issue with the storage group renam/remove.
              However, I can explain what happend before, which goes briefly like this:

              - tried to create a sr on a rather large iSCSI LUN (5.5 TB m in this case), which took more than 2 mins. what caused OVMM to abort the job, instead of being patient and waiting for the ocfs2 initialization to finish
              - then you tried that again and failed since there was already an ocfs2 volume structure on the LUN, which OVMM refused to overwrite, which is okay, since you wouldn't want OVMM to proceed on a LUN that maybe holds actual data

              I have also tried to create the new SR using the ovm_utils, but ovm_reprocontrol didn't manage to find any suitable LUN on my server, which really surprises me, since OVMM did not had that problem. I have, however, wiped the LUN with zeros (well the first 100 MB) before starting another attempt.

              I have now opened a SR wih Oracle support, since I need this 5.5 TB iSCSI LUN to be operational.
              • 4. Re: Can't create repository as iSCSI physical disk
                944494
                Yes, I need this iSCSI lun to work, as i am charged with testing OVM's reputed Advanced cloning, and I can't do it unless there's ZFS on the SAN backend.

                At the moment I can only mount a repo as an NFS share, and let me just say, that doing cloning on an NFS share involves the 'dd' command, and not the zfs clone commands. You can imagine just how long it is taking to get the 30GB Ebiz DB template to clone!
                • 5. Re: Can't create repository as iSCSI physical disk
                  budachst
                  Well… I think I will get some feedback from Oracle Support today, outlining how to bypass OVMM for creating SRs on iSCSI LUNs (bigger than 2 TB). If it's something I can share - I will do it.
                  • 6. Re: Can't create repository as iSCSI physical disk
                    budachst
                    Hi,

                    I just got a note from Oracle Support. There is a patch available for OVMM, for those who have support coverage. This OVMM update hasn't made it to edelivery yet, but I guess it will be available there shortly as well.

                    For anyone who has acces to MOS, this is the patch number:

                    Patch 14227416: ORACLE VM MANAGER PTACH 3.1.1-365 RELEASE (Patch)

                    I have installed it and tried it out on a 5.5 TB iSCSI LUN and it worked as advertised.

                    Cheers,
                    budy

                    Edited by: budachst on Jun 22, 2012 2:38 PM

                    If you don't have support coverage for OVM, you can try out the ovm_utils, which in my case failed to find a suitable LUN on my VM servers, to create a new repo, but ymmw:

                    [https://blogs.oracle.com/wim/entry/using_ovm_utils]
                    • 7. Re: Can't create repository as iSCSI physical disk
                      944494
                      I applied the patch to OVMM, and lo and behold, I was able to attach the iSCSI LUN and use it as a repository.

                      Indeed, it took a long time for the repository to be created on the virtual ZFS-SA I had (about 5 minutes!), but my patience was eventually rewarded.