This discussion is archived
6 Replies Latest reply: Dec 6, 2012 4:57 AM by user12273962 RSS

Unable to add new VM server 3.1.1 to existing Server Pool

964141 Newbie
Currently Being Moderated
I tried to add new vm server to existing server pool which has two vm servers already.
We're using iSCSI for shared disk and each 3 servers connected correctly.
But when i tried to add 3rd server(lxdcvirt03.test.com) to the server pool, below error occured.

---------------------------------------------------------------------------------------------------------------------------
Job Construction Phase
----------------------
begin()
Appended operation 'Server Role Update' to object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'.
Appended operation 'Server Join Server Pool' to object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'.
Appended operation 'Server Pool Member Update' to object '0004fb00000200001f4653fad1610ee6 (DcOvmCluster)'.
Appended operation 'Server Cluster Configuration Update' to object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'.
Appended operation 'Server Cluster Configuration Update' to object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'.
Appended operation 'Server Cluster Configure' to object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'.
Appended operation 'Server Cluster Join' to object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'.
commit()
Completed Step: COMMIT

Objects and Operations
----------------------
Object (IN_USE): [Cluster] 1f4653fad1610ee6
Object (IN_USE): [ServerPool] 0004fb00000200001f4653fad1610ee6 (DcOvmCluster)
Operation: Server Pool Member Update
Object (IN_USE): [Server] 44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)
Operation: Server Cluster Configuration Update
Object (IN_USE): [Server] 44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)
Operation: Server Cluster Configuration Update
Object (IN_USE): [Server] 44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)
Operation: Server Role Update
Operation: Server Join Server Pool
Operation: Server Cluster Configure
Operation: Server Cluster Join

Job Running Phase at 13:38 on Fri, Nov 30, 2012
----------------------------------------------
Job Participants: []


Actioner
--------
Starting operation 'Server Pool Member Update' on object '0004fb00000200001f4653fad1610ee6 (DcOvmCluster)'
Completed operation 'Server Pool Member Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Role Update' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Role Update' completed with direction ==> DONE
Starting operation 'Server Join Server Pool' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Join Server Pool' completed with direction ==> LATER
Starting operation 'Server Pool Member Update' on object '0004fb00000200001f4653fad1610ee6 (DcOvmCluster)'
Completed operation 'Server Pool Member Update' completed with direction ==> DONE
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configure' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Cluster Configure' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Join' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Cluster Join' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Join Server Pool' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Join Server Pool' completed with direction ==> DONE
Starting operation 'Server Cluster Configure' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Cluster Configure' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> DONE
Starting operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> DONE
Starting operation 'Server Cluster Join' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Completed operation 'Server Cluster Join' completed with direction ==> LATER
Starting operation 'Server Cluster Configure' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: lxdcvirt03.test.com failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.168.21.205:8899/api/2 configure_server_for_cluster lun /dev/mapper/36000d310001e7c000000000000000088 0004fb0000050000496ec5c83dcab2c6 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/36000d310001e7c000000000000000088', '/poolfsmnt/0004fb0000050000496ec5c83dcab2c6'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/36000d310001e7c000000000000000088 on /poolfsmnt/0004fb0000050000496ec5c83dcab2c6. Check 'dmesg' for more information on this error.
stdout:
Fri Nov 30 13:38:51 EST 2012
Fri Nov 30 13:38:51 EST 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
at com.oracle.ovm.mgr.action.ClusterAction.configureServerForCluster(ClusterAction.java:88)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.configureCluster(ServerClusterConfigure.java:139)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.action(ServerClusterConfigure.java:58)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.168.21.205:8899/api/2 configure_server_for_cluster lun /dev/mapper/36000d310001e7c000000000000000088 0004fb0000050000496ec5c83dcab2c6 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/36000d310001e7c000000000000000088', '/poolfsmnt/0004fb0000050000496ec5c83dcab2c6'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/36000d310001e7c000000000000000088 on /poolfsmnt/0004fb0000050000496ec5c83dcab2c6. Check 'dmesg' for more information on this error.
stdout:
Fri Nov 30 13:38:51 EST 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
... 30 more


FailedOperationCleanup
----------
Starting failed operation 'Server Cluster Configure' cleanup on object 'lxdcvirt03.test.com'
Complete rollback operation 'Server Cluster Configure' completed with direction=lxdcvirt03.test.com

Rollbacker
----------
Executing rollback operation 'Server Pool Member Update' on object '0004fb00000200001f4653fad1610ee6 (DcOvmCluster)'
Complete rollback operation 'Server Pool Member Update' completed with direction=LATER
Executing rollback operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)'
Complete rollback operation 'Server Cluster Configuration Update' completed with direction=DONE
Executing rollback operation 'Server Cluster Configuration Update' on object '44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)'
Complete rollback operation 'Server Cluster Configuration Update' completed with direction=DONE
Executing rollback operation 'Server Cluster Configure' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Complete rollback operation 'Server Cluster Configure' completed with direction=DONE
Executing rollback operation 'Server Join Server Pool' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Complete rollback operation 'Server Join Server Pool' completed with direction=DONE
Executing rollback operation 'Server Role Update' on object '44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)'
Complete rollback operation 'Server Role Update' completed with direction=DONE
Executing rollback operation 'Server Pool Member Update' on object '0004fb00000200001f4653fad1610ee6 (DcOvmCluster)'
Complete rollback operation 'Server Pool Member Update' completed with direction=DONE

Objects To Be Rolled Back
-------------------------
Object (IN_USE): [Cluster] 1f4653fad1610ee6
Object (IN_USE): [ServerPool] 0004fb00000200001f4653fad1610ee6 (DcOvmCluster)
Object (IN_USE): [Server] 44:45:4c:4c:39:00:10:4a:80:58:b2:c0:4f:43:32:53 (lxdcvirt02.test.com)
Object (IN_USE): [Server] 44:45:4c:4c:34:00:10:38:80:53:c4:c0:4f:43:32:53 (lxdcvirt01.test.com)
Object (IN_USE): [Server] 44:45:4c:4c:39:00:10:34:80:59:c3:c0:4f:39:32:53 (lxdcvirt03.test.com)


Write Methods Invoked
-------------------
Class=InternalJobDbImpl vessel_id=106845 method=addTransactionIdentifier accessLevel=6
Class=ServerPoolDbImpl vessel_id=505 method=addServer accessLevel=6
Class=ServerDbImpl vessel_id=105910 method=lock accessLevel=6
Class=ServerDbImpl vessel_id=105910 method=addServerRole accessLevel=6
Class=ServerDbImpl vessel_id=105910 method=addServerRole accessLevel=6
Class=ServerDbImpl vessel_id=105910 method=addServerRole accessLevel=6
Class=ServerPoolDbImpl vessel_id=505 method=addServerInternal accessLevel=6
Class=ServerDbImpl vessel_id=105910 method=setServerPool accessLevel=6
Class=ClusterDbImpl vessel_id=511 method=allocateSlotForServer accessLevel=6
Class=ClusterDbImpl vessel_id=511 method=addServer accessLevel=6
.......
Class=ServerDbImpl vessel_id=105910 method=nextJobOperation accessLevel=6
Class=ServerPoolDbImpl vessel_id=505 method=nextJobOperation accessLevel=6
Completed Step: ROLLBACK
Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: dispatch to server: lxdcvirt03.test.com failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.168.21.205:8899/api/2 configure_server_for_cluster lun /dev/mapper/36000d310001e7c000000000000000088 0004fb0000050000496ec5c83dcab2c6 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/36000d310001e7c000000000000000088', '/poolfsmnt/0004fb0000050000496ec5c83dcab2c6'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/36000d310001e7c000000000000000088 on /poolfsmnt/0004fb0000050000496ec5c83dcab2c6. Check 'dmesg' for more information on this error.
stdout:
Fri Nov 30 13:38:51 EST 2012
Fri Nov 30 13:38:51 EST 2012
com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: lxdcvirt03.test.com failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.168.21.205:8899/api/2 configure_server_for_cluster lun /dev/mapper/36000d310001e7c000000000000000088 0004fb0000050000496ec5c83dcab2c6 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/36000d310001e7c000000000000000088', '/poolfsmnt/0004fb0000050000496ec5c83dcab2c6'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/36000d310001e7c000000000000000088 on /poolfsmnt/0004fb0000050000496ec5c83dcab2c6. Check 'dmesg' for more information on this error.
stdout:
Fri Nov 30 13:38:51 EST 2012
Fri Nov 30 13:38:51 EST 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:507)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:444)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:378)
at com.oracle.ovm.mgr.action.ClusterAction.configureServerForCluster(ClusterAction.java:88)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.configureCluster(ServerClusterConfigure.java:139)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.action(ServerClusterConfigure.java:58)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1009)
at sun.reflect.GeneratedMethodAccessor494.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:822)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:245)
at com.oracle.ovm.mgr.api.physical.ServerProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:218)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:309)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1140)
at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:330)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:290)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:773)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@192.168.21.205:8899/api/2 configure_server_for_cluster lun /dev/mapper/36000d310001e7c000000000000000088 0004fb0000050000496ec5c83dcab2c6 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/36000d310001e7c000000000000000088', '/poolfsmnt/0004fb0000050000496ec5c83dcab2c6'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/36000d310001e7c000000000000000088 on /poolfsmnt/0004fb0000050000496ec5c83dcab2c6. Check 'dmesg' for more information on this error.
stdout:
Fri Nov 30 13:38:51 EST 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:798)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:503)
... 30 more


----------
End of Job
----------



*/var/log/messages*
---------------------------------------------------------------------------------------------------------------------------------
Nov 30 13:38:24 lxdcvirt03 twisted: [-] Log opened.
Nov 30 13:38:24 lxdcvirt03 twisted: [-] twistd 8.2.0 (/usr/bin/python 2.4.3) starting up.
Nov 30 13:38:24 lxdcvirt03 twisted: [-] reactor class: twisted.internet.selectreactor.SelectReactor.
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor] Rescanning all plugins
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.xen_plugin] Starting plugin process /usr/lib/python2.4/site-packages/monitor/plugins/xen_plugin.py
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.xen_plugin] Plugin process /usr/lib/python2.4/site-packages/monitor/plugins/xen_plugin.py launched, PID 14519
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.xen_plugin] Process 14519 started, launching watchdog with gracetime 3600
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.oel] Starting plugin process /usr/lib/python2.4/site-packages/monitor/plugins/oel.py
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.oel] Plugin process /usr/lib/python2.4/site-packages/monitor/plugins/oel.py launched, PID 14520
Nov 30 13:38:24 lxdcvirt03 twisted: [monitor.plugin.oel] Process 14520 started, launching watchdog with gracetime 3600
Nov 30 13:38:26 lxdcvirt03 kernel: OCFS2 Node Manager 1.8.0
Nov 30 13:38:26 lxdcvirt03 kernel: OCFS2 DLM 1.8.0
Nov 30 13:38:26 lxdcvirt03 kernel: ocfs2: Registered cluster interface o2cb
Nov 30 13:38:26 lxdcvirt03 kernel: OCFS2 DLMFS 1.8.0
Nov 30 13:38:26 lxdcvirt03 kernel: OCFS2 User DLM kernel interface loaded
Nov 30 13:38:26 lxdcvirt03 o2cb.init: online 1f4653fad1610ee6
Nov 30 13:38:26 lxdcvirt03 kernel: o2hb: Heartbeat mode set to global
Nov 30 13:38:31 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:31 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:33 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:33 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:35 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:35 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:37 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:37 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:39 lxdcvirt03 kernel: o2hb: Heartbeat started on region 0004FB0000050000496EC5C83DCAB2C6 (dm-12)
Nov 30 13:38:39 lxdcvirt03 o2hbmonitor: Starting
Nov 30 13:38:39 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:39 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:40 lxdcvirt03 kernel: OCFS2 1.8.0
Nov 30 13:38:40 lxdcvirt03 kernel: o2cb: This node is not connected to nodes: 0 1.
Nov 30 13:38:40 lxdcvirt03 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Nov 30 13:38:40 lxdcvirt03 kernel: (mount.ocfs2,14869,14):ocfs2_dlm_init:3001 ERROR: status = -22
Nov 30 13:38:40 lxdcvirt03 kernel: (mount.ocfs2,14869,14):ocfs2_mount_volume:1883 ERROR: status = -22
Nov 30 13:38:40 lxdcvirt03 kernel: ocfs2: Unmounting device (252,12) on (node 0)
Nov 30 13:38:40 lxdcvirt03 kernel: (mount.ocfs2,14869,14):ocfs2_fill_super:1240 ERROR: status = -22
Nov 30 13:38:41 lxdcvirt03 kernel: o2hb: Region 0004FB0000050000496EC5C83DCAB2C6 (dm-12) is now a quorum device
Nov 30 13:38:41 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:41 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:43 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:43 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:45 lxdcvirt03 kernel: o2cb: This node is not connected to nodes: 0 1.
Nov 30 13:38:45 lxdcvirt03 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Nov 30 13:38:45 lxdcvirt03 kernel: (mount.ocfs2,14971,12):ocfs2_dlm_init:3001 ERROR: status = -22
Nov 30 13:38:45 lxdcvirt03 kernel: (mount.ocfs2,14971,12):ocfs2_mount_volume:1883 ERROR: status = -22
Nov 30 13:38:45 lxdcvirt03 kernel: ocfs2: Unmounting device (252,12) on (node 0)
Nov 30 13:38:45 lxdcvirt03 kernel: (mount.ocfs2,14971,12):ocfs2_fill_super:1240 ERROR: status = -22
Nov 30 13:38:45 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:45 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:47 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:47 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:49 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:49 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:50 lxdcvirt03 kernel: o2cb: This node is not connected to nodes: 0 1.
Nov 30 13:38:50 lxdcvirt03 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Nov 30 13:38:50 lxdcvirt03 kernel: (mount.ocfs2,15055,14):ocfs2_dlm_init:3001 ERROR: status = -22
Nov 30 13:38:50 lxdcvirt03 kernel: (mount.ocfs2,15055,14):ocfs2_mount_volume:1883 ERROR: status = -22
Nov 30 13:38:50 lxdcvirt03 kernel: ocfs2: Unmounting device (252,12) on (node 0)
Nov 30 13:38:50 lxdcvirt03 kernel: (mount.ocfs2,15055,14):ocfs2_fill_super:1240 ERROR: status = -22
Nov 30 13:38:50 lxdcvirt03 o2cb.init: offline ocfs2 0
Nov 30 13:38:51 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:51 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7
Nov 30 13:38:53 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
Nov 30 13:38:53 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7


Any ideas?

Thanks,
Jay

Edited by: 961138 on Dec 2, 2012 3:40 PM
  • 1. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    964141 Newbie
    Currently Being Moderated
    I have found something wired.

    Here is our server's network configration

    Management : 192.168.21.x
    Cluster Heartbeat : 10.20.200.x / 10.10.201.x

    -----------------------------------------------------------------------
    [root@lxdcvirt01 ~]# cat /etc/ocfs2/cluster.conf
    heartbeat:
    region = 0004FB0000050000496EC5C83DCAB2C6
    cluster = 1f4653fad1610ee6

    node:
    ip_port = 7777
    ip_address = 10.10.200.1
    number = 0
    name = lxdcvirt01.test.com
    cluster = 1f4653fad1610ee6

    node:
    ip_port = 7777
    ip_address = 10.10.200.2
    number = 1
    name = lxdcvirt02.test.com
    cluster = 1f4653fad1610ee6

    cluster:
    node_count = 2
    heartbeat_mode = global
    name = 1f4653fad1610ee6
    -----------------------------------------------------------------------

    But the system use management IP Add for heartbeat network as below;

    [root@lxdcvirt01 ~]# netstat -na |grep 7777
    tcp 0 0 192.168.21.201:7777 0.0.0.0:* LISTEN
    tcp 0 0 192.168.21.201:7777 192.168.21.203:50211 ESTABLISHED


    That's why message logged that connection other nodes failed...

    Nov 30 13:38:51 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt01.test.com (num 0) at 10.10.200.1:7777 shutdown, state 7
    Nov 30 13:38:51 lxdcvirt03 kernel: o2net: Connection to node lxdcvirt02.test.com (num 1) at 10.10.200.2:7777 shutdown, state 7

    Any ideas?

    Thanks,
    Jay
  • 2. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    user12273962 Pro
    Currently Being Moderated
    You must define logical networks to separate traffic from the default bond. Look under your "network" tab in the VM manager.

    Why do you list two separate subnets for the heartbeat?
  • 3. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    964141 Newbie
    Currently Being Moderated
    Actually, each Oracle VM server has 8 nic

    2 for management : bonded - *192.168.21.x*
    2 for Virtual Machine : bonded
    2 for iSCSI
    2 for Heartbeat & live migration : *10.10.200.x / 10.10.201.x*

    The problem is 10.10.200.x 10.10.201.x have been set for hearbeat at ovm manager but as you can see below,
    192.168.21.x which is for management used for heartbeat.

    [root@lxdcvirt02 ~]# netstat -na | grep 7777
    tcp 0 0 192.168.21.203:7777 0.0.0.0:* LISTEN
    tcp 0 0 192.168.21.203:50211 192.168.21.201:7777 ESTABLISHED

    After changing nic role to heartbeat, the system hasn't been rebooted.
    I think does it need to be rebooted to recognize which port is for heartbeat?

    Thanks,
    Jay
  • 4. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    user12273962 Pro
    Currently Being Moderated
    You might have to reboot.... did you define separate logical networks for heartbeat and live migration?
  • 5. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    964141 Newbie
    Currently Being Moderated
    2 nics is used for both heartbeat and live migrate.
    Do i need to separate these two network?

    Also is it possible to configure two nic port for heartbeat or live migration?
    or only each one nic can be function as heartbeat and live migrate even it can be multi checked on ovm manager?

    Thanks
    Jay

    Edited by: 961138 on Dec 4, 2012 3:19 PM
  • 6. Re: Unable to add new VM server 3.1.1 to existing Server Pool
    user12273962 Pro
    Currently Being Moderated
    personally... I prefer nothing on the heartbeat subnet but the heartbeat itself. Yes. I would seperate the two. Just my opinion. if you must share... share the live migration and the management subnet.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points