This discussion is archived
5 Replies Latest reply: Apr 29, 2013 12:08 AM by Alexander Ryzhkov RSS

Can't add server in server_pool. mount.ocfs2: Invalid argument while mount

Alexander Ryzhkov Newbie
Currently Being Moderated
Hello,

I can't add server in server_pool.
Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovms2 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.35.72:8899/api/3 configure_server_for_cluster lun /dev/mapper/1HITACHI_D60046110000 0004fb000005000079d59f40fe50afa1 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/1HITACHI_D60046110000', '/poolfsmnt/0004fb000005000079d59f40fe50afa1'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
dmesg
Apr 24 16:49:26 ovms2 kernel: OCFS2 Node Manager 1.8.0
Apr 24 16:49:26 ovms2 kernel: OCFS2 DLM 1.8.0
Apr 24 16:49:26 ovms2 kernel: ocfs2: Registered cluster interface o2cb
Apr 24 16:49:26 ovms2 kernel: OCFS2 DLMFS 1.8.0
Apr 24 16:49:26 ovms2 kernel: OCFS2 User DLM kernel interface loaded
Apr 24 16:49:26 ovms2 o2cb.init: online c9bb270f8059b4c6
Apr 24 16:49:26 ovms2 kernel: o2hb: Heartbeat mode set to global
Apr 24 16:49:30 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:48817
Apr 24 16:49:30 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:52749
Apr 24 16:49:30 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:32 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:56738
Apr 24 16:49:32 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:52543
Apr 24 16:49:32 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:34 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:40711
Apr 24 16:49:34 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:34499
Apr 24 16:49:34 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:36 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:40404
Apr 24 16:49:36 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:59315
Apr 24 16:49:36 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:38 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:44580
Apr 24 16:49:38 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:45247
Apr 24 16:49:38 ovms2 kernel: o2hb: Heartbeat started on region 0004FB000005000079D59F40FE50AFA1 (dm-1)
Apr 24 16:49:38 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:38 ovms2 o2hbmonitor: Starting
Apr 24 16:49:38 ovms2 kernel: OCFS2 1.8.0
Apr 24 16:49:38 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
Apr 24 16:49:38 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_dlm_init:3001 ERROR: status = -22
Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_mount_volume:1883 ERROR: status = -22
Apr 24 16:49:38 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_fill_super:1238 ERROR: status = -22
Apr 24 16:49:40 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:39694
Apr 24 16:49:40 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:34463
Apr 24 16:49:40 ovms2 kernel: o2hb: Region 0004FB000005000079D59F40FE50AFA1 (dm-1) is now a quorum device
Apr 24 16:49:40 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:42 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:33941
Apr 24 16:49:42 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:42105
Apr 24 16:49:42 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:43 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
Apr 24 16:49:43 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_dlm_init:3001 ERROR: status = -22
Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_mount_volume:1883 ERROR: status = -22
Apr 24 16:49:43 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_fill_super:1238 ERROR: status = -22
Apr 24 16:49:44 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:48827
Apr 24 16:49:44 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:56865
Apr 24 16:49:44 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:46 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:42245
Apr 24 16:49:46 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:37841
Apr 24 16:49:46 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:48 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:57559
Apr 24 16:49:48 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:42451
Apr 24 16:49:48 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
Apr 24 16:49:49 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
Apr 24 16:49:49 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_dlm_init:3001 ERROR: status = -22
Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_mount_volume:1883 ERROR: status = -22
Apr 24 16:49:49 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_fill_super:1238 ERROR: status = -22
Apr 24 16:49:49 ovms2 o2cb.init: offline ocfs2 0

Regards,
Alexander.
  • 1. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
    user12273962 Pro
    Currently Being Moderated
    Looks like you have a node in the pool that is panicking and rebooting. Is the ServerPool file system a FC, iScsi or NFS pool? Check to make sure the server in question has access to the Server Pool path.
  • 2. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
    Alexander Ryzhkov Newbie
    Currently Being Moderated
    Server pool on FC file system. Server has access to Server Pool path.
    [root@ovms2 ~]# blkid
    /dev/sda1: LABEL="/boot1" UUID="6e5ef02d-b594-437e-a971-2001634346c4" TYPE="ext3"
    /dev/sda2: LABEL="/1" UUID="ce2b38df-78a1-48e6-bfc2-ab7ceb241e86" TYPE="ext3"
    /dev/sda3: LABEL="SWAP-sda3" UUID="6cb23cef-c331-48eb-9f7b-c58c9c202125" TYPE="swap"
    /dev/sr0: LABEL="Oracle VM Server cd 1 20130219" TYPE="iso9660"
    /dev/mapper/1HITACHI_D60046110000: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
    /dev/mapper/1HITACHI_D60046110001: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
    /dev/mapper/3600144f0a1eee03a000051092c620003: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
    /dev/sdd: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
    /dev/sde: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
    /dev/sdg: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
    /dev/sdf: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
    /dev/sdj: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
    /dev/sdl: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
    /dev/sdk: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
    /dev/sdr: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
    /dev/sdp: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
  • 4. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
    user12273962 Pro
    Currently Being Moderated
    I actually just ran into the same issue. Exact same error adding a server to a server pool. I'll let you know what I find out.

    added...

    Looks like I had a server crash in the middle of adding to the pool and the master server became unstable. I ended up rebooting the master server.

    Edited by: user12273962 on Apr 26, 2013 1:18 PM
  • 5. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
    Alexander Ryzhkov Newbie
    Currently Being Moderated
    I rebooted master server(ovms3). Now master server is another server - ovms1.
    ovms3 lost mounts all repositories and cluster file system and can't mount with same error:
    [root@ovms3 ~]# mount -t ocfs2 -o rw,_netdev,heartbeat=global /dev/mapper/1HITACHI_D60046110000 /poolfsmnt/0004fb000005000079d59f40fe50afa1
    mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
    Apr 29 11:05:00 ovms3 kernel: OCFS2 1.8.0
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,20):dlm_send_nodeinfo:1301 ERROR: node mismatch -22, node 0
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,20):dlm_try_to_join_domain:1685 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):dlm_join_domain:1955 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):dlm_register_domain:2214 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):o2cb_cluster_connect:358 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_dlm_init:3001 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_mount_volume:1883 ERROR: status = -22
    Apr 29 11:05:00 ovms3 kernel: ocfs2: Unmounting device (252,1) on (node 0)
    Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_fill_super:1238 ERROR: status = -22

    [root@ovms3 ~]# o2cluster -r
    o2cb,c9bb270f8059b4c6,global
    [root@ovms3 ~]#
    [root@ovms3 ~]# /etc/init.d/o2cb status
    Driver for "configfs": Loaded
    Filesystem "configfs": Mounted
    Stack glue driver: Loaded
    Stack plugin "o2cb": Loaded
    Driver for "ocfs2_dlmfs": Loaded
    Filesystem "ocfs2_dlmfs": Mounted
    Checking O2CB cluster "c9bb270f8059b4c6": Online
    Heartbeat dead threshold: 61
    Network idle timeout: 60000
    Network keepalive delay: 2000
    Network reconnect delay: 2000
    Heartbeat mode: Global
    Checking O2CB heartbeat: Active
    0004FB000005000079D59F40FE50AFA1 /dev/dm-1
    Nodes in O2CB cluster: 0 1 2 3 4
    [root@ovms3 ~]# df -h
    [root@ovms3 ~]# mount -t ocfs2 -o rw,_netdev,heartbeat=global /dev/mapper/1HITACHI_D60046110000 /poolfsmnt/0004fb000005000079d59f40fe50afa1
    mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
    [root@ovms3 ~]# mount
    /dev/sda2 on / type ext3 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)
    debugfs on /sys/kernel/debug type debugfs (rw)
    xenfs on /proc/xen type xenfs (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    none on /var/lib/xenstored type tmpfs (rw)
    configfs on /sys/kernel/config type configfs (rw)
    ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
    --------------
    Maybe need run fsck.ocfs2 on ovms1, but I'm afraid ?
    [root@ovms1 opt]# fsck.ocfs2 /dev/mapper/1HITACHI_D60046110000
    fsck.ocfs2 1.8.2

    WARNING!!! Running fsck.ocfs2 on a mounted filesystem may cause SEVERE filesystem damage.

    Do you really want to continue (y/N): n

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points