5 Replies Latest reply: Apr 29, 2013 2:08 AM by Alexander Ryzhkov RSS

    Can't add server in server_pool. mount.ocfs2: Invalid argument while mount

    Alexander Ryzhkov
      Hello,

      I can't add server in server_pool.
      Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: ovms2 failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.35.72:8899/api/3 configure_server_for_cluster lun /dev/mapper/1HITACHI_D60046110000 0004fb000005000079d59f40fe50afa1 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:Command: ['mount', '/dev/mapper/1HITACHI_D60046110000', '/poolfsmnt/0004fb000005000079d59f40fe50afa1'] failed (1): stderr: mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
      dmesg
      Apr 24 16:49:26 ovms2 kernel: OCFS2 Node Manager 1.8.0
      Apr 24 16:49:26 ovms2 kernel: OCFS2 DLM 1.8.0
      Apr 24 16:49:26 ovms2 kernel: ocfs2: Registered cluster interface o2cb
      Apr 24 16:49:26 ovms2 kernel: OCFS2 DLMFS 1.8.0
      Apr 24 16:49:26 ovms2 kernel: OCFS2 User DLM kernel interface loaded
      Apr 24 16:49:26 ovms2 o2cb.init: online c9bb270f8059b4c6
      Apr 24 16:49:26 ovms2 kernel: o2hb: Heartbeat mode set to global
      Apr 24 16:49:30 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:48817
      Apr 24 16:49:30 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:52749
      Apr 24 16:49:30 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:32 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:56738
      Apr 24 16:49:32 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:52543
      Apr 24 16:49:32 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:34 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:40711
      Apr 24 16:49:34 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:34499
      Apr 24 16:49:34 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:36 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:40404
      Apr 24 16:49:36 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:59315
      Apr 24 16:49:36 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:38 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:44580
      Apr 24 16:49:38 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:45247
      Apr 24 16:49:38 ovms2 kernel: o2hb: Heartbeat started on region 0004FB000005000079D59F40FE50AFA1 (dm-1)
      Apr 24 16:49:38 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:38 ovms2 o2hbmonitor: Starting
      Apr 24 16:49:38 ovms2 kernel: OCFS2 1.8.0
      Apr 24 16:49:38 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
      Apr 24 16:49:38 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
      Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_dlm_init:3001 ERROR: status = -22
      Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_mount_volume:1883 ERROR: status = -22
      Apr 24 16:49:38 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
      Apr 24 16:49:38 ovms2 kernel: (mount.ocfs2,7366,5):ocfs2_fill_super:1238 ERROR: status = -22
      Apr 24 16:49:40 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:39694
      Apr 24 16:49:40 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:34463
      Apr 24 16:49:40 ovms2 kernel: o2hb: Region 0004FB000005000079D59F40FE50AFA1 (dm-1) is now a quorum device
      Apr 24 16:49:40 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:42 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:33941
      Apr 24 16:49:42 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:42105
      Apr 24 16:49:42 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:43 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
      Apr 24 16:49:43 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
      Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_dlm_init:3001 ERROR: status = -22
      Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_mount_volume:1883 ERROR: status = -22
      Apr 24 16:49:43 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
      Apr 24 16:49:43 ovms2 kernel: (mount.ocfs2,7448,1):ocfs2_fill_super:1238 ERROR: status = -22
      Apr 24 16:49:44 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:48827
      Apr 24 16:49:44 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:56865
      Apr 24 16:49:44 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:46 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:42245
      Apr 24 16:49:46 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:37841
      Apr 24 16:49:46 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:48 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.75:57559
      Apr 24 16:49:48 ovms2 kernel: o2net: Attempt to connect from unknown node at 172.16.35.74:42451
      Apr 24 16:49:48 ovms2 kernel: o2net: Connection to node ovms1 (num 0) at 172.16.35.71:7777 shutdown, state 7
      Apr 24 16:49:49 ovms2 kernel: o2cb: This node is not connected to nodes: 0 2.
      Apr 24 16:49:49 ovms2 kernel: o2cb: Cluster check failed. Fix errors before retrying.
      Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_dlm_init:3001 ERROR: status = -22
      Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_mount_volume:1883 ERROR: status = -22
      Apr 24 16:49:49 ovms2 kernel: ocfs2: Unmounting device (252,1) on (node 0)
      Apr 24 16:49:49 ovms2 kernel: (mount.ocfs2,7537,5):ocfs2_fill_super:1238 ERROR: status = -22
      Apr 24 16:49:49 ovms2 o2cb.init: offline ocfs2 0

      Regards,
      Alexander.
        • 1. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
          user12273962
          Looks like you have a node in the pool that is panicking and rebooting. Is the ServerPool file system a FC, iScsi or NFS pool? Check to make sure the server in question has access to the Server Pool path.
          • 2. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
            Alexander Ryzhkov
            Server pool on FC file system. Server has access to Server Pool path.
            [root@ovms2 ~]# blkid
            /dev/sda1: LABEL="/boot1" UUID="6e5ef02d-b594-437e-a971-2001634346c4" TYPE="ext3"
            /dev/sda2: LABEL="/1" UUID="ce2b38df-78a1-48e6-bfc2-ab7ceb241e86" TYPE="ext3"
            /dev/sda3: LABEL="SWAP-sda3" UUID="6cb23cef-c331-48eb-9f7b-c58c9c202125" TYPE="swap"
            /dev/sr0: LABEL="Oracle VM Server cd 1 20130219" TYPE="iso9660"
            /dev/mapper/1HITACHI_D60046110000: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
            /dev/mapper/1HITACHI_D60046110001: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
            /dev/mapper/3600144f0a1eee03a000051092c620003: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
            /dev/sdd: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
            /dev/sde: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
            /dev/sdg: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
            /dev/sdf: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
            /dev/sdj: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
            /dev/sdl: LABEL="OVS_POOL_FILESYSTEM" UUID="0004fb00-0005-0000-79d5-9f40fe50afa1" TYPE="ocfs2"
            /dev/sdk: LABEL="OVSd930cfdb8190f" UUID="0004fb00-0005-0000-c31d-930cfdb8190f" TYPE="ocfs2"
            /dev/sdr: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
            /dev/sdp: LABEL="OVSf87a927dc32d4" UUID="0004fb00-0005-0000-b92f-87a927dc32d4" TYPE="ocfs2"
            • 4. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
              user12273962
              I actually just ran into the same issue. Exact same error adding a server to a server pool. I'll let you know what I find out.

              added...

              Looks like I had a server crash in the middle of adding to the pool and the master server became unstable. I ended up rebooting the master server.

              Edited by: user12273962 on Apr 26, 2013 1:18 PM
              • 5. Re: Can't add server in server_pool. mount.ocfs2: Invalid argument while mount
                Alexander Ryzhkov
                I rebooted master server(ovms3). Now master server is another server - ovms1.
                ovms3 lost mounts all repositories and cluster file system and can't mount with same error:
                [root@ovms3 ~]# mount -t ocfs2 -o rw,_netdev,heartbeat=global /dev/mapper/1HITACHI_D60046110000 /poolfsmnt/0004fb000005000079d59f40fe50afa1
                mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
                Apr 29 11:05:00 ovms3 kernel: OCFS2 1.8.0
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,20):dlm_send_nodeinfo:1301 ERROR: node mismatch -22, node 0
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,20):dlm_try_to_join_domain:1685 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):dlm_join_domain:1955 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):dlm_register_domain:2214 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):o2cb_cluster_connect:358 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_dlm_init:3001 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_mount_volume:1883 ERROR: status = -22
                Apr 29 11:05:00 ovms3 kernel: ocfs2: Unmounting device (252,1) on (node 0)
                Apr 29 11:05:00 ovms3 kernel: (mount.ocfs2,24508,21):ocfs2_fill_super:1238 ERROR: status = -22

                [root@ovms3 ~]# o2cluster -r
                o2cb,c9bb270f8059b4c6,global
                [root@ovms3 ~]#
                [root@ovms3 ~]# /etc/init.d/o2cb status
                Driver for "configfs": Loaded
                Filesystem "configfs": Mounted
                Stack glue driver: Loaded
                Stack plugin "o2cb": Loaded
                Driver for "ocfs2_dlmfs": Loaded
                Filesystem "ocfs2_dlmfs": Mounted
                Checking O2CB cluster "c9bb270f8059b4c6": Online
                Heartbeat dead threshold: 61
                Network idle timeout: 60000
                Network keepalive delay: 2000
                Network reconnect delay: 2000
                Heartbeat mode: Global
                Checking O2CB heartbeat: Active
                0004FB000005000079D59F40FE50AFA1 /dev/dm-1
                Nodes in O2CB cluster: 0 1 2 3 4
                [root@ovms3 ~]# df -h
                [root@ovms3 ~]# mount -t ocfs2 -o rw,_netdev,heartbeat=global /dev/mapper/1HITACHI_D60046110000 /poolfsmnt/0004fb000005000079d59f40fe50afa1
                mount.ocfs2: Invalid argument while mounting /dev/mapper/1HITACHI_D60046110000 on /poolfsmnt/0004fb000005000079d59f40fe50afa1. Check 'dmesg' for more information on this error.
                [root@ovms3 ~]# mount
                /dev/sda2 on / type ext3 (rw)
                proc on /proc type proc (rw)
                sysfs on /sys type sysfs (rw)
                devpts on /dev/pts type devpts (rw,gid=5,mode=620)
                /dev/sda1 on /boot type ext3 (rw)
                tmpfs on /dev/shm type tmpfs (rw)
                debugfs on /sys/kernel/debug type debugfs (rw)
                xenfs on /proc/xen type xenfs (rw)
                none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
                sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
                none on /var/lib/xenstored type tmpfs (rw)
                configfs on /sys/kernel/config type configfs (rw)
                ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
                --------------
                Maybe need run fsck.ocfs2 on ovms1, but I'm afraid ?
                [root@ovms1 opt]# fsck.ocfs2 /dev/mapper/1HITACHI_D60046110000
                fsck.ocfs2 1.8.2

                WARNING!!! Running fsck.ocfs2 on a mounted filesystem may cause SEVERE filesystem damage.

                Do you really want to continue (y/N): n