9 Replies Latest reply: Feb 12, 2013 1:25 PM by 960874 RSS

    Oracle VM Manager 3.1.1: Discovering SAN Servers

    960874
      Even though I've gone through the discovery process for the iSCSI target iqn.201209.com.example.sn2:target1 (192.168.1.2) using the Oracle VM Manger 3.1.1 interface and both OracleVM 3.1.1 servers OVM-1 & OVM-2 have active sessions...
      [root@ovm-1 ~]# iscsiadm -m node
      192.168.1.1:3260,1 iqn.2011-09.com.example.sn1:target1
      192.168.1.2:3260,1 iqn.2011-09.com.example.sn2:target1
      
      [root@ovm-1 ~]# ls -1 /dev/disk/by-path/ip-192.168.1.2*
      /dev/disk/by-path/ip-192.168.1.2:3260-iscsi-iqn.2011-09.com.example.sn2:target1-lun-1
      /dev/disk/by-path/ip-192.168.1.2:3260-iscsi-iqn.2011-09.com.example.sn2:target1-lun-2
      [root@ovm-2 ~]# iscsiadm -m node
      192.168.1.2:3260,1 iqn.2011-09.com.example.sn2:target1
      192.168.1.1:3260,1 iqn.2011-09.com.example.sn1:target1
      
      [root@ovm-2 ~]# ls -1 /dev/disk/by-path/ip-192.168.1.2*
      /dev/disk/by-path/ip-192.168.1.2:3260-iscsi-iqn.2011-09.com.example.sn2:target1-lun-1
      /dev/disk/by-path/ip-192.168.1.2:3260-iscsi-iqn.2011-09.com.example.sn2:target1-lun-2
      ...the LUN's still are NOT displayed in the Oracle VM Manager 3.1.1 interface.

      Ideas? Suggestions?

      Eric Pretorious
      Truckee, CA
        • 1. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
          budachst
          Do the targets show up under /dev/mapper ?
          • 2. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
            Jurajl-Oracle
            Some suggestions:
            -make sure you have iscssi session OK: iscsiadm -m session
            -check if multipath is working fine: multipath -ll
            -make sure system time is correct on both nodes. and namager.
            -standard steps in Manager after OVM reinstall (ie.: perhaps rediscovering servers will help in your case):
            1. Discover Server
            2. Register Storage Array
            3. Present Repository
            4. Refresh Repository
            5. Rediscover Server

            :j
            • 3. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
              960874
              budachst wrote:
              Do the targets show up under /dev/mapper ?
              The targets do appear in the /dev/mapper directory of the two Oracle VM Server hosts +(but not the Oracle VM Manager host)+ but the directory contents are not the same (even though the iscsid sessions are the same):
              [root@ovm-1 ~]# ll /dev/mapper
              total 0
              brw-rw---- 1 root disk 252,   1 Oct  5 23:44 1IET_00010001
              brw-rw---- 1 root disk 252,   4 Oct  5 23:44 1IET_00010001p1
              brw-rw---- 1 root disk 252,   5 Oct  5 23:44 1IET_00010001p2
              brw-rw---- 1 root disk 252,   6 Oct  5 23:44 1IET_00010001p3
              brw-rw---- 1 root disk 252,   7 Oct  5 23:44 1IET_00010001p4
              brw-rw---- 1 root disk 252,   8 Oct  5 23:44 1IET_00010001p5
              brw-rw---- 1 root disk 252,   9 Oct  5 23:44 1IET_00010001p6
              brw-rw---- 1 root disk 252,  10 Oct  5 23:44 1IET_00010001p7
              brw-rw---- 1 root disk 252,   0 Oct  5 23:44 1IET_00010002
              brw-rw---- 1 root disk 252,   3 Oct  5 23:44 1IET_00010003
              brw-rw---- 1 root disk 252,   2 Oct  5 23:44 1IET_00010004
              crw------- 1 root root  10, 236 Oct  5 10:58 control
              brw-rw---- 1 root disk 252,  12 Oct  5 10:58 ovspoolfs
              [root@ovm-2 ~]# ll /dev/mapper
              total 0
              brw-rw---- 1 root disk 252,   0 Oct  5 23:45 1IET_00010001
              brw-rw---- 1 root disk 252,   2 Oct  5 23:45 1IET_00010002
              brw-rw---- 1 root disk 252,   3 Oct  5 23:45 1IET_00010003
              brw-rw---- 1 root disk 252,   1 Oct  5 23:45 1IET_00010004
              crw------- 1 root root  10, 236 Oct  2 21:03 control
              brw-rw---- 1 root disk 252,   5 Oct  3 18:26 dmnfs0
              brw-rw---- 1 root disk 252,   6 Oct  5 19:06 dmnfs1
              brw-rw---- 1 root disk 252,   4 Oct  2 21:03 ovspoolfs
              Eric Pretorious
              Truckee, CA
              • 4. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                960874
                jurajl wrote:
                Some suggestions:
                -make sure you have iscssi session OK: iscsiadm -m session
                -check if multipath is working fine: multipath -ll
                -make sure system time is correct on both nodes. and namager.
                -standard steps in Manager after OVM reinstall (ie.: perhaps rediscovering servers will help in your case):
                1. Discover Server
                2. Register Storage Array
                3. Present Repository
                4. Refresh Repository
                5. Rediscover Server
                <ol>
                <li>Yes, iscsid is running on both Oracle VM Servers...</li>
                [root@ovm-1 ~]# iscsiadm -m session -P 3
                iSCSI Transport Class version 2.0-870
                version 2.0-872
                Target: iqn.2011-09.com.example.sn1:target1
                   <<<snip>>>
                          ************************
                          Attached SCSI devices:
                          ************************
                          Host Number: 12     State: running
                          scsi12 Channel 00 Id 0 Lun: 0
                          scsi12 Channel 00 Id 0 Lun: 1
                          scsi12 Channel 00 Id 0 Lun: 2
                          scsi12 Channel 00 Id 0 Lun: 3
                          scsi12 Channel 00 Id 0 Lun: 4
                Target: iqn.2011-09.com.example.sn2:target1
                     Current Portal: 192.168.1.2:3260,1
                     Persistent Portal: 192.168.1.2:3260,1
                   <<<snip>>>
                          ************************
                          Attached SCSI devices:
                          ************************
                          Host Number: 13     State: running
                          scsi13 Channel 00 Id 0 Lun: 0
                          scsi13 Channel 00 Id 0 Lun: 1
                          scsi13 Channel 00 Id 0 Lun: 2
                [root@ovm-2 ~]# iscsiadm -m session -P 3
                iSCSI Transport Class version 2.0-870
                version 2.0-872
                Target: iqn.2011-09.com.example.sn1:target1
                   <<<snip>>>
                          ************************
                          Attached SCSI devices:
                          ************************
                          Host Number: 15     State: running
                          scsi15 Channel 00 Id 0 Lun: 0
                          scsi15 Channel 00 Id 0 Lun: 1
                          scsi15 Channel 00 Id 0 Lun: 2
                          scsi15 Channel 00 Id 0 Lun: 3
                          scsi15 Channel 00 Id 0 Lun: 4
                Target: iqn.2011-09.com.example.sn2:target1
                   <<<snip>>>
                          ************************
                          Attached SCSI devices:
                          ************************
                          Host Number: 14     State: running
                          scsi14 Channel 00 Id 0 Lun: 0
                          scsi14 Channel 00 Id 0 Lun: 1
                          scsi14 Channel 00 Id 0 Lun: 2
                <li>Yes, multipath is running on both Oracle VM Servers...</li>
                [root@ovm-1 ~]# multipath -ll
                1IET_00010001 dm-1 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                |-+- policy='round-robin 0' prio=1 status=active
                | `- 12:0:0:1 sdb 8:16 active ready running
                `-+- policy='round-robin 0' prio=1 status=enabled
                  `- 13:0:0:1 sdd 8:48 active ready running
                1IET_00010004 dm-2 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                `-+- policy='round-robin 0' prio=1 status=active
                  `- 12:0:0:4 sdg 8:96 active ready running
                1IET_00010003 dm-3 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                `-+- policy='round-robin 0' prio=1 status=active
                  `- 12:0:0:3 sdf 8:80 active ready running
                1IET_00010002 dm-0 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                |-+- policy='round-robin 0' prio=1 status=active
                | `- 12:0:0:2 sdc 8:32 active ready running
                `-+- policy='round-robin 0' prio=1 status=enabled
                  `- 13:0:0:2 sde 8:64 active ready running
                [root@ovm-2 ~]# multipath -ll
                1IET_00010001 dm-0 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                |-+- policy='round-robin 0' prio=1 status=active
                | `- 14:0:0:1 sdb 8:16 active ready running
                `-+- policy='round-robin 0' prio=1 status=enabled
                  `- 15:0:0:1 sdd 8:48 active ready running
                1IET_00010004 dm-1 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                `-+- policy='round-robin 0' prio=1 status=active
                  `- 15:0:0:4 sdg 8:96 active ready running
                1IET_00010003 dm-3 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                `-+- policy='round-robin 0' prio=1 status=active
                  `- 15:0:0:3 sdf 8:80 active ready running
                1IET_00010002 dm-2 IET,VIRTUAL-DISK
                size=25G features='0' hwhandler='0' wp=rw
                |-+- policy='round-robin 0' prio=1 status=active
                | `- 15:0:0:2 sde 8:64 active ready running
                `-+- policy='round-robin 0' prio=1 status=enabled
                  `- 14:0:0:2 sdc 8:32 active ready running
                <li>There's a *77-second difference* between the two Oracle VM Servers (i.e., the two iSCSI initiators):</li>
                [root@ovm-1 ~]# date
                Mon Oct  8 15:52:28 PDT 2012
                [root@ovm-2 ~]# date
                Mon Oct  8 15:53:45 PDT 2012
                <li>Forgetting+rediscovering may be an option, but I think that I'll reserve it for later. ;)</li>
                </ol>
                I think that I'll start by improving time-synchronization between the two Oracle VM Server nodes and then see about stoping/restarting the Oracle VM Manager node.
                • 5. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                  budachst
                  This look all good to me. I can't imagine, why OVMM wouldn't discover the targets as valid local storage. Besides that synchronizing your VM server's clocks to OVMM (which is what I did), I don't really expect this to help in this case. I'd re-discover both VM servers and see, if they LUNs show un as local storage in their respective Physical Disks perspective.
                  Maybe it would be a good idea to watch the ovm-agent.log on the VM servers, while re-discovering them.
                  • 6. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                    Jurajl-Oracle
                    The time syncro is important - suggestion to syncronise it towards OVMM server is right - BTW this should be the default behaviour of OVM - to syncronize dom0 servers against OVMM server (check ntpq -p; default ntpd settings on your dom0 servers and make sure OVMM server has a good time ( or time source), aswell).
                    BTW - the whole virtualised infrastructure should use ntp (even domU servers). See for inspiration "http://blogs.oracle.com/virtualization/entry/configuring_ntp_services_for_oracle"
                    Taking a look into /var/log/ovs-agent.log on dom0 servers is another good suggestion.
                    You should not need to unregister and register servers, start just by rediscovering them via OVMM gui (you can rediscover dom0, refresh Storage, refresh Repositories with no problem - even in infrastructure with running domU servers).
                    :j
                    • 7. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                      960874
                      epretorious wrote:
                      <ol start=4>
                      <li>Forgetting+rediscovering may be an option, but I think that I'll reserve it for later. ;)</li>
                      </ol>
                      I think that I'll start by improving time-synchronization between the two Oracle VM Server nodes and then see about stoping/restarting the Oracle VM Manager node.
                      After shutting down the Oracle VM Manager node and the two Oracle VM Server nodes for a day and restarting all three nodes, the two VM's that I'd created would not start. So I took the opportunity to delete both storage servers (iqn.2011-09.com.example.sn1:target1 & iqn.2011-09.com.example.sn2:target1) and start all over.

                      +In the examples below, target of iqn.2011-09.com.example.sn1 has been renamed target2 as part of the troubleshooting process. This had no effect on OVM's behavior so I didn't bother changing it back.+

                      What I found was quite interesting: Oracle VM Manager will not allow two LUN's with the same SCSI ID & LUN number! e.g., OVM will not use LUN's 1 & 2 of iqn.2011-09.com.example.sn1:target2 if iqn.2011-09.com.example.sn2:target1also has two LUN's numbered 1 & 2: e.g., this scenario does NOT work...
                      [root@ovm-2 ~]# sudo iscsiadm -m session -P 3
                      iSCSI Transport Class version 2.0-870
                      version 2.0-872
                      Target: iqn.2011-09.com.example.sn1:target2
                           Current Portal: 192.168.1.1:3260,1
                           Persistent Portal: 192.168.1.1:3260,1
                           <<<snip>>>
                                ************************
                                Attached SCSI devices:
                                ************************
                                Host Number: 15     State: running
                                scsi15 Channel 00 Id 0 Lun: 0
                                scsi15 Channel 00 Id 0 Lun: 1
                                scsi15 Channel 00 Id 0 Lun: 2
                                scsi15 Channel 00 Id 0 Lun: 3
                                scsi15 Channel 00 Id 0 Lun: 4
                      Target: iqn.2011-09.com.example.sn2:target1
                           Current Portal: 192.168.1.2:3260,1
                           Persistent Portal: 192.168.1.2:3260,1
                           <<<snip>>>
                                ************************
                                Attached SCSI devices:
                                ************************
                                Host Number: 10     State: running
                                scsi10 Channel 00 Id 0 Lun: 0
                                scsi10 Channel 00 Id 0 Lun: 1
                                scsi10 Channel 00 Id 0 Lun: 2
                      However, if I manually specify the SCSI ID or LUN ID of the four target LUN's of iqn.2011-09.com.example.sn1:target2, OVM will make all four available for use: e.g., this scenario works as expected...
                      [root@ovm-1 ~]# sudo iscsiadm -m session -P 3
                      iSCSI Transport Class version 2.0-870
                      version 2.0-872
                      Target: iqn.2011-09.com.example.sn1:target2
                           Current Portal: 192.168.1.1:3260,1
                           Persistent Portal: 192.168.1.1:3260,1
                           <<<snip>>>
                                ************************
                                Attached SCSI devices:
                                ************************
                                Host Number: 17     State: running
                                scsi17 Channel 00 Id 0 Lun: 0
                                scsi17 Channel 00 Id 0 Lun: 5
                                scsi17 Channel 00 Id 0 Lun: 6
                                scsi17 Channel 00 Id 0 Lun: 7
                                scsi17 Channel 00 Id 0 Lun: 8
                      Target: iqn.2011-09.com.example.sn2:target1
                           Current Portal: 192.168.1.2:3260,1
                           Persistent Portal: 192.168.1.2:3260,1
                           <<<snip>>>
                                ************************
                                Attached SCSI devices:
                                ************************
                                Host Number: 10     State: running
                                scsi10 Channel 00 Id 0 Lun: 0
                                scsi10 Channel 00 Id 0 Lun: 1
                                scsi10 Channel 00 Id 0 Lun: 2
                      • 8. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                        Looney128
                        I had the same problems creating iSCSI connections to my SANs. I use the HP P4300 series and the way it manages the groups and clusters made it impossible to use with OVM 3.1.1.

                        But good news. The new Beta 3.2.1 (build 358) seems to have fixed this problem.

                        Edited by: Looney128 on Nov 13, 2012 8:31 AM
                        • 9. Re: Oracle VM Manager 3.1.1: Discovering SAN Servers
                          960874
                          Looney128 wrote:
                          I had the same problems creating iSCSI connections to my SANs. I use the HP P4300 series and the way it manages the groups and clusters made it impossible to use with OVM 3.1.1.

                          But good news. The new Beta 3.2.1 (build 358) seems to have fixed this problem.
                          Thanks, Looney!

                          If OVS 3.2.1 is able to meet Oracle VM vs SuSE Linux Enterprise Server (SLES) I'll give OVMM 3.2.1 a try - otherwise it's just too damned much work to reprovision the servers that we're currently using!