7 Replies Latest reply: May 12, 2014 6:25 PM by Poorfrodo-Oracle RSS

    How to enable OVM 3.2.X to see more then 256 LUNs?

    Roddy Rodstein

      Greetings,

       

      We have twelve 3.2.8 Oracle VM Servers Pools, each with roughly 350 LUNs (will go up to over 500), i.e. phy ASM disks. With Oracle VM's 3.2 storage maximum of LUNs at 1000 (see link below), even after implementing the suggested modprobe.conf and grub.conf settings (see below) the dom0s can only see 256 LUNs. We are testing both dm-multipath and Powerpath, both only see 265 LUNs.

       

      Could you please share your hack to allow Oracle VM 3.2.x see over 256 LUNs?

       

      Thank you in advance for your support!

       

      References:

      Bug 18359452 : LUNS WITH A LUN-ID GREATER THE 255 ARE NOT VISIBLE ON OUR OVM SERVER 3.2.6

      http://docs.oracle.com/cd/E35328_01/E35329/html/vmrns-limits.html - Table 8 Storage maximums

       

      Recommended solution that did not work with dm-multipath or Powerpath:

      1.  add to modprobe.conf

      options lpfc lpfc_max_luns=1024

      No Spaces, other wise it will fail

       

      2. Add to the kernel line in grub.conf

      In UEK kernels, "scsi_mod" is compiled into the kernel instead of having a separate module.

      "scsi_mod.max_luns=1024" and "scsi_mod.max_report_luns=1023" parameters should be given to the kernel in "/boot/grub/grub.conf":

       

      Respectfully,

      Roddy

        • 1. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
          user12273962

          Do you need all those LUNS visible to every one of the 8 VM Server? If not, then you could do some creative zoning and also restrict LUN visibility by WWNs. At least you can in most storage I've used. Haven't touched a EMC product in some time.

           

          I know this doesn't address your exact problem but maybe it could be a work around.

          • 2. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
            Roddy Rodstein

            We do need the LUNs visible to each dom0, i.e. more than 256. Apparently this is a well known limitation with Linux that the Linux vendors offer solutions for. Not quite there for OVM ;-)

             

            Thank you for your support!

            Roddy

            • 3. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
              Poorfrodo-Oracle

              Hi Roddy,

               

              It looks like you are close, but Bug 18359452 shows things in the wrong place sometimes as well as things you might not need.  I'm not sure you really need the lpfc option (could be wrong), use the recipe below without the lpfc option in modprobe.conf; however if the lpfc module is actually being loaded then perhaps you should leave it in - up to you...)

               

              Please forgive the use of simple language as I know you are probably better at Linux than I am - I'm writing for you as well as other people looking for answers...

               

              Keep in mind the following rules:

               

              • Number of paths affect number of accessible LUNs
                • You will be able to access 1024 LUNs if you have 2 paths per LUN
                • You will be able to access 512 LUNs if you have 4 paths per LUN
                • You will be able to access 256 LUNs if you have 8 paths per LUN
                • You will be able to access 128 LUNs if you have 16 paths per LUN
              • You must increase Dom0 memory to accommodate additional LUNs

               

               

              Process Overview (at least this works for me on my 3.2.8 servers):

               

              • Modify /boot/grub/grub.conf
                • Increase RAM allocated to Dom0
                • Add scsi_mod.max_luns & scsi_mod.max_report_luns
              • Modify /etc/modprob.conf (only if you are using iSCSI)
                • Add iscsi_tcp max_lun option
              • Reboot server
              • Validate changes

               

              Formula for calculating required Dom0 memory:

               

              • Assuming 32G of physical memory (Dom0_mem=987M):
              • Assuming you are adding 512 additional LUNs (for a total of 1024)

               

              ((3.5M * 512) + 987M)=2779M

               

               

              Process Details:

               

              Step 1: Modify /boot/grub/grub.conf (changes highlighted in red - note that "_luns=" is plural):

               

              default=0

              timeout=5

              title Oracle VM Server-ovs (xen-4.1.3 2.6.39-300.32.6.el5uek)

                      root (hd0,0)

                      kernel /xen.gz dom0_mem=2779M allowsuperpage

                      module /vmlinuz-2.6.39-300.32.6.el5uek ro root=UUID=246e7305-0deb-4740-84f6-2ff4784dd5d3 scsi_mod.max_luns=1024 scsi_mod.max_report_luns=1023

                      module /initrd-2.6.39-300.32.6.el5uek.img

               

              Step 2: Modify /etc/modprob.conf (only if you are using iSCSI - doesn't look like you are using iSCSI - note that "_lun=" is singular):

               

              Append options iscsi_tcp max_lun=1024 to /etc/modprobe.conf (use vi or just echo "options iscsi_tcp max_lun=1024" >>/etc/modprobe.conf)


              Step 3: Reboot the server


              Step 4: Verify the changes:


              cat /sys/module/iscsi_tcp/parameters/max_lun (only if you are using iSCSI)

              cat /sys/module/scsi_mod/parameters/max_luns

              cat /sys/module/scsi_mod/parameters/max_report_lun

               

              Let me know if the above four steps work and I will write a KM note

               

              -Greg

              • 4. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
                budachst

                The lpfc options come into play, if you are using an Emulex HBA, e.g. Emulex LightPeek, hence lpfc (light peek fibre channel).

                • 5. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
                  Poorfrodo-Oracle

                  Thanks... I did not know they had changed the HBA model name from "Light Pulse" to Light Peek

                  • 6. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
                    budachst

                    Ahh, sorry - of course it's still Light Pulse… damn…

                    • 7. Re: How to enable OVM 3.2.X to see more then 256 LUNs?
                      Poorfrodo-Oracle

                      ...now you sound like me :-)