1 2 Previous Next 19 Replies Latest reply: Jul 30, 2013 8:27 AM by user12273962 RSS

    See an HBA interface in a Virtualized Oracle Linux 6.3

    1024945

      Hi all,

       

      I have a pool of 6 virtualized machines. Those machines are connected to a storage throug a fiber optic switch. I want to see from one virtualized machine a backup library which is also connected to the fiber optic switch.

       

      I would like to know how my virtual machine would be able to see that backup library. I guess I would have to configure something in OVM Manager.

       

      Best regards

        • 1. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
          WadhahDaouehi

          Hi,

          Do you mean you have a pool with 6 Oracle VM Server (each one on a physical machine).

          I have a pool of 6 virtualized machines

          The Virtual Machines cannot see the backup repository.

          how my virtual machine would be able to see that backup library

          Can you more explain?

           

          Note: Oracle VM Server is an hypervisor/virtualization server which installed on physical Machine. And Virtual Machine is a VM which is installed or deployed on an Oracle VM Server. and Oracle VM Server can support many VMs. Oracle VM Manager is to manage many Oracle VM Server and with OVM Manager you can configure the Network and storage connection for Oracle VM Server. and to create a backup or snapshot Oracle VM Manager do that.

           

           

          Thank's

          Best Regards

          • 2. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
            1024945

            Hi,

             

            I have three physical machines. Each physical machine has installed Oracle VM Server.

             

            One physical machine has Oracle VM Manager virtualized.

             

            See atached image. I want that the machine called OVM-SSTP-MANGR can see the backup library.

             

            https://www.dropbox.com/s/2vonqbil3itbeee/Pool.png

             

            I use netbackup enterprise server 7.5. I cannot see the robot nor drives. Backup library and servers are connected through fiber optic switch.

             

            Thanks

            • 3. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
              WadhahDaouehi

              Hi,

              To take a backup for Virtual Machines you want to use the NetBackup Enterprise when the  NetBackup Enterprise server is installed in Oracle VM Manager host and NetBackup Enterprise client is installed on your Virtual Machines.

               

              Note: The networking infrastructure in the Oracle VM environment comprises connections between Oracle VM Servers, between Oracle VM Servers and Oracle VM Manager, between the Oracle VM Servers and their storage sub-systems, as well as communications among virtual machines deployed in the environment, and between virtual machines and external private or public networks.

              In Oracle VM Manager, network configuration is the mapping of available network interfaces to a set of logical Ethernet networks. The physical network is the collection of physical connections in Oracle VM Manager and all Oracle VM Servers, and the switches and routers that allow information to reach its destination.

               

              • Oracle VM identifies different network functions: server management, live migrate, cluster heartbeat, virtual machine, and storage.

               

              So to Oracle VM Manager communicate with Virtual Machines on Oracle VM Sever so The network server management and virtual machine should have the same network. Then edit you network with Oracle VM Manager and add/edit the virtual Machine Network to the Management network.

               

              I hope this can solve you problem

              Best Regards.

              • 4. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                budachst

                I guess, you want pass a pci device right through to a virtual machine, right? This can be done, but I had no great success with it, when trying to access a Qlogic HBA directly from my VM. On the other hand, I had good success connecting directly to a Intel NIC from my VM using pci-passthrough.

                 

                However, there are (at least) two issues with that:

                 

                a) PCI-passthrough is not supported by Oracle

                b) the latest builds of Xen on Oracle VM, even bail out, when they find the pci-passthrough directive in the vm.cfg

                c) you will need to prepare the VM host to be include the xen-pcipassthrough kernel module, rendering OVM updates non-trivial, or better: non-automatic, since these changes won't stick and you'll have to redo them on each kernel-update to your OVM server(s).

                d) you will loose the ability to run the VM on any host on the OVM cluster, pinning the affected VMs to a specific OVM server

                 

                Hth,

                budy

                • 5. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                  1024945

                  you are right. I want to access an HP 82Q 8Gb Dual Port PCI-e FC HBA (AJ764A).

                   

                  a) It does not matter if it is not supported.

                  b)Where do I have to configure this vm.cfg, in the oracle vm manager host?

                  c)Can you tell me the steps and prerequisites

                  d) I have made an independent virtual machine, only to make the backup. OVM Manager will be in another machine. Will this loose these abilites?

                   

                  Best regards

                  • 6. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                    budachst

                    Actually, I have covered the procedure in this post: https://forums.oracle.com/message/10329576#10329576

                     

                    The only thing that has changed is, that as of OVM 3.2.1, it's not possible to have the pci-config inside the vm.cfg file anylonger. This seems not to be Oracle's fault, since Xen simply refuses the vm.cfg when the guest gets started with a Error: (22, 'Invalid argument') error. My workaround currently is, to assign the pci-device to the guest after it has started using Xen's pci-attach command on the comand line like this:

                     

                    xm pci-attach 0004fb0000060000f3bcd21af24884eb 0000:07:00.0

                    xm pci-attach 0004fb0000060000f3bcd21af24884eb 0000:07:00.1

                     

                    This of course also means, that you can't have this guest as a HA one…

                     

                    Cheers

                     

                    P.S. I just checked the Xen mailing lists and there seems not yet to be a solution for this issue, which seems to originate from some xen-utils (namely pciif.py)

                    • 7. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                      1024945

                      Dear budhachst.

                       

                      Thanks for your answer. I have now more clear the steps to follow, but as I am new in this things I still have some doubts.

                       

                      My main doubt is: Do I have to make any modification to the physical server where my OVM guests are placed? Please have a look to the link attached where you can see the pool of machines I have: https://www.dropbox.com/s/2vonqbil3itbeee/Pool.png

                       

                      Actually I have created a new virtual machine to place the netbackup server on it. I don't need to have it in HA, but I need to know if the commands showed need to be ran in the SRVDAR01 (physical machine where I have the Oracle Server) or in the guest machine I have created (OVMSSTP_MGR).

                       

                      Take into account also that the machine who has the OVM Manager (3.2.1 version) is also a virtual machine.

                       

                      Thanks

                      • 8. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                        budachst

                        You will have to follow the procedure on each host, where you need physical access to the tape libraries. So, if the tape library only needs to be accessible from the Netbackup Enterprise Server, you will have to modify only SRVDAR01. It also isn't relevant where OVMM itself is running, since you'll wanto to startup that particular VM manually using xm create vm.cfg running from the directory where OVMM has stored the vm.cfg file, which gives you more control on the startup process, since you will have to manually add the FC HBA to the guest anyway.

                         

                        So all, commands shown in my old post will have to be run in SRVDAR01.

                         

                        Hth,

                        budy

                        • 9. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                          1024945

                          Thank you so much.

                           

                          In a first step lspci command executed on the host shows me the following:

                           

                          04:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                          04:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                          07:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                          07:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                           

                          So, it seems I only have to find the "id" of the guest virtual machine where I want to see the library. In your case is 0004fb0000060000f3bcd21af24884eb.

                           

                          In my case, please confirm I have to do the following steps:


                          1. Prepare SRVDAR01 with the steps showed in the link: https://forums.oracle.com/message/10329576#10329576 so that I would be able to execute modprobe xen-pciback. That means the modification of the grub and a reboot. I have also read in someplace maybe I have to activate something in the BIOS.
                          2. Manually attach the pci-device to the guest after it has started using Xen's pci-attach command on the command line like this: xm pci-attach  "id of my guest" 0000:07:00.0

                           

                          Only one final doubt: is it possible then to modify vm.cfg file to automatically attach the pci device or as I am using OVM 3.2.1 is not possible?

                           

                          Best regards.

                          • 10. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                            budachst

                            Yes, that's basically it. You can get the ID of your guest shown by clicking on the edit button while selecting your guest, or by clicking on the litte triangle. Both ways will show you the guest ID. You'd then cd onto the resp. OVMS into /OVMS/<repoID>/VirtualMachines/<guest ID> and issue either the xm create command and the xm pci-attach command from there, although it's not necessary for the latter one.

                             

                            Your OVMS will have to have VT-d enabled in it's BIOS, but yet have to encounter a modern server, where this isn't enabled by default and none of my HPs had it disabled.

                            I also didn't mess around with grub, but modified the rc.local instead to hide the pci devices I intended to use with my guest. I circumvented the need to execute modprobe xen-pciback by creating a new initrd, with this module enabled like this:

                             

                            mkinitrd -v -f initrd-pciback-`uname -r`.img `uname -r` --with=xen-pciback

                             

                            and then modifying grub accordingly to load the new initrd, which loads xen-pciback automatically. Also bear in mind, that those seem to be Dual-Port HBAs, so you cannot split their usage across different guests, you can use none, one or both ports only  in the same guest. Trying to bind the 2nd port to another guest will cause your OVMS to reboot instantly.

                             

                            Due to the issue in Xen 4.1 and up, you won't be able to put the pci config into the vm.cfg itself, which is, what I previously did myself, but as of Xen 4.1, this yields the afore mentioned error.

                             

                            Cheers,

                            budy

                            • 11. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                              1024945

                              Dear Budy,

                               

                              Really appreciate your help. One more thing I have in mind. This dual port HBA is connected through a fiber optic switch, so that physical and virtual machines  can see a data storage, also connected with the fiber optic switch. Tape library is also connected to the fiber optic switch.

                               

                              If I make the pci-passthrough from SRVDAR01 to the specific virtual machine, where netbackup server is installed, will I put in risk the other virtual machines which run in SRVDAR01 so that the would not see the storage assigned to them?

                               

                              Best regards

                              • 12. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                                budachst

                                No - only the guest that has the HBA directly connected can access the FC fabric. That's the whole point with pci-passtrough anyway. It's designed to provide direct access to a pci device for a specific guest. Not even the host, that is the OVMS in this case, can make use of the HBA once it's been directly assigned to the NetBackup guest.

                                Remember, that you first hide the device from the pci bus on the host and then make it visible inside the guest.

                                 

                                This is, btw, also mandantory since having two instances controlling one pci resource at the same time will surely lead to immediate trouble…

                                 

                                I am really curious, if you will succeed with this project, as I said earlier, trying to use a QLogic HBA on my OVMS didn't work out well. I was able to see the LUNs, but other than that, I wasn't able to get stable transfer established between my guest and the fabric.

                                • 13. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                                  1024945

                                  Woow man, then I have to modify the way to proceed....

                                   

                                  As you have seen SRVDAR01 has two dual port HBA pci device in two different PCI slots. That makes the four fibre channel ports I showed in one of my answers....

                                   

                                  04:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                                  04:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                                  07:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                                  07:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

                                   

                                  Then I would have to:

                                   

                                  1. As I am using one fibre channel from one HBA and one fibre channel from another HBA, in order to have redundancy, I shall first Identify clearly the fibre channel ports which I am not using.
                                  2. Then connect the two free ports to the brocade through fiber optic cables.
                                  3. Make the WWN assignation in order to MAP the backup library to these free ports.
                                  4. Follow then the steps you told me in one of your answers:
                                    1. Prepare SRVDAR01 with the steps showed in the link: https://forums.oracle.com/message/10329576#10329576 so that I would be able to execute modprobe xen-pciback. That means the modification of the grub and a reboot.
                                    2. Manually attach the pci-device to the guest after it has started using Xen's pci-attach command on the command line like this: xm pci-attach  "id of my guest" 0000:07:00.0

                                  And after all that work, although the procedure is correctly executed, it might not work....


                                  Let's see if it worth to do all this job. Thanks again!!

                                  • 14. Re: See an HBA interface in a Virtualized Oracle Linux 6.3
                                    budachst

                                    Good luck. And maybe you want to report back with either success or failure as I am still interested in using my HBAs directly from my guests and maybe things have evolved.

                                    1 2 Previous Next