Deploy and Manage Oracle Linux KVM and Oracle Linux Virtualization Manager in Oracle Cloud Infrastructure

Version 3

    This document is for test and educational purposes only.

    This document is still under review; sections of this document could change and further enhancements and/or options can be introduced on the same.

    Introduction

    The goal of this document it to offer a solution to deploy Oracle Linux KVM (OL-KVM) and Oracle Linux Virtualization Manager (OLVM) 4.3 release on Oracle Cloud Infrastructure.

     

    Assumptions

    • OL-KVM hosts can run only on BM.DenseIO2.52 and BM.Standard2.52 shapes (BM.Standard.E2.64 not tested actually)
    • OLVM host can run on VM.Standard2.2 shape or higher
    • OL-KVM Virtual Machines can only rely on the 2nd physical NIC of the Bare-Metal server (for a total of 26 vNICs dedicated to OL-KVM Virtual Machines running)
    • Live-Migration is not available on this configuration (OL-KVM / OLVM running on OCI)
    • The OLVM Datacenter has to be configured for "Shared Storage"

     

    Networking configuration: VCN and Subnet(s)

    The setup requires proper VCN / Subnet(s) configuration; the architecture requires:

    • one unique VCN, where to setup the entire networking configuration for OLVM / OL-KVM and Virtual machines will run on top.
    • Internet access for OLVM and OL-KVM(s) instances
    • On this example, within the OCI VCN (10.0.0.0/16) the subnet(s) configured are:

     

    NameCIDR BlockSubnet AccessUsage
    OCI10.0.0.0/24Private (Regional)Dedicated to OCI (Storage and Services)
    OLVM10.0.1.0/24Public (Regional)Dedicated to OLVM Service Access, OLVM/KVM Server SSH Access
    KVM-VM10.0.2.0/24Public (Regional)Dedicated to KVM Virtual Machine vNICs

     

    Screen Shot 2020-04-08 at 11.23.00 PM.png

    Architecture

    olvm-kvm-oci.png

     

    Oracle Linux Virtualization Manager deployment requirements

    Oracle Linux Virtualization Manager, built on OL7 image (latest available), has following requirements:

    • Built and created from latest OL7.7 (or higher but lower than OL8) image
    • vNIC (1): associated to "OLVM" Public subnet => Public IP address enabled
    • vNIC (2): associated to "OCI" Private subnet => No Public IP address

    Screen Shot 2020-04-08 at 11.28.02 PM.png

    Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

    • vNIC(1): olvm (Public IP address available)
    • vNIC(2): oci-olvm
    Note: for vNIC(1), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

    Oracle Linux KVM Server deployment requirements

    Oracle Linux KVM Server, built on OL7 image (latest available), has following requirements:

    • Built and created from latest OL7.7 (or higher but lower than OL8) image
    • vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IP Address
    • vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

    kvm-vnic-01.png

    Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

    • vNIC(1): olkvm01
    • vNIC(2): vdsm01 (Public IP address available)

    Note: for vNIC(2), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

    Oracle Linux Virtualization Manager installation

    By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM.

    So, the first step requires to disable the non-required Yum-channels on the OLVM instance by executing the following command:

    yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

     

    Yum channels for Oracle Linux Virtualization Manager (ovirt4.3 and ovirt4.3-extra) yum channels are not synced to OCI.

    Due to this missing sync, before proceeding with the installation following steps are required:

    yum install http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-ovirt-release-el7-1.0-1.el7.x86_64.rpm -y
    sed -i 's/yum$ociregion/yum/' /etc/yum.repos.d/oracle-ovirt-ol7.repo
    

     

    Take "SELinux" to "permissive" level:

    sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
    setenforce 0
    

     

    Update the system to the latest set-of-packages available:

    yum update -y
    

     

    Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

    [root@ol7-olvm network-scripts]# cat ifcfg-ens5
    DEVICE=ens5
    ONBOOT=yes
    IPADDR=10.0.1.2
    NETMASK=255.255.255.0
    BOOTPROTO=none
    HWADDR=00:00:17:01:AB:EC
    MTU=1500
    DEFROUTE=no
    NM_CONTROLLED=no
    IPV6INIT=no
    DNS1=169.254.169.254
    

     

    Enable the IP address for the second device, dedicated to VDSM; example:

    # ifdown ens5
    # ifup ens5
    

     

    Install "Oracle Linux Virtualization Manager" by executing following command:

    yum install ovirt-engine -y
    

     

    Generate "ssh-keys" for your OLVM Instance; this step is required to then get access to the KVM-Server on the first boot.

    NB: the KVM Server instance will boot on a "Private Subnet" (OCI) and this one will be only accessible from IPs/devices running on the same VCN.

    [opc@ca-ovsx51 ~]$ ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/opc/.ssh/id_rsa):
    Created directory '/home/opc/.ssh'.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/opc/.ssh/id_rsa.
    Your public key has been saved in /home/opc/.ssh/id_rsa.pub.
    The key fingerprint is:
    SHA256:Xg17qxVZczew9OMrlFskXIZv74IsNT8SAM8UMPEd0Q8 opc@ca-ovsx51.us.oracle.com
    The key's randomart image is:
    +---[RSA 2048]----+
    |        +o..o=.o |
    |        .o..ooE  |
    |         =o .++Bo|
    |          ++ o===|
    |        S o.=o.o.|
    |       . . .=oo o|
    |        .  ooB o |
    |          .o+ = .|
    |          .. . o |
    +----[SHA256]-----+
    

     

    Save the content of your public-key, required while deploying the OL-KVM Bare-metal instance.

    [opc@ca-ovsx51 ~]$ cat .ssh/id_rsa.pub
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCscZTV40F5st6G/snruzWVuoYmdqxzDhNvtJI3TtNTuHD2UbNIzSGREVH2fjZv7ANwAmu8Vx7en+rVEUjVByUhQm1vHq9cGbDoia541Cau0qtUp2ZaN54oVQpl4Utz4JLRAsJel03PC0DHyKUmM/uj0CwM2a0Kz9OnCuAmcZ2ttCegUC9FpSj3WvmtNB3Ca/1kyNFyUmil9J7r3Rc/nbXtydoPVJrd/zec5gwIIn/cDxFOvaoIYGCD0Yshb0Fih8b9VOKWfTQaEuzuwx9BBzu7NIQ7jtqeYlDwpdVqIFvmiA5XtaMuIfm2+BFbSN4ZFoDeIqFoBfbDKQBTtv5Fb0K5 opc@ca-ovsx51
    

     

    Proceed to the "Oracle Linux Virtualization Manager" configuration by following the official Oracle Documentation, available at:

    https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/getstart/manager-install-prepare.html#mana…

     

    Oracle Linux Virtualization Manager OCI customization

    On OCI the "ovirt-engine" service listens on the private-IP address while the service is accessed by the OCI Public IP address.

    To properly get the OLVM web interface available following configuration is required:

     

    • Edit the file "/etc/ovirt-engine/engine.conf.d/11-setup-sso.conf" and change the "SSO_ALTERNATE_ENGINE_FQDNS" with proper FQDN of your OLVM (you can get it from OCI console); example:
    [root@ol7-olvm ~]# cat /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf
    ENGINE_SSO_CLIENT_ID="ovirt-engine-core"
    ENGINE_SSO_CLIENT_SECRET="SHy8iaClAv0avdJveqPwroVaxE51Bast"
    ENGINE_SSO_AUTH_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"
    ENGINE_SSO_SERVICE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"
    ENGINE_SSO_SERVICE_SSL_VERIFY_HOST=false
    ENGINE_SSO_SERVICE_SSL_VERIFY_CHAIN=true
    SSO_ALTERNATE_ENGINE_FQDNS="ol7-olvm.olvregional.olvvcn.oraclevcn.com"
    SSO_ENGINE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/"
    
    NB: Consider that this FQDN will have to be resolved by your client(s) accessing the web-interface (by proper DNS or hosts file).

    Oracle Linux KVM Server installation

    Note: Respect following vNIC configuration for your OL-KVM Bare-Metal Instance:

    • vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IP Address
    • vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

    Get access to the OL-KVM Bare-Metal Instance by OLVM Instance:

    client > ssh opc@<olvm> ==> olvm > ssh opc@<kvm>

     

    By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM.

    So, the first step requires to disable the non-required Yum-channels on the OLVM instance by executing the following command:

    yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

     

    Yum channels for Oracle Linux Virtualization Manager (ovirt4.3 and ovirt4.3-extra) yum channels are not synced to OCI.

    Due to this missing sync, before proceeding with the installation following steps are required:

    yum install http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-ovirt-release-el7-1.0-1.el7.x86_64.rpm -y
    sed -i 's/yum$ociregion/yum/' /etc/yum.repos.d/oracle-ovirt-ol7.repo
    

     

    Take "SELinux" to "permissive" level:

    sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
    setenforce 0
    

     

    Update the system to the latest set-of-packages available:

    yum update -y
    

     

    Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

    [root@ol7-olvm network-scripts]# cat ifcfg-eno3d1
    DEVICE=eno3d1
    ONBOOT=yes
    IPADDR=10.0.1.3
    NETMASK=255.255.255.0
    BOOTPROTO=none
    HWADDR=00:10:e0:ec:e4:69
    MTU=1500
    DEFROUTE=no
    NM_CONTROLLED=no
    IPV6INIT=no
    DNS1=169.254.169.254
    

     

    Enable the IP address for the second device, dedicated to VDSM; example:

    # ifdown eno3d1
    # ifup eno3d1
    

     

    Due to compatibility issues between "Ksplice" and the OVA import process managed by OLVM, uninstall Ksplice:

    yum remove ksplice* -y
    rm -f /sbin/modprobe
    mv /sbin/modprobe.ksplice-orig /sbin/modprobe
    

     

    Proceed to the OL-KVM host configuration by following the official Oracle Documentation, available at:

    https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/getstart/manager-install-kvm.html#manager-…

    Note: the discover will have to happen on the dedicated "VDSM" subnet (2nd physical NIC of the Bare-Metal Instance)

     

    Oracle Linux KVM Server OCI customization

    This chapter is dedicated to the required customization to get an OCI OL-KVM Bare-Metal Instance manageable by Oracle Linux Virtualization Manager.

    The required customization is related to the NIC(s) and Virtual-Function(s) management; on OCI, for each reboot of the BM Instance, the Virtual-Functions change their HW-ADDR (or Mac addressed).

    Due to this important change, we need to instruct OLVM, with proper updates.

    Following steps will show how-to create one new Linux service, dedicated to this target:

     

    • Open "/etc/default/grub" and add the following line to the end of the "GRUB_CMDLINE_LINUX" entry:
    intel_iommu=on
    

     

    The file will look as follows:

    GRUB_CMDLINE_LINUX="crashkernel=auto LANG=en_US.UTF-8 console=tty0 console=ttyS0,9600 rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ip=dhcp netroot=iscsi:169.254.0.2::::iqn.2015-02.oracle.boot:uefi iscsi_param=node.session.timeo.replacement_timeout=6000 intel_iommu=on"
    
    • Enable "tuned" and set the performance optimization for "virtual-host":
    systemctl enable tuned
    systemctl start tuned
    tuned-adm profile virtual-host
    

     

    • Commit the changes so that this configuration will always be used at boot time:
    cp /boot/efi/EFI/redhat/grub.cfg /boot/efi/EFI/redhat/grub.cfg.orig
    grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
    

     

    • Get OLVM aware of NIC "Virtual Function(s)" now enabled on the OL-KVM Server.

    Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host

    net-00.png

    Click on "Network Interfaces" tab and click on "Setup Host Networks" button.

    net-01.png

    Click on the "Edit" icon of the "2nd Physical NIC" to manage the SR-IOV NIC options.

    net-03.png

    On the "Edit Virtual Functions (SR-IOV) configuration of <NIC>" window, expand the "Number of VFs setting" option; then write "26", the number of Virtual Function(s) today supported on OCI BM Shapes.

    net-04.png

    Confirm with "OK" and then, again, click on "OK" button to proceed to the Virtual Functions discovery and enablement process.

    Those OCI Virtual Functions will be then leveraged by KVM Virtual Machines for networking.

    DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

    • [OPTIONAL] Reboot your "Oracle Linux KVM server" by leveraging OCI web management interface to boot with updated system.

     

    Oracle Linux KVM Server: create and define Storage Domain(s)

    One requirement to get your OLVM Datacenter up (and, so, enabled) is to configure, at least, one storage domain for your Datacenter.

    To accomplish this step you can use local NVME storage (on BM.DenseIO2.52 shape) or OCI Block-Volumes (on BM.DenseIO2.52 and BM.Standard2.52 shapes).

    For details on how-to get the storage correctly configured see the Oracle Documentation available at:

    https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/admin/storage-tasks.html#local-storage-pre…

     

    The example below shows the option to add an OCI Block Volume as an OLVM "Storage Domain"; the BV IP address is recognized automatically by OLVM.

    block-volume-01.png

     

    KVM Virtual Machine creation

    There are different ways to create one virtual-machine on OLVM / OL-KVM; the same can be created starting from an OVA or directly installed from an Operating-System ISO.

    For further details on how-to import or create a Virtual Machine you can follow the steps available on Oracle Documentation at:

     

    Network Management for Oracle Linux KVM Virtual Machines

    Note: following steps can be applied to each required Virtual Machine Virtual NIC creation

    To proceed to correctly configure networking for your Oracle Linux KVM Virtual Machines, proceed with the following steps:

    • Create the Virtual NIC on OCI web interface respecting following parameters
      • Name: <use the VM-name so it will be easier to recognize which vNIC is used by which Virtual Machine>
      • Subnet: kvm-vm-01
      • Physical NIC: NIC 1 (a must)
      • Skip Source/Destination Check: enabled
      • Private IP address: <your choice>
      • Assign Public IP address: <optional>
      • Hostname: <hostname that will be used within the Virtual Machine>
    • Values example:
      • Name: vm05-vnic01
      • Subnet: kvm-vm-01
      • Physical NIC: NIC 1
      • Skip Source/Destination Check: enabled
      • Private IP address: 10.0.2.201
      • Assign Public IP address: Yes
      • Hostname: vm05

     

    OCI vNIC information required in the following steps are:

    • OCI vNIC Private IP address
    • OCI vNIC Public IP address (if defined)
    • OCI vNIC Mac Address
    • OCI vNIC VLAN Tag

     

    • Get the same vNIC defined on OLVM / OL-KVM host.

    Note: With OLVM running on OCI, each OLVM Network will correspond to an OCI vNIC that, used as passthrough device, will be dedicated to a single VM.

    Standard bridging networking is not possible on OCI.

    Open the OLVM web-interface, "Network" => "Networks" and click on "New" button

    net-creation.png

     

    On the Network creation window, supply following details:

    • Name: <use same name used on OCI, so it will be easier to identify the correct association>
    • Enable VLAN Tagging checkbox and insert the proper VLAN Tag you've on OCI vNIC
    • Leave all the other parameters on their default values

     

    net-creation-02.png

     

    Open the OLVM web-interface, "Network" => "vNIC Profiles", select the profile created for your "Network" (same name) and click on "Edit" button.

    On the "Edit" Windows enable the "Passthrough" option as in the following example:

    net-creation-03.png

    • Associate the OLVM Network with proper OCI NIC Virtual Function.

    The target, here, is to associate an "OLVM Logical Network" to an "OCI Virtual Function".

    DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

    Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host.

    Click on "Network Interfaces" tab and click on "Show Virtual Functions" button

    show-vfunctions-01.png

    Above you can see the list of "OCI Virtual Functions" available that can be used to associate the "OLVM Logical Network" created.

    Click on the "Setup Host Networks" button to process the connection between the "OLVM Logical Network" and "OCI Virtual Function"; on the "Setup Host Networks" click on "Show Virtual Functions" checkbox

    net-05.png

    On the Picture above, you can see:

    • Interfaces (left): list of physical NICs (see SR-IOV logo) and Virtual Functions (see vFunc logo)
    • Assigned Logical Networks (center): Logical Networks created on OLVM associated to Physical NIC(s) or Virtual Functions
    • Unassigned Logical Networks (right): Logical Networks created and actually not associated to any NIC/Virtual Function

     

    Drag your "Logical Network" to one of the empty and available "Virtual Functions"

    passthrough-conf-02.png

     

    • Configure Virtual Machine vNIC with proper HW-ADDR, as supplied by OCI web interface.

    Open the OLVM web-interface, "Compute" => "Virtual Machines" and click on your VM name to open its details.

    vm-conf-01.png

    On the "Virtual Machine Configuration" section, click on "Network Interfaces" tab and edit "vNIC Settings"

    vm-conf-02.png

    On the "Edit Network Interface" window, enable the "Custom MAC address" checkbox and insert the "HW Address" supplied by OCI for this "Virtual Function"

    vm-conf-03.png

    Start your OLVM/OL-KVM Virtual Machine

    Open the OLVM web-interface, "Compute" => "Virtual Machines" and select the line that identifies your Virtual Machine and start the same.

    vm-start-01.png

    Useful options that you could apply to your environment:

    • for the Virtual Machine configuration use the FQDN supplied by OCI
    • on the Virtual Machine field "Comment" add the (optional) OCI Public IP address as a reference