Deploy and Manage Oracle Linux KVM and Oracle Linux Virtualization Manager in Oracle Cloud Infrastructure

Version 3

Created by Simon Coter-Oracle on Apr 7, 2020 10:13 AM. Last modified by Simon Coter-Oracle on Apr 24, 2020 11:59 PM.

This document is for test and educational purposes only.

This document is still under review; sections of this document could change and further enhancements and/or options can be introduced on the same.

Introduction

The goal of this document it to offer a solution to deploy Oracle Linux KVM (OL-KVM) and Oracle Linux Virtualization Manager (OLVM) 4.3 release on Oracle Cloud Infrastructure.

Assumptions

OL-KVM hosts can run only on BM.DenseIO2.52 and BM.Standard2.52 shapes (BM.Standard.E2.64 not tested actually)
OLVM host can run on VM.Standard2.2 shape or higher
OL-KVM Virtual Machines can only rely on the 2nd physical NIC of the Bare-Metal server (for a total of 26 vNICs dedicated to OL-KVM Virtual Machines running)
Live-Migration is not available on this configuration (OL-KVM / OLVM running on OCI)
The OLVM Datacenter has to be configured for "Shared Storage"

Networking configuration: VCN and Subnet(s)

The setup requires proper VCN / Subnet(s) configuration; the architecture requires:

one unique VCN, where to setup the entire networking configuration for OLVM / OL-KVM and Virtual machines will run on top.
Internet access for OLVM and OL-KVM(s) instances
On this example, within the OCI VCN (10.0.0.0/16) the subnet(s) configured are:

Name	CIDR Block	Subnet Access	Usage
OCI	10.0.0.0/24	Private (Regional)	Dedicated to OCI (Storage and Services)
OLVM	10.0.1.0/24	Public (Regional)	Dedicated to OLVM Service Access, OLVM/KVM Server SSH Access
KVM-VM	10.0.2.0/24	Public (Regional)	Dedicated to KVM Virtual Machine vNICs

Architecture

Oracle Linux Virtualization Manager deployment requirements

Oracle Linux Virtualization Manager, built on OL7 image (latest available), has following requirements:

Built and created from latest OL7.7 (or higher but lower than OL8) image
vNIC (1): associated to "OLVM" Public subnet => Public IP address enabled
vNIC (2): associated to "OCI" Private subnet => No Public IP address

Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

vNIC(1): olvm (Public IP address available)
vNIC(2): oci-olvm

Note: for vNIC(1), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

Oracle Linux KVM Server deployment requirements

Oracle Linux KVM Server, built on OL7 image (latest available), has following requirements:

Built and created from latest OL7.7 (or higher but lower than OL8) image
vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IP Address
vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

Note: use proper hostnames for both Virtual-NICs because this kind of setup will help on the next steps of the configuration; example:

vNIC(1): olkvm01
vNIC(2): vdsm01 (Public IP address available)

Note: for vNIC(2), dedicated to vdsm/engine communication, select the "Skip Source/Destination Check" checkbox while creating the vNIC on OCI

Oracle Linux Virtualization Manager installation

By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM.

So, the first step requires to disable the non-required Yum-channels on the OLVM instance by executing the following command:

yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

Yum channels for Oracle Linux Virtualization Manager (ovirt4.3 and ovirt4.3-extra) yum channels are not synced to OCI.

Due to this missing sync, before proceeding with the installation following steps are required:

yum install http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-ovirt-release-el7-1.0-1.el7.x86_64.rpm -y
sed -i 's/yum$ociregion/yum/' /etc/yum.repos.d/oracle-ovirt-ol7.repo

Take "SELinux" to "permissive" level:

sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
setenforce 0

Update the system to the latest set-of-packages available:

yum update -y

Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

[root@ol7-olvm network-scripts]# cat ifcfg-ens5
DEVICE=ens5
ONBOOT=yes
IPADDR=10.0.1.2
NETMASK=255.255.255.0
BOOTPROTO=none
HWADDR=00:00:17:01:AB:EC
MTU=1500
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
DNS1=169.254.169.254

Enable the IP address for the second device, dedicated to VDSM; example:

# ifdown ens5
# ifup ens5

Install "Oracle Linux Virtualization Manager" by executing following command:

yum install ovirt-engine -y

Generate "ssh-keys" for your OLVM Instance; this step is required to then get access to the KVM-Server on the first boot.

NB: the KVM Server instance will boot on a "Private Subnet" (OCI) and this one will be only accessible from IPs/devices running on the same VCN.

[opc@ca-ovsx51 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/opc/.ssh/id_rsa):
Created directory '/home/opc/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/opc/.ssh/id_rsa.
Your public key has been saved in /home/opc/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Xg17qxVZczew9OMrlFskXIZv74IsNT8SAM8UMPEd0Q8 opc@ca-ovsx51.us.oracle.com
The key's randomart image is:
+---[RSA 2048]----+
|        +o..o=.o |
|        .o..ooE  |
|         =o .++Bo|
|          ++ o===|
|        S o.=o.o.|
|       . . .=oo o|
|        .  ooB o |
|          .o+ = .|
|          .. . o |
+----[SHA256]-----+

Save the content of your public-key, required while deploying the OL-KVM Bare-metal instance.

[opc@ca-ovsx51 ~]$ cat .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCscZTV40F5st6G/snruzWVuoYmdqxzDhNvtJI3TtNTuHD2UbNIzSGREVH2fjZv7ANwAmu8Vx7en+rVEUjVByUhQm1vHq9cGbDoia541Cau0qtUp2ZaN54oVQpl4Utz4JLRAsJel03PC0DHyKUmM/uj0CwM2a0Kz9OnCuAmcZ2ttCegUC9FpSj3WvmtNB3Ca/1kyNFyUmil9J7r3Rc/nbXtydoPVJrd/zec5gwIIn/cDxFOvaoIYGCD0Yshb0Fih8b9VOKWfTQaEuzuwx9BBzu7NIQ7jtqeYlDwpdVqIFvmiA5XtaMuIfm2+BFbSN4ZFoDeIqFoBfbDKQBTtv5Fb0K5 opc@ca-ovsx51

Proceed to the "Oracle Linux Virtualization Manager" configuration by following the official Oracle Documentation, available at:

https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/getstart/manager-install-prepare.html#mana…

Oracle Linux Virtualization Manager OCI customization

On OCI the "ovirt-engine" service listens on the private-IP address while the service is accessed by the OCI Public IP address.

To properly get the OLVM web interface available following configuration is required:

Edit the file "/etc/ovirt-engine/engine.conf.d/11-setup-sso.conf" and change the "SSO_ALTERNATE_ENGINE_FQDNS" with proper FQDN of your OLVM (you can get it from OCI console); example:

[root@ol7-olvm ~]# cat /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf
ENGINE_SSO_CLIENT_ID="ovirt-engine-core"
ENGINE_SSO_CLIENT_SECRET="SHy8iaClAv0avdJveqPwroVaxE51Bast"
ENGINE_SSO_AUTH_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"
ENGINE_SSO_SERVICE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/sso"
ENGINE_SSO_SERVICE_SSL_VERIFY_HOST=false
ENGINE_SSO_SERVICE_SSL_VERIFY_CHAIN=true
SSO_ALTERNATE_ENGINE_FQDNS="ol7-olvm.olvregional.olvvcn.oraclevcn.com"
SSO_ENGINE_URL="https://${ENGINE_FQDN}:443/ovirt-engine/"

NB: Consider that this FQDN will have to be resolved by your client(s) accessing the web-interface (by proper DNS or hosts file).

Oracle Linux KVM Server installation

Note: Respect following vNIC configuration for your OL-KVM Bare-Metal Instance:

vNIC (1): created on the first physical NIC and associated to "OCI" subnet => No Public IP Address
vNIC (2): created on the second physical NIC and associated to "OLVM" subnet => Public IP address enabled

Get access to the OL-KVM Bare-Metal Instance by OLVM Instance:

client > ssh opc@<olvm> ==> olvm > ssh opc@<kvm>

By default, the OL7 image, has yum-channels not required that can also create RPM dependency issues for OLVM.

So, the first step requires to disable the non-required Yum-channels on the OLVM instance by executing the following command:

yum-config-manager --disable ol7_developer ol7_developer_EPEL ol7_ksplice ol7_software_collections

Yum channels for Oracle Linux Virtualization Manager (ovirt4.3 and ovirt4.3-extra) yum channels are not synced to OCI.

Due to this missing sync, before proceeding with the installation following steps are required:

yum install http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-ovirt-release-el7-1.0-1.el7.x86_64.rpm -y
sed -i 's/yum$ociregion/yum/' /etc/yum.repos.d/oracle-ovirt-ol7.repo

Take "SELinux" to "permissive" level:

sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
setenforce 0

Update the system to the latest set-of-packages available:

yum update -y

Configure the 2nd vNIC of your system (dedicated to vdsm communication); example:

[root@ol7-olvm network-scripts]# cat ifcfg-eno3d1
DEVICE=eno3d1
ONBOOT=yes
IPADDR=10.0.1.3
NETMASK=255.255.255.0
BOOTPROTO=none
HWADDR=00:10:e0:ec:e4:69
MTU=1500
DEFROUTE=no
NM_CONTROLLED=no
IPV6INIT=no
DNS1=169.254.169.254

Enable the IP address for the second device, dedicated to VDSM; example:

# ifdown eno3d1
# ifup eno3d1

Due to compatibility issues between "Ksplice" and the OVA import process managed by OLVM, uninstall Ksplice:

yum remove ksplice* -y
rm -f /sbin/modprobe
mv /sbin/modprobe.ksplice-orig /sbin/modprobe

Proceed to the OL-KVM host configuration by following the official Oracle Documentation, available at:

https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/getstart/manager-install-kvm.html#manager-…

Note: the discover will have to happen on the dedicated "VDSM" subnet (2nd physical NIC of the Bare-Metal Instance)

Oracle Linux KVM Server OCI customization

This chapter is dedicated to the required customization to get an OCI OL-KVM Bare-Metal Instance manageable by Oracle Linux Virtualization Manager.

The required customization is related to the NIC(s) and Virtual-Function(s) management; on OCI, for each reboot of the BM Instance, the Virtual-Functions change their HW-ADDR (or Mac addressed).

Due to this important change, we need to instruct OLVM, with proper updates.

Following steps will show how-to create one new Linux service, dedicated to this target:

Open "/etc/default/grub" and add the following line to the end of the "GRUB_CMDLINE_LINUX" entry:

intel_iommu=on

The file will look as follows:

GRUB_CMDLINE_LINUX="crashkernel=auto LANG=en_US.UTF-8 console=tty0 console=ttyS0,9600 rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ip=dhcp netroot=iscsi:169.254.0.2::::iqn.2015-02.oracle.boot:uefi iscsi_param=node.session.timeo.replacement_timeout=6000 intel_iommu=on"

Enable "tuned" and set the performance optimization for "virtual-host":

systemctl enable tuned
systemctl start tuned
tuned-adm profile virtual-host

Commit the changes so that this configuration will always be used at boot time:

cp /boot/efi/EFI/redhat/grub.cfg /boot/efi/EFI/redhat/grub.cfg.orig
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Get OLVM aware of NIC "Virtual Function(s)" now enabled on the OL-KVM Server.

Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host

Click on "Network Interfaces" tab and click on "Setup Host Networks" button.

Click on the "Edit" icon of the "2nd Physical NIC" to manage the SR-IOV NIC options.

On the "Edit Virtual Functions (SR-IOV) configuration of <NIC>" window, expand the "Number of VFs setting" option; then write "26", the number of Virtual Function(s) today supported on OCI BM Shapes.

Confirm with "OK" and then, again, click on "OK" button to proceed to the Virtual Functions discovery and enablement process.

Those OCI Virtual Functions will be then leveraged by KVM Virtual Machines for networking.

DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

[OPTIONAL] Reboot your "Oracle Linux KVM server" by leveraging OCI web management interface to boot with updated system.

Oracle Linux KVM Server: create and define Storage Domain(s)

One requirement to get your OLVM Datacenter up (and, so, enabled) is to configure, at least, one storage domain for your Datacenter.

To accomplish this step you can use local NVME storage (on BM.DenseIO2.52 shape) or OCI Block-Volumes (on BM.DenseIO2.52 and BM.Standard2.52 shapes).

For details on how-to get the storage correctly configured see the Oracle Documentation available at:

https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/admin/storage-tasks.html#local-storage-pre…

The example below shows the option to add an OCI Block Volume as an OLVM "Storage Domain"; the BV IP address is recognized automatically by OLVM.

KVM Virtual Machine creation

There are different ways to create one virtual-machine on OLVM / OL-KVM; the same can be created starting from an OVA or directly installed from an Operating-System ISO.

For further details on how-to import or create a Virtual Machine you can follow the steps available on Oracle Documentation at:

Creating a Virtual Machine: https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/getstart/template-vm-creation.html
Import an OVA to create a Virtual Machine: https://blogs.oracle.com/scoter/import-configure-oracle-linux-7-template-for-oracle-linux-kvm

Network Management for Oracle Linux KVM Virtual Machines

Note: following steps can be applied to each required Virtual Machine Virtual NIC creation

To proceed to correctly configure networking for your Oracle Linux KVM Virtual Machines, proceed with the following steps:

Create the Virtual NIC on OCI web interface respecting following parameters
- Name: <use the VM-name so it will be easier to recognize which vNIC is used by which Virtual Machine>
- Subnet: kvm-vm-01
- Physical NIC: NIC 1 (a must)
- Skip Source/Destination Check: enabled
- Private IP address: <your choice>
- Assign Public IP address: <optional>
- Hostname: <hostname that will be used within the Virtual Machine>
Values example:
- Name: vm05-vnic01
- Subnet: kvm-vm-01
- Physical NIC: NIC 1
- Skip Source/Destination Check: enabled
- Private IP address: 10.0.2.201
- Assign Public IP address: Yes
- Hostname: vm05

OCI vNIC information required in the following steps are:

OCI vNIC Private IP address
OCI vNIC Public IP address (if defined)
OCI vNIC Mac Address
OCI vNIC VLAN Tag

Get the same vNIC defined on OLVM / OL-KVM host.

Note: With OLVM running on OCI, each OLVM Network will correspond to an OCI vNIC that, used as passthrough device, will be dedicated to a single VM.

Standard bridging networking is not possible on OCI.

Open the OLVM web-interface, "Network" => "Networks" and click on "New" button

On the Network creation window, supply following details:

Name: <use same name used on OCI, so it will be easier to identify the correct association>
Enable VLAN Tagging checkbox and insert the proper VLAN Tag you've on OCI vNIC
Leave all the other parameters on their default values

Open the OLVM web-interface, "Network" => "vNIC Profiles", select the profile created for your "Network" (same name) and click on "Edit" button.

On the "Edit" Windows enable the "Passthrough" option as in the following example:

Associate the OLVM Network with proper OCI NIC Virtual Function.

The target, here, is to associate an "OLVM Logical Network" to an "OCI Virtual Function".

DO NOT CLICK AND/OR USE THE "Sync All Networks" BUTTON; USING THIS OPTION COULD COMPROMISE THE STATUS OF OL-KVM/OLVM DEPLOYMENT

Open the OLVM web-interface, "Compute" => "Hosts" and select your OL-KVM host.

Click on "Network Interfaces" tab and click on "Show Virtual Functions" button

Above you can see the list of "OCI Virtual Functions" available that can be used to associate the "OLVM Logical Network" created.

Click on the "Setup Host Networks" button to process the connection between the "OLVM Logical Network" and "OCI Virtual Function"; on the "Setup Host Networks" click on "Show Virtual Functions" checkbox

On the Picture above, you can see:

Interfaces (left): list of physical NICs (see SR-IOV logo) and Virtual Functions (see vFunc logo)
Assigned Logical Networks (center): Logical Networks created on OLVM associated to Physical NIC(s) or Virtual Functions
Unassigned Logical Networks (right): Logical Networks created and actually not associated to any NIC/Virtual Function

Drag your "Logical Network" to one of the empty and available "Virtual Functions"

Configure Virtual Machine vNIC with proper HW-ADDR, as supplied by OCI web interface.

Open the OLVM web-interface, "Compute" => "Virtual Machines" and click on your VM name to open its details.

On the "Virtual Machine Configuration" section, click on "Network Interfaces" tab and edit "vNIC Settings"

On the "Edit Network Interface" window, enable the "Custom MAC address" checkbox and insert the "HW Address" supplied by OCI for this "Virtual Function"

Start your OLVM/OL-KVM Virtual Machine

Open the OLVM web-interface, "Compute" => "Virtual Machines" and select the line that identifies your Virtual Machine and start the same.

Useful options that you could apply to your environment:

for the Virtual Machine configuration use the FQDN supplied by OCI
on the Virtual Machine field "Comment" add the (optional) OCI Public IP address as a reference