Use Vagrant and VirtualBox to setup Oracle Container Services for use with Kubernetes

Version 4

    Introduction

     

    Recently we announced the Oracle Vagrant boxes GitHub repository. Now you have a more streamlined way to create virtual machines with Oracle software fully configured and ready to go inside of them. With Oracle VM VirtualBox, Oracle Linux and Vagrant, you can easily build a consistent workflow to create disposable Linux environment whose configuration and dependencies are isolated from your development machine on every platform (Windows, MacOS or Linux).

     

    The example below is to show how simple to use Vagrant to set up Oracle Container Services for use with Kubernetes.

     

    Install Required Software

     

    Before you get started, install the required software:

     

     

    Clone Oracle Vagrant GitHub Repository

    If you haven't cloned or downloaded the GitHub repository, perform git clone to prepare for your own Vagrant boxes.

     

    $ git clone https://github.com/oracle/vagrant-boxes

    Cloning into 'vagrant-boxes'...

    remote: Counting objects: 342, done.

    remote: Compressing objects: 100% (58/58), done.

    remote: Total 342 (delta 42), reused 71 (delta 31), pack-reused 249

    Receiving objects: 100% (342/342), 69.52 KiB | 827.00 KiB/s, done.

    Resolving deltas: 100% (170/170), done.

     

    Setup Master Node

     

    Go the specific sub-folder.

    $ cd vagrant-boxes/Kubernetes/

    The Vagrantfile will provision a Kubernetes cluster with one master and n worker nodes. This example shows 2 worker nodes.

     

    First we'll set up the master node.

    $ vagrant up master

    Bringing machine 'master' up with 'virtualbox' provider...

    ==> master: Importing base box 'ol7-latest'...

    ==> master: Matching MAC address for NAT networking...

    ==> master: Setting the name of the VM: Kubernetes_master_1521995861580_23054

    ==> master: Clearing any previously set network interfaces...

    ==> master: Preparing network interfaces based on configuration...

        master: Adapter 1: nat

        master: Adapter 2: hostonly

    ==> master: Forwarding ports...

        master: 8001 (guest) => 8001 (host) (adapter 1)

        master: 22 (guest) => 2222 (host) (adapter 1)

    ==> master: Running 'pre-boot' VM customizations...

    ==> master: Booting VM...

    ...
    ...

    ==> master: Running provisioner: shell...

        master: Running: C:/Software/Cygwin64/tmp/vagrant-shell20180325-8000-ch4j4k.sh

        master: Installing and configuring Docker Engine

        master: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version

        master: Resolving Dependencies

        master: --> Running transaction check

        master: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed

        ...

        ...

            master: Dependencies Resolved

        master:

        master: ================================================================================

        master:  Package                  Arch     Version                  Repository     Size

        master: ================================================================================

        master: Installing:

        master:  docker-engine            x86_64   17.12.0.ol-1.0.1.el7     ol7_preview    30 M

        master: Installing for dependencies:

        master:  audit-libs-python        x86_64   2.7.6-3.el7              ol7_latest     73 k

        master:  checkpolicy              x86_64   2.5-4.el7                ol7_latest    290 k

        master:  container-selinux        noarch   2:2.21-1.el7             ol7_addons     28 k

        master:  libcgroup                x86_64   0.41-13.el7              ol7_latest     64 k

        master:  libsemanage-python       x86_64   2.5-8.el7                ol7_latest    104 k

        master:  libtool-ltdl             x86_64   2.4.2-22.el7_3           ol7_latest     48 k

        master:  policycoreutils-python   x86_64   2.5-17.1.0.1.el7         ol7_latest    445 k

        master:  python-IPy               noarch   0.75-6.el7               ol7_latest     32 k

        master:  setools-libs             x86_64   3.3.8-1.1.el7            ol7_latest    611 k

        master:

        master: Transaction Summary

        master: ================================================================================

        master: Install  1 Package (+9 Dependent packages)

        master: Total download size: 32 M

        master: Installed size: 128 M

        master: Downloading packages:

        master: --------------------------------------------------------------------------------

        master: Total                                              7.3 MB/s |  32 MB  00:04

        ...

        master: Installed:

        master:   docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7

        master:

        master: Dependency Installed:

        ...

        master: Complete!

        master: Creating 'btrfs' file system on: /dev/sdb

        master: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

        master: Installing and configuring Kubernetes packages

        master: Resolving Dependencies

        ...

        master: Dependencies Resolved

        master:

        master: ================================================================================

        master:  Package                    Arch       Version             Repository      Size

        master: ================================================================================

        master: Installing:

        master:  kubeadm                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

        master: Installing for dependencies:

        master:  kubectl                    x86_64     1.9.1-2.0.2.el7     ol7_addons     8.9 M

        master:  kubelet                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

        master:  kubernetes-cni             x86_64     0.6.0-2.0.1.el7     ol7_addons     797 k

        master:  kubernetes-cni-plugins     x86_64     0.6.0-2.0.1.el7     ol7_addons     8.5 M

        master:  socat                      x86_64     1.7.3.2-2.el7       ol7_latest     289 k

        master:

        master: Transaction Summary

        master: ================================================================================

        master: Install  1 Package (+5 Dependent packages)

        master:

        master: Total download size: 52 M

        master: Installed size: 279 M

        master: Downloading packages:

        master: --------------------------------------------------------------------------------

        master: Total                                              5.0 MB/s |  52 MB  00:10

    ...

        master: Installed:

        master:   kubeadm.x86_64 0:1.9.1-2.0.2.el7

        master:

         ...

        master: Complete!

        master: net.bridge.bridge-nf-call-ip6tables = 1

        master: net.bridge.bridge-nf-call-iptables = 1

        master: Your Kubernetes VM is ready to use!

    ==> master: Configuring proxy for Docker...

    ==> master: Running provisioner: shell...

        master: Running: inline script

    ==> master: Configuring proxy for Docker...

    ==> master: Running provisioner: shell...

        master: Running: inline script

    ==> master: Configuring proxy for Docker...

    Login to the master node virtual machine and run as root to configure the master node. In this step, you will be asked to sign into the Oracle Container Registry.

    $ vagrant ssh master

     

    Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

     

    The Oracle Linux End-User License Agreement can be viewed here:

     

        * /usr/share/eula/eula.en_US

     

    For additional packages, updates, documentation and community help, see:

     

        * http://yum.oracle.com/

     

    [vagrant@master ~]$ su root

     

    [root@master vagrant]# /vagrant/scripts/kubeadm-setup-master.sh

    /vagrant/scripts/kubeadm-setup-master.sh: Login to container registry

    Username: honglin.su@oracle.com

    Password:

    Login Succeeded

    /vagrant/scripts/kubeadm-setup-master.sh: Setup Master node

    Starting to initialize master node ...

    Checking if env is ready ...

    Checking whether docker can pull busybox image ...

    Checking access to container-registry.oracle.com/kubernetes ...

    v1.9.1: Pulling from kubernetes/kube-proxy-amd64

    Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

    Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

    Checking whether docker can run container ...

    Checking iptables default rule ...

    Checking br_netfilter module ...

    Checking sysctl variables ...

    Enabling kubelet ...

    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

    Check successful, ready to run 'up' command ...

    Waiting for kubeadm to setup master cluster...

    Please wait ...

    \ - 75% completed

    Waiting for the control plane to become ready ...

    ...............

    100% completed

    clusterrole "flannel" created

    clusterrolebinding "flannel" created

    serviceaccount "flannel" created

    configmap "kube-flannel-cfg" created

    daemonset "kube-flannel-ds" created

     

    Installing kubernetes-dashboard ...

     

    ...

    /vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf for vagrant user

    /vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf into host directory

    /vagrant/scripts/kubeadm-setup-master.sh: Saving token for worker nodes

    /vagrant/scripts/kubeadm-setup-master.sh: Master node ready, run

            /vagrant/scripts/kubeadm-setup-worker.sh

    on the worker nodes

     

    [root@master vagrant]# exit

    Setup Worker Nodes

     

    Go back to your development environment and setup the first worker node (worker1).

    $ vagrant up worker1

    Bringing machine 'worker1' up with 'virtualbox' provider...

    ==> worker1: Importing base box 'ol7-latest'...

    ==> worker1: Matching MAC address for NAT networking...

    ==> worker1: Setting the name of the VM: Kubernetes_worker1_1521998480587_67371

    ==> worker1: Fixed port collision for 22 => 2222. Now on port 2200.

    ==> worker1: Clearing any previously set network interfaces...

    ==> worker1: Preparing network interfaces based on configuration...

        worker1: Adapter 1: nat

        worker1: Adapter 2: hostonly

    ==> worker1: Forwarding ports...

        worker1: 22 (guest) => 2200 (host) (adapter 1)

    ==> worker1: Running 'pre-boot' VM customizations...

    ==> worker1: Booting VM...

            ...

        worker1: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version

        worker1: Resolving Dependencies

        worker1: --> Running transaction check

        worker1: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed

        worker1: Dependencies Resolved

        ...

        worker1:

        worker1: ================================================================================

        worker1:  Package                  Arch     Version                  Repository     Size

        worker1: ================================================================================

        worker1: Installing:

        worker1:  docker-engine            x86_64   17.12.0.ol-1.0.1.el7     ol7_preview    30 M

        worker1: Installing for dependencies:

        worker1:  audit-libs-python        x86_64   2.7.6-3.el7              ol7_latest     73 k

        worker1:  checkpolicy              x86_64   2.5-4.el7                ol7_latest    290 k

        worker1:  container-selinux        noarch   2:2.21-1.el7             ol7_addons     28 k

        worker1:  libcgroup                x86_64   0.41-13.el7              ol7_latest     64 k

        worker1:  libsemanage-python       x86_64   2.5-8.el7                ol7_latest    104 k

        worker1:  libtool-ltdl             x86_64   2.4.2-22.el7_3           ol7_latest     48 k

        worker1:  policycoreutils-python   x86_64   2.5-17.1.0.1.el7         ol7_latest    445 k

        worker1:  python-IPy               noarch   0.75-6.el7               ol7_latest     32 k

        worker1:  setools-libs             x86_64   3.3.8-1.1.el7            ol7_latest    611 k

        worker1:

        worker1: Transaction Summary

        worker1: ================================================================================

        worker1: Install  1 Package (+9 Dependent packages)

        worker1: Total download size: 32 M

        worker1: Installed size: 128 M

        worker1: Downloading packages:

        worker1: --------------------------------------------------------------------------------

        worker1: Total                                              8.6 MB/s |  32 MB  00:03

        ...

        worker1: Installed:

        worker1:   docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7

        worker1:

        worker1: Dependency Installed:

           ...

        worker1: Complete!

        worker1: Creating 'btrfs' file system on: /dev/sdb

        worker1: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

        worker1: Installing and configuring Kubernetes packages

        worker1: Resolving Dependencies

        worker1: --> Running transaction check

        worker1: ---> Package kubeadm.x86_64 0:1.9.1-2.0.2.el7 will be installed

        worker1: Dependencies Resolved

        ...

        worker1:

        worker1: ================================================================================

        worker1:  Package                    Arch       Version             Repository      Size

        worker1: ================================================================================

        worker1: Installing:

        worker1:  kubeadm                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

        worker1: Installing for dependencies:

        worker1:  kubectl                    x86_64     1.9.1-2.0.2.el7     ol7_addons     8.9 M

        worker1:  kubelet                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

        worker1:  kubernetes-cni             x86_64     0.6.0-2.0.1.el7     ol7_addons     797 k

        worker1:  kubernetes-cni-plugins     x86_64     0.6.0-2.0.1.el7     ol7_addons     8.5 M

        worker1:  socat                      x86_64     1.7.3.2-2.el7       ol7_latest     289 k

        worker1:

        worker1: Transaction Summary

        worker1: ================================================================================

        worker1: Install  1 Package (+5 Dependent packages)

        worker1: Total download size: 52 M

        worker1: Installed size: 279 M

        worker1: Downloading packages:

        worker1: --------------------------------------------------------------------------------

        worker1: Total                                               17 MB/s |  52 MB  00:03

        ...

        worker1: Installed:

        worker1:   kubeadm.x86_64 0:1.9.1-2.0.2.el7

        worker1:

        worker1: Dependency Installed:

        worker1:   kubectl.x86_64 0:1.9.1-2.0.2.el7

        worker1:   kubelet.x86_64 0:1.9.1-2.0.2.el7

        worker1:   kubernetes-cni.x86_64 0:0.6.0-2.0.1.el7

        worker1:   kubernetes-cni-plugins.x86_64 0:0.6.0-2.0.1.el7

        worker1:   socat.x86_64 0:1.7.3.2-2.el7

        worker1: Complete!

        worker1: net.bridge.bridge-nf-call-ip6tables = 1

        worker1: net.bridge.bridge-nf-call-iptables = 1

        worker1: Your Kubernetes VM is ready to use!

    ==> worker1: Configuring proxy for Docker...

    Log into the first worker node and configure the worker node. Similarly, you will be asked to sign into the Oracle Container Registry.

    $ vagrant ssh worker1

     

    Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

     

    The Oracle Linux End-User License Agreement can be viewed here:

     

        * /usr/share/eula/eula.en_US

     

    For additional packages, updates, documentation and community help, see:

     

        * http://yum.oracle.com/

     

    [vagrant@worker1 ~]$ su root

    [root@worker1 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

    /vagrant/scripts/kubeadm-setup-worker.sh: Login to container registry

    Username: honglin.su@oracle.com

    Password:

    Login Succeeded

    /vagrant/scripts/kubeadm-setup-worker.sh: Setup Worker node

    Starting to initialize worker node ...

    Checking if env is ready ...

    Checking whether docker can pull busybox image ...

    Checking access to container-registry.oracle.com/kubernetes ...

    v1.9.1: Pulling from kubernetes/kube-proxy-amd64

    Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

    Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

    Checking whether docker can run container ...

    Checking iptables default rule ...

    Checking br_netfilter module ...

    Checking sysctl variables ...

    Enabling kubelet ...

    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

    Check successful, ready to run 'join' command ...

    [preflight] Running pre-flight checks.

    [validation] WARNING: kubeadm doesn't fully support multiple API Servers yet

    [discovery] Trying to connect to API Server "192.168.99.100:6443"

    [discovery] Trying to connect to API Server "192.168.99.100:6443"

    [discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

    [discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

    [discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

    [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

    [discovery] Successfully established connection with API Server "192.168.99.100:6443"

    [discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

    [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

    [discovery] Successfully established connection with API Server "192.168.99.100:6443"

     

    This node has joined the cluster:

    * Certificate signing request was sent to master and a response

      was received.

    * The Kubelet was informed of the new secure connection details.

     

     

    Run 'kubectl get nodes' on the master to see this node join the cluster.

    /vagrant/scripts/kubeadm-setup-worker.sh: Worker node ready

    Repeat the previous steps to set up the second worker node (worker2).

    $ vagrant up worker2

    Bringing machine 'worker2' up with 'virtualbox' provider...

    ...

    $ vagrant ssh worker2

    ...

    [vagrant@worker2 ~]$ su root

    [root@worker2 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

    ...

     

    Validate the Kubernetes Cluster Configuration

    Now the cluster is ready. You log into the master node to verify your cluster setup.

    $ vagrant ssh master

    [vagrant@master ~]$ kubectl cluster-info

    Kubernetes master is running at https://192.168.99.100:6443

    KubeDNS is running at https://192.168.99.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

     

    To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

     

    [vagrant@master ~]$ kubectl get nodes

    NAME                 STATUS    ROLES     AGE       VERSION

    master.vagrant.vm    Ready     master    49m       v1.9.1+2.0.2.el7

    worker1.vagrant.vm   Ready     <none>    24m       v1.9.1+2.0.2.el7

    worker2.vagrant.vm   Ready     <none>    6m        v1.9.1+2.0.2.el7

     

    [vagrant@master ~]$ kubectl get pods --namespace=kube-system

    NAME                                        READY     STATUS             RESTARTS   AGE

    etcd-master.vagrant.vm                      1/1       Running            0          54m

    kube-apiserver-master.vagrant.vm            1/1       Running            0          54m

    kube-controller-manager-master.vagrant.vm   1/1       Running            0          55m

    kube-dns-855949bbf-fdtmq                    3/3       Running            0          55m

    kube-flannel-ds-5t29z                       1/1       Running            0          13m

    kube-flannel-ds-78x8g                       1/1       Running            0          55m

    kube-flannel-ds-qnc9g                       1/1       Running            0          30m

    kube-proxy-2rxzw                            1/1       Running            0          30m

    kube-proxy-lpjxc                            1/1       Running            0          55m

    kube-proxy-zdvj6                            1/1       Running            0          13m

    kube-scheduler-master.vagrant.vm            1/1       Running            0          55m

    kubernetes-dashboard-7c966ddf6d-sjks4       0/1       ImagePullBackOff   0          55m

    and you can see the virtual machine running status from Oracle VM VirtualBox Manager user interface.

     

    VirtualBox-Vagrant-Box-K8S.png

     

    Customize Vagrantfile

     

    The Vagrantfile from the GitHub can be used as-is; there are a couple of parameters you can set to tailor the installation to your needs.

    • NB_WORKERS (default: 2): the number of worker nodes to provision.
    • USE_PREVIEW (default: true): when true, Vagrant provisioning script will use the Oracle Linux 7 Preview and Add-ons channels for both Docker Engine and Kubernetes (latest version is select by yum). Otherwhise it will only use the Add-ons channel.
    • MANAGE_FROM_HOST (default: false): when true, Vagrant will bind port 6443 from the master node to the host. This allows you to manage the cluster from the host itself using the generated admin.conf file (assuming kubectl is installed on the host).
    • BIND_PROXY (default: true): when true, Vagrant will bind the Kubernetes Proxy port from the master node to the host. Useful to access the Dashboard or any other application from outside the cluster. It is an easier alternative to ssh tunnel.
    • MEMORY (default: 2048): all VMs are provisioned with 2GB memory. This can be slightly reduced if memory is a concern.

     

    Additional Information

     

    For additional information about Oracle Linux, visit