HOL5303 - Secure Container Orchestration using Oracle Linux Cloud Native (Kubernetes/Katacontainer)

Version 2
Visibility: Open to anyone

    Abstract

    Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs. In this lab, students will use Vagrant to automatically deploy an Oracle Container Services for use with Kubernetes cluster on Oracle Linux 7 virtual machines using VirtualBox. Once the cluster is deployed, attendees will learn how to deploy secured containers by Kata Containers.

     

    Lab Objective

    This hands-on lab takes you through the planning and deployment of Oracle Container Services for use with Kubernetes with Docker, initially, and then leveraging Kata-Containers.

    During this lab, we will create an Oracle Linux KVM demo environment on a single laptop machine, using KVM with nested virtualization; nested virtualization is required by Kata-Containers.

     

    Minimal Configuration to run this lab

    This document can be used to run the lab at home or at your office on your own laptop/desktop/server machine.

     

    In this lab, we use Oracle Linux KVM on the host to create 3 virtual machines that will later be used by Oracle Linux as Kubernetes Master and Worker nodes (2), so that we can install all software components on a single physical machine.

    Since the hypervisor used on the host is KVM, the native operating system on the laptop/desktop/server machine has to be Linux based.

     

    The minimal configuration needed for your laptop/desktop/server is:

    • 16 GB of RAM memory
    • Modern Intel/AMD x86 CPU
    • ~ 80GB of disk space

    The KVM Virtual Machine requirements requirements are:

    • Virtual Machine dedicated to KVM Compute node
      • 2 vCPUs
      • 4GB RAM
      • 50GB of disk space
    • Virtual Machine dedicated to Oracle Linux Virtualization Manager node
      • 2 vCPUs
      • 6GB RAM
      • 30GB of disk space

    For the installation of the Oracle Linux Container Services for use with Kubernetes you can follow documentation available at: https://docs.oracle.com/cd/E52668_01/E88884/html/kubernetes_install_upgrade.html or leverage the Oracle Linux Vagrant Boxes, public available on GitHub at https://github.com/oracle/vagrant-boxes

     

    Global Architecture Picture

    arch_hol5303.png

    Important Notes

    Acronyms

    In the present document, we will use the following acronyms:

    • OL for “Oracle Linux"
    • OL KVM for "Oracle Linux Kernel-based Virtual Machine"
    • VM for “Virtual machine”
    • K8S for Kubernetes
    • Kata for Kata Containers

    Nested Virtualization

    In this lab, 2 Virtualization layers are used to emulate a compute/management architecture:

    • Oracle Linux with Container Services for use with Kubernetes Master node (1)
    • Oracle Linux with Container Services for use with Kubernetes Worker nodes (1 & 2)

     

    Lab Execution

    The first step is to start the Virtual Machines required to run the lab.

    Those Virtual Machines have been creating on KVM and can be managed by the local GUI utility called "virt-manager"; so, to proceed, open a terminal with "lab" user and execute:

     

    # virt-manager

     

    The "Virtual Machine Manager" windows will open:

    01.png

    As you can see more Virtual Machines are available; the three Virtual Machines required for HOL-5303 are:

     

    • k8s-master (Oracle Linux with Oracle Container Services for use with Kubernetes - master-node)
    • k8s-worker1 (Oracle Linux with Oracle Container Services for use with Kubernetes - worker-node 1 of 2)
    • k8s-worker2 (Oracle Linux with Oracle Container Services for use with Kubernetes - worker-node 2 of 2)

     

    Select each of those Virtual Machines and click on "Play" button to start them.

     

    Lab Checks

    Before starting to work on the K8s environment, proceed to check that everything is correctly configured.

     

    Master Node

    • Open a terminal and connect by "ssh" to the "Master Node" and verify that all the K8s pods are up&running

    # ssh root@192.168.99.100

    # kubectl get pods --namespace=kube-system

     

    01.png

    • On the same terminal verify that also the "Worker nodes" are in status "Ready"

    # kubectl get nodes

     

    02.png

    • Open a terminal and connect by "ssh" to the "Worker Nodes (1 and 2)" and verify that all the K8s containers are up & running

    # ssh root@192.168.99.101

    # docker ps

    03.png

    # ssh root@192.168.99.102

    # docker ps

    04.png

    Known issue

    None

    Lab Steps

    During this lab following operations will be executed:

    • [TO DO]

     

    Generate Service Account Token to get access to the K8s dashboard

    By this section we'll generate a service account with "cluster-admin" role that will have access to all K8s resources.

    • Connect by "ssh" to the K8s "Master Node" (192.168.99.100) and execute following commands

    # kubectl create serviceaccount cluster-admin-dashboard-sa

    # kubectl create clusterrolebinding cluster-admin-dashboard-sa --clusterrole=cluster-admin --serviceaccount=default:cluster-admin-dashboard-sa

     

    • Copy the token from generated secret into a dedicated file

     

    # kubectl get secret | grep cluster-admin-dashboard-sa

    # kubectl describe secret cluster-admin-dashboard-sa-token-xgdjk (value obtained by the previous step)

    05.png

    • Execute following command to save the token in a file

    # kubectl describe secret cluster-admin-dashboard-sa-token-xgdjk |grep "token:" |cut -d ":" -f2 |sed -e 's/ //g' > /home/vagrant/token

    06.png

    Connect to the K8s dashboard from the host laptop browser

    • To get access to the K8s dashboard from the Host we need to create an ssh service forwarding process for port 8001 (from the VM to the Host)

    # ssh -L 8001:localhost:8001 root@192.168.99.100

    07.png

    • Once in-place, we can then open a browser (Firefox) on the Host and connect to the K8s Dashboard

    http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

    • Get access to the K8s Dashboard by the token generated above

    09.png

    • K8s Dashboard Overview

    10.png

     

    Start Kubernetes pods on Worker nodes (leveraging Docker container)

    During this section we run an application by creating a Kubernetes Deployment object; by the K8S Dashboard we upload a YAML file containing the deployment description..

    This YAML file describes a Deployment that runs the "nginx:1.7.9" Docker image.

    Here the content of the YAML file we use:

     

    apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2

    kind: Deployment

    metadata:

      name: nginx-deployment

    spec:

      selector:

        matchLabels:

          app: nginx

      replicas: 2 # tells deployment to run 2 pods matching the template

      template:

        metadata:

          labels:

            app: nginx

        spec:

          containers:

          - name: nginx

            image: nginx:1.7.9

            ports:

            - containerPort: 80

     

    To properly start those 2 containers on the K8s workers proceed with following steps.

    • On K8s dashboard click on "+ CREATE" icon in the right-upper corner

    11.png

    • On the "Create from text input" window, copy the YAML content available above and click on "Upload" button

    12.png

    • Once deployed, the two pods can be showed by both the "K8s dashboard" as well as "CLI"

    13.png

    # kubectl get pods

    14.png

    Those deployed pods expose a web-service on port 80; following steps will show how-to get simple access to this web-service and verify the same is correctly working.

    • Get the IP address where the "nginx" service is exposed; click on "Workloads => Pods" and select one of the two "nginx-deployment-*" pods, by clicking on its name, running on "worker1"

    15.png

    • On details webpage we can see the "Docker container" IP address

    16.png

    [OPTIONAL] By a terminal, opened on one of the 3 Virtual Machines (master, worker1 or worker2) you can try to execute the following command to test the web-server:

    # wget http://[IP-ADDRESS-NGINX]

     

    Those pods are based on Docker containers, running on the two K8s worker nodes configured; to verity those "Docker containers" we can execute following commands

    • From the "K8s master" node we can verify the container kernel running (same of the worker node)

    # kubectl get pods

    # kubectl exec -it <nginx-pod-name> -- uname -a

    17.png

    Kernel running on the K8s workers is "4.14.35-1902.4.8.el7uek.x86_64" and, obviously, the same run on Docker containers, managed by K8s, on top.

     

    Scale existing Application and update nginx software release at the same time

    By K8s Dashboard is also possible to extend the service capability (number of pods) as well as update the software release running (nginx).

    To complete this section, proceed with following steps:

     

    • By K8s Dashboard, open "Workloads => Deployments" and click on "nginx-deployment"

    18.png

    • On the "Deployment Details" webpage, click on "Edit" button (right-upper corner)

    19.png

    • On the "Edit a Deployment" window, update following parameters
      • "replicas": 8 (two entries require update)
      • "image": "nginx:1.8"
      • "updatedReplicas": 8
      • "readyReplicas": 8
      • "availableReplicas": 8

    and then click on "Update" button.

    The new deployment will appear with pods as below:

    20.png

    Reset Kubernetes Installation to prepare to the new environment

     

    Reset Kubernetes Installation on master and worker nodes

    • Connect to the master and both the worker nodes by "ssh" and execute following command (confirm with "y")

    # kubeadm reset

      • master

    21.png

      • worker1

    22.png

      • worker2

    23.png

    Enable new Yum channels, update Kubernetes and install Kata Container

    In this section new Yum channels will be enabled:

    • [ol7_developer_olcne] - Developer Preview for Oracle Linux Cloud Native Environment
    • [ol7_kvm_utils] - Oracle Linux KVM Utilities

    Once enabled, following steps will be executed:

    • Kubernetes upgrade from 1.12 to 1.14 Developer release
    • Kata Container 1.5.x installation

     

    Following steps have to be executed on all the KVM Virtual Machines (master, worker1 and worker2)

    • Enable "ol7_developer_olcne" Yum channel (required to upgrade Kubernetes and install Kata Container)

    # yum install oracle-olcne-release-el7.x86_64 -y

    24.png

    • Enabled "ol7_kvm_utils" Yum channel (containing Kata Container dependencies)

    # yum-config-manager --enable ol7_kvm_utils

    25.png

    • Update the machine

    # yum update -y

    26.png

    • Install Kata Container

    # yum install kata-runtime -y

    27.png

    Configure new Kubernetes cluster with CRI (containerd plugin) and Kata Container

    In order to allow Kubelet to use containerd (using the CRI interface), configure the service to point to the containerdsocket; to proceed, execute following steps as "root" user:

     

    K8s Master Node

    • Connect to the Master node by "ssh"

    # ssh root@192.168.99.100

     

    • Configure Kubernetes to use "containerd" (instead of Docker) as Container Runtime

    # mkdir -p  /etc/systemd/system/kubelet.service.d/

    # cat << EOF |tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf

    [Service]

    Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

    EOF

     

    • Inform systemd about the new configuration on "kubelet" service

    # systemctl daemon-reload

    28.png

    • Restart "containerd" service and verify it is up and running

    # systemctl restart containerd

    # systemctl status containerd

    29.png

    • Install "crictl" utility and verify it's then available in the PATH; it provides a CLI for CRI-compatible container runtimes. This utility, currently in Beta and still under quick iterations, is hosted at the cri-tools repository.

    # VERSION="v1.15.0"

    # wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz

    # tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin

    # rm -f crictl-$VERSION-linux-amd64.tar.gz

    # which crictl

    30.png

    • Start K8s Cluster using "kubeadm"; please, take care of the output (see picture below) and save the content selected in a text file.

    # kubeadm init --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=swap

    31.png

    • To start using the "K8s cluster", we have to remove old config file and run the first text selected in the picture above as "root" user

    # rm -f $HOME/.kube/config

    # mkdir -p $HOME/.kube

    # cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

    # chown $(id -u):$(id -g) $HOME/.kube/config

     

    • Verify nodes and pods status on the new K8s master node

    # kubectl get nodes

    # kubectl get pods --namespace=kube-system

    33.png

    • Deploy a pod network service, required to get pods to communicate with each other; "flannel" is going to be used (as it was on the previous K8s cluster)

    # kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

    34.png

    • Check pods status and verify both "flannel" and "coredns" are running

    35.png

     

    K8s Worker 1 Node - Join cluster as Docker Container Runtime node

    • Install all required packages and update the system

    # yum install oracle-olcne-release-el7.x86_64 -y

    # yum-config-manager --enable ol7_kvm_utils

    # yum update -y

    # yum install kata-runtime -y

    # yum install iproute-tc

     

    • Setup updated Docker configuration (to be used with K8s)

    # cat > /etc/docker/daemon.json <<EOF

    {

      "exec-opts": ["native.cgroupdriver=systemd"],

      "log-driver": "json-file",

      "log-opts": {

        "max-size": "100m"

      },

      "storage-driver": "overlay2",

      "storage-opts": [

        "overlay2.override_kernel_check=true"

      ]

    }

    EOF

     

    • Restart "Docker" system service

    # systemctl daemon-reload

    # systemctl restart docker

    • Join "Worker1" node to the K8s cluster (you need to use the command saved on Master step above, with proper hash. (this is only an example)

    # kubeadm join 10.0.2.210:6443 --token yd6erz.akin2eqjyscughar --discovery-token-ca-cert-hash sha256:c032e1625181dcf87647fd361bfa2a7288d1e1b8e2f78973c8972ae6f84867de --ignore-preflight-errors=Swap

    36.png

    • From the "master" node, check K8s nodes now configured on your cluster ("master" and "worker1" should be there once completed); the option "wide" will also show the "container-runtime" used

    # kubectl get nodes -o wide

    38.png

    • For listing containers on "worker1" node (Docker), you now must use docker command:

    # docker ps

    42.png

    K8s Worker 2 Node - Join cluster with Containerd Container Runtime node

    • Install all required packages and update the system

    # yum install oracle-olcne-release-el7.x86_64 -y

    # yum-config-manager --enable ol7_kvm_utils

    # yum update -y

    # yum install kata-runtime -y

    # yum install iproute-tc

     

    • Setup "containerd" for the integration with "kubelet"

    # modprobe overlay

    # modprobe br_netfilter

    # cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF

    net.bridge.bridge-nf-call-iptables  = 1

    net.ipv4.ip_forward                 = 1

    net.bridge.bridge-nf-call-ip6tables = 1

    EOF

     

    # sysctl --system

     

    • Configure "kubelet" to start with "containerd" as Container Runtime

    # cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf

    [Service]

    Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-on=false"

    EOF

     

    # echo "KUBELET_EXTRA_ARGS= --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-on=false" > /etc/sysconfig/kubelet

    # echo "KUBELET_EXTRA_ARGS= --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-on=false" > /etc/default/kubelet

     

    • Get systemd aware of service changes and restart "containerd" and "kubelet" services

    # systemctl daemon-reload

    # systemctl restart containerd

    # systemctl restart kubelet

     

    • Setup updated Docker configuration (to be used with K8s)

    # cat > /etc/docker/daemon.json <<EOF

    {

      "exec-opts": ["native.cgroupdriver=systemd"],

      "log-driver": "json-file",

      "log-opts": {

        "max-size": "100m"

      },

      "storage-driver": "overlay2",

      "storage-opts": [

        "overlay2.override_kernel_check=true"

      ]

    }

    EOF

     

    # systemctl daemon-reload

    # systemctl restart docker

     

    • Join "Worker2" node to the K8s cluster (you need to use the command saved on Master step above, with proper hash. (this is only an example)

    # kubeadm join 10.0.2.210:6443 --token yd6erz.akin2eqjyscughar --discovery-token-ca-cert-hash sha256:c032e1625181dcf87647fd361bfa2a7288d1e1b8e2f78973c8972ae6f84867de --ignore-preflight-errors=Swap

    39.png

    • From the "master" node, check K8s nodes now configured on your cluster ("master","worker1" and "worker2" should be there once completed); the option "wide" will also show the "container-runtime" used

    # kubectl get nodes -o wide

    40.png

    • For listing containers on "worker2" node (containerd), you now must use ctr command:

    # ctr --namespace k8s.io containers ls

    41.png

    Deploy Kata Container support into the K8s Cluster

    Good, now we have the right configured "kubelet", "containerd" in place on "worker2" node and we can move on and install Kata containers inside our cluster.

    • Run next commands on "master" node to deploy Kata containers support into the Kubernetes cluster

    # kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-rbac.yaml

    # kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-deploy.yaml

    # kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/k8s-1.14/kata-qemu-runtimeClass.yaml

    46.png

    • Run following command to check Kata Container support has been deployed on the K8s cluster

    # kubectl get pods --namespace=kube-system

    47.png

     

    Deploy K8s Dashboard and get cluster-admin token to get access to the GUI

    • Deploying dashboard is easy and straight forward; execute following command on the "master" node

    # kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta1/aio/deploy/recommended.yaml

    43.png

    • Run "kubectl proxy" in the background to open the Dashboard access on localhost:8001

    # kubectl proxy &

    44.png

     

    • On the K8s "Master Node" (192.168.99.100) and execute following commands

    # kubectl create serviceaccount cluster-admin-dashboard-sa

    # kubectl create clusterrolebinding cluster-admin-dashboard-sa --clusterrole=cluster-admin --serviceaccount=default:cluster-admin-dashboard-sa

     

    • Copy the token from generated secret into a dedicated file

    # kubectl get secret | grep cluster-admin-dashboard-sa

    # kubectl describe secret cluster-admin-dashboard-sa-token-xgdjk (value obtained by the previous step)

    05.png

    • Execute following command to save the token in a file

    # kubectl describe secret cluster-admin-dashboard-sa-token-xgdjk |grep "token:" |cut -d ":" -f2 |sed -e 's/ //g' > /home/vagrant/token

     

    • To get access to the K8s dashboard from the Host we need to create an ssh service forwarding process for port 8001 (from the VM to the Host) -- this port redirection could be already active

    # ssh -L 8001:localhost:8001 root@192.168.99.100

     

    • From the host, open a browser, connect to the following URL by using the token generated above

    http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

    45.png

    • By the K8s Dashboard we can also see the difference, in term of support, between "worker1" and "worker2" nodes, where "worker2" also supports "Kata Container".
    • Click on "Cluster => Nodes" and then expand the "Labels" for each "Worker" nodes, you'll see the support for Kata Container associated to "worker2"

    48.png

     

    Start Kubernetes pods on Worker nodes, leveraging Docker container as well as Containerd with Kata

     

    Deploy pods by leveraging Docker and Containerd Container Runtimes

    During this section we run an application by creating a Kubernetes Deployment object; by the K8S Dashboard we upload a YAML file containing the deployment description..

    This YAML file describes a Deployment that runs the "nginx:1.7.9" image.

    Here the content of the YAML file we use:

     

    apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2

    kind: Deployment

    metadata:

      name: nginx-deployment

    spec:

      selector:

        matchLabels:

          app: nginx

      replicas: 2 # tells deployment to run 2 pods matching the template

      template:

        metadata:

          labels:

            app: nginx

        spec:

          containers:

          - name: nginx

            image: nginx:1.7.9

            ports:

            - containerPort: 80

     

    To properly start those 2 containers on the K8s workers proceed with following steps.

    • On K8s dashboard click on "+" icon in the right-upper corner
    • On the "Create from text input" window, copy the YAML content available above and click on "Upload" button

    50.png

    • Once deployed, the two pods can be showed by both the "K8s dashboard" as well as "CLI"

    51.png

    # kubectl get pods

    52.png

    The big difference between the two pods is that:

    • nginx-deployment pod running on "worker1" uses "Docker" as Container Service
    • nginx-deployment pod running on "worker2" uses "Containerd" as Container Service

     

    Deploy pods by leveraging Kata Container Runtime

    This YAML file describes a Deployment that runs a "php/apache" image.

    Here the content of the YAML file we use:

     

    apiVersion: apps/v1

    kind: Deployment

    metadata:

      labels:

        run: php-apache-kata-qemu

      name: php-apache-kata-qemu

    spec:

      replicas: 1

      selector:

        matchLabels:

          run: php-apache-kata-qemu

      template:

        metadata:

          labels:

            run: php-apache-kata-qemu

        spec:

          runtimeClassName: kata-qemu

          containers:

          - image: k8s.gcr.io/hpa-example

            imagePullPolicy: Always

            name: php-apache

            ports:

            - containerPort: 80

              protocol: TCP

            resources:

              requests:

                cpu: 200m

          restartPolicy: Always

      nodeSelector:

        katacontainers.io/kata-runtime: "true"

       kubernetes.io/hostname: worker2.vagrant.vm

    ---

    apiVersion: v1

    kind: Service

    metadata:

      name: php-apache-kata-qemu

    spec:

      ports:

      - port: 80

        protocol: TCP

        targetPort: 80

      selector:

        run: php-apache-kata-qemu

      sessionAffinity: None

      type: ClusterIP

     

    To properly start this single Kata container on the K8s worker2 proceed with following steps.

    • On K8s dashboard click on "+" icon in the right-upper corner
    • On the "Create from text input" window, copy the YAML content available above and click on "Upload" button
    • From the "worker2" node we can verify that a new "qemu" process has started (Kata Container process)

    # ps -edaf |grep qemu

    54.png

    • From the "K8s master" node we can verify the container kernel running that will be
      • Same of the worker node (host) while running with Docker or Containerd
      • Dedicated and different while running with Kata Container

    # kubectl get pods

    # kubectl exec -it <nginx-pod-name> -- uname -a

    55.png

    Kernel running on the K8s workers is "4.14.35-1902.4.8.el7uek.x86_64" and, obviously, the same run on Docker as well as Containerd containers; different is the kernel running on the Kata Container because this pod has its own kernel and boot up as a lightweight Virtual Machine "4.19.52".