Oracle Container Services for use with Kubernetes V1.1.12 - Certificates (and how to update them)!

Version 6
Visibility: Open to anyone

    Introduction

     

    There are quite a few certificates involved with Kubernetes, and unfortunately they are not in one place. Information about managing these keys is not included in our current Documentation, so we will add this information to future versions of the "Oracle Container Services for use with Kubernetes User's Guide",

     

    For now, this Oracle Linux Community Document should help provide a lot of information about these certificates plus some instructions for managing these certificates and keep them from expiring!

     

    Big thanks to Mauricio Saenz for the Version 1.1.12 research, testing and information (contained in MyOracle Support Doc ID 2635389.1)


    Also note: The procedures listed here are NOT to be used with Oracle Linux Cloud Native Environment. Certificate renewal for Oracle Linux Cloud Native Environment will be handled by the API Server and documented separately.

     

    Certification Usage

     

    Certification usage in Kubernetes is important to understand, but it is a bit complicated. The TLS certs are being used broadly for 2 different purposes:

     

    • For running apiserver and kubelet in https mode which we will call "Server Side Certificates" . This is similar to running a https website .
    • For authenticating Kubernetes end users/clients (both internal as well as external) which we will call "Client Certificates" for authentication of users.

     

    In general on a Master Node, most of the certs live in /etc/kubernetes/pki, and Client certificates are usually with the client configuration files (kubelet.conf, controller-manager.conf, scheduler.conf ,admin.conf etc). The Master Node is also usually configured as a Worker Node within the cluster, so it is important to understand the usage and location of all of the keys in both Node types. Lets explore these in a bit more detail.

     

    Master Node

     

    In a typical setup the following files are found in /etc/kubernetes/pki on the Master node:

    • ca.key, ca.crt

    This is an autogenerated CA pair which has validity of 10 years from the date generated and is not a concern or candidate while rotating certificates. Since all client certificates are signed by this CA, avoid recreating it as doing so will require making changes across all worker nodes for kubelet. The command kubeadm can manage this key.

    • apiserver.key, apiserver.crt

    This is the server side key pair for running API Server in https mode. It is valid for 1 year from the date generated. The command kubeadm can manage this key.

    • apiserver-kubelet-client.key, apiserver-kubelet-client.crt

    This is the client certificate pair, used by apiserver to get authenticated to kubelet of each worker node running in HTTPS mode. It is valid for 1 year from the date generated. The command kubeadm can manage this key.

    • sa.pub, sa.key

    This is the key pair used for Service account and it does not need any rotation

    • front-proxy-ca.key, front-proxy-ca.crt

    This is the CA pair for api proxy server. It is valid for 10 years and is not a concern or candidate while rotating certs.

    • front-proxy-client.key, front-proxy-client.crt

    This is the key pair used by apiserver to authenticate for proxy server. It is valid for 1 year from the date generated. The command kubeadm can manage this key.

     

    The Master node also contains many client certificates which are stored withiin their respective .conf files, not as separate key files. This is a bit unusual. These configuration files are contained in /etc/kubernetes. All of the following .conf files can be managed by using the command # kubeadm alpha phase kubeconfig

    • admin.conf

    Client/user is kubernetes-admin and key pair mentioned are base64 values in the field of client-certificate-data and client-key-data. It is valid for 1 year from the date generated.

    • kubelet.conf

    Client/user is system:node:<master node name> and key pair mentioned are base64 values in the field of client-certificate-data and client-key-data. It is valid for 1 year from the date generated.

    • controller-manager.conf

    Client/user is system:kube-controller-manager  and key pair mentioned are base64 values in the field of client-certificate-data and client-key-data. It is valid for 1 year from the date generated.

    • scheduler.conf

    Client/user is system:kube-scheduler and key pair mentioned are base64 values in the field of client-certificate-data and client-key-data. It is valid for 1 year from the date generated.

     

    Worker Nodes

     

    In a typical setup the following files are found in /var/lib/kubelet/pki on each Worker node:

    • kubelet.key, kubelet.crt

    This is the server key pair which is generated and auto signed by kubelet when it gets setup and is used to run kubelet in https mode. It is valid for 1 year from the date generated. Important Note: kubeadm does NOT manage this key pair! To refresh this key pair, remove them and restart kubelet.  Doing this will generate a new self signed pair, valid for 1 year from the date generated. Step 7 below shows the commands to accomplish this.

    • kubelet-client.key, kubelet-client.crt

    This is the client key pair of kubelet which has been signed by the default CA on the Master server, and is used by kubelet to authenticate itself to Master. It is valid for 1 year from the date generated. This key pair is of little concern as it gets auto renewed by kubelet when it gets near its expiry date. The parameter for this is --rotate-certificates=true which has been setup on each kubelet by default .

     

    Note: In Oracle Container Services, Master Nodes are also configured as Worker Nodes. Please refresh the kubelet.key on each Master Node by running the procedure found  above.

     

    Check the Version

    Run the following command as a non-root use on the Master node to discover the Version:

     

    $ kubectl get nodes

    Here is an example of some results:

     

    NAME       STATUS    ROLES      AGE       VERSION

    master     Ready     master     4m37s     v1.12.10.10+1.0.10.el7

    worker1    Ready     <none>     3m27s     v1.12.10.10+1.0.10.el7

     

     

    Suggested procedure

     

    1.Perform a Cluster Backup

     

    Single Master Cluster

    Please use the procedure documented in section 4.3.1 of the Oracle Linux Container Services for use with Kubernetes User's Guide:

    https://docs.oracle.com/en/operating-systems/oracle-linux/kubernetes/kube-admin-masterclust.html

     

    High Availability Cluster

    Please use the procedure documented in section 4.3.2 of the Oracle Linux Container Services for use with Kubernetes User's Guide:

    https://docs.oracle.com/en/operating-systems/oracle-linux/kubernetes/kube-admin-hacluster.html

     

    2. Backup current PKI certificates

     

    Enter the following commands on the Master Node while logged in as root:

     

    # cd /etc/kubernetes

    # mkdir cert-backup

    # cd pki

    # for f in `ls apiserver* front-*client*`; do cp $f /etc/kubernetes/cert-backup/$f.old; done

     

    Check the backup at this point if desired:

     

    # ls /etc/kubernetes/cert-backup/

    apiserver.crt.old               apiserver-kubelet-client.crt.old
    apiserver-etcd-client.crt.old   apiserver-kubelet-client.key.old
    apiserver-etcd-client.key.old   front-proxy-client.crt.old
    apiserver.key.old               front-proxy-client.key.old

     

    3. Backup current etcd certificates

     

    Enter the following commands in order while logged into the Kubernetes Master Node as root:

    # cd /etc/kubernetes/pki/etcd

    # for f in `ls heal* peer* server*`; do cp $f /etc/kubernetes/cert-backup/$f.old; done

    To check the backup directory at this point:

    # ls /etc/kubernetes/cert-backup/

    apiserver.crt.old                    front-proxy-client.key.old
    apiserver-etcd-client.crt.old        healthcheck-client.crt.old
    apiserver-etcd-client.key.old        healthcheck-client.key.old
    apiserver.key.old                    peer.crt.old
    apiserver-kubelet-client.crt.old     peer.key.old 
    apiserver-kubelet-client.key.old     server.crt.old  
    front-proxy-client.crt.old           server.key.old

     

    4. Renew certificates for Kubernetes

     

    Enter the following commands on the Master Node while logged in as root:

    # cd /etc/kubernetes

    # kubeadm alpha phase certs renew all

    You should see results similar to the following - these are normal warnings (the upstream current version is newer)

    I0205 19.34.02.209875     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.05.027132     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.05.950231     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.06.649565     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.08.935669     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.10.499370     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12
    I0205 19.34.11.228048     6557     version.go:236] remote version is much newer: v1.17.2; falling back to: stable-1.12

     

    For a multi-master configuration, repeat steps 2 - 4 on each master node.

     

    5. Verify that the certificates have been renewed

    In the example below the renewed certificates have the 19:34 timestamp. The expected certificates have been renewed.

    [root@master kubernetes]# ls -la /etc/kubernetes/pki

    total 68

    drwxr-xr-x. 3 root root 4096 Feb  5 18:41 .

    drwxr-xr-x. 5 root root 4096 Feb  5 19:22 ..

    -rw-r--r--. 1 root root 1233 Feb  5 19:34 apiserver.crt

    -rw-r--r--. 1 root root 1090 Feb  5 19:34 apiserver-etcd-client.crt

    -rw-------. 1 root root 1679 Feb  5 19:34 apiserver-etcd-client.key

    -rw-------. 1 root root 1675 Feb  5 19:34 apiserver.key

    -rw-r--r--. 1 root root 1099 Feb  5 19:34 apiserver-kubelet-client.crt

    -rw-------. 1 root root 1675 Feb  5 19:34 apiserver-kubelet-client.key

    -rw-r--r--. 1 root root 1025 Feb  5 18:41 ca.crt

    -rw-------. 1 root root 1675 Feb  5 18:41 ca.key

    drwxr-xr-x. 2 root root 4096 Feb  5 18:41 etcd

    -rw-r--r--. 1 root root 1038 Feb  5 18:41 front-proxy-ca.crt

    -rw-------. 1 root root 1679 Feb  5 18:41 front-proxy-ca.key

    -rw-r--r--. 1 root root 1058 Feb  5 19:34 front-proxy-client.crt

    -rw-------. 1 root root 1679 Feb  5 19:34 front-proxy-client.key

    -rw-------. 1 root root 1675 Feb  5 18:41 sa.key

    -rw-------. 1 root root  451 Feb  5 18:41 sa.pub

     

    [root@master kubernetes]# ls -la /etc/kubernetes/pki/etcd

    total 40

    drwxr-xr-x. 2 root root 4096 Feb  5 18:41 .

    drwxr-xr-x. 3 root root 4096 Feb  5 18:41 ..

    -rw-r--r--. 1 root root 1017 Feb  5 18:41 ca.crt

    -rw-------. 1 root root 1675 Feb  5 18:41 ca.key

    -rw-r--r--. 1 root root 1094 Feb  5 19:34 healthcheck-client.crt

    -rw-------. 1 root root 1675 Feb  5 19:34 healthcheck-client.key

    -rw-r--r--. 1 root root 1155 Feb  5 19:34 peer.crt

    -rw-------. 1 root root 1679 Feb  5 19:34 peer.key

    -rw-r--r--. 1 root root 1147 Feb  5 19:34 server.crt

    -rw-------. 1 root root 1679 Feb  5 19:34 server.key

     

    If desired you can run the following commands to check the expiration dates on your certificates:

     

    [root@master ~]# cd /etc/kubernetes/pki

    [root@master pki]# for i in apiserver.crt apiserver-etcd-client.crt apiserver-kubelet-client.crt front-proxy-ca.crt front-proxy-client.crt; do echo "Doing:" $i; openssl x509 -enddate -noout -in $i; done

    Doing: apiserver.crt

    notAfter=Feb  4 19:34:04 2021 GMT

    Doing: apiserver-etcd-client.crt

    notAfter=Feb  4 19:34:08 2021 GMT

    Doing: apiserver-kubelet-client.crt

    notAfter=Feb  4 19:34:05 2021 GMT

    Doing: front-proxy-ca.crt

    notAfter=Feb  2 18:41:07 2030 GMT

    Doing: front-proxy-client.crt

    notAfter=Feb  4 19:34:12 2021 GMT

     

    [root@master ~]# cd /etc/kubernetes/pki/etcd

    [root@master etcd]# for i in healthcheck-client.crt peer.crt server.crt; do echo "Doing:" $i; openssl x509 -enddate -noout -in $i; done

    Doing: healthcheck-client.crt

    notAfter=Feb  4 19:34:06 2021 GMT

    Doing: peer.crt

    notAfter=Feb  4 19:34:10 2021 GMT

    Doing: server.crt

    notAfter=Feb  4 19:34:10 2021 GMT

     

    6. Copy new config file to proper directory

     

    While logged in as a non-root user on the Master node, enter the following command:

    $ sudo cp /etc/kubernetes/admin.conf $HOME/

    $ sudo chown $(id -u):$(id -g) $HOME/admin.conf

    $ export KUBECONFIG=$HOME/admin.conf

    7. Generate new certificates for kubelet on worker nodes

    Log into each worker node and enter the following commands as a root user on each worker:

    # cd /var/lib/kubelet/pki

    # rm kubelet.key kubelet.crt

    # systemctl restart kubelet

    8. (Optional) Renew the certificate for the Kubernetes admin user

    The admin user certificate has a 1 year expiration. Here are the steps to renew it:

     

    On the master node (or a master node in a multi master cluster) while logged in as a root user:

     

    # cd /etc/kubernetes

    # mkdir admin

    # cd admin

     

    The next command generates the private key for admin user:

     

    # openssl genrsa -out admin.key 2048

    Generating RSA private key, 2048 bit long modulus

    .........................................................................................................+++

    ..............+++

    e is 65537 (0x10001)

     

     

    The next command generates the CSR for the admin user - no output is returned to the command line if the command is successful.

     

    # openssl req -new -key admin.key -subj "/CN=kubernetes-admin/O=system:masters" -out admin.csr

     

    The next command signs the certificate for the admin user using the server CA private key. NOTE: Replace <1000> with the number of days desired until the certificate expires.

     

    # openssl x509 -req -in admin.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out admin.crt -days 1000

    Signature ok

    subject=/CN=kubernetes-admin/O=system:masters

    Getting CA Private Key

     

    The next steps are to extract the certificates for entry into the admin user's config file.

     

    Run the following commands - note there will be a long sequence of text generated with each command.

     

    # cat admin.crt | base64 | tr -d '\n'

    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4VENDQWEwQ0NRRFdYc25nbVpQMGxEQU5CZ2txaGtpRzl3MEJBUXNGQURBVk1STXdFUVlEVlFRREV3cHIKZFdKbGNtNWxkR1Z6TUI0WERUSXdNREl3TlRJeE5ERXhNMW9YRFRJeU1URXdNVEl4TkRFeE0xb3dOREVaTUJjRwpBMVVFQXd3UWEzVmlaWEp1WlhSbGN5MWhaRzFwYmpFWE1CVUdBMVVFQ2d3T2MzbHpkR1Z0T20xaGMzUmxjbk13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUM4MWZwdUptOXVsbTc3TFFxZytaZE0KUG1zWmpnK0p6NW9ST2FaVVpYU1lUbVZTZjdGbkcrNjNJbUVaSG9uQXRzZkdpSzhQQUZ3TkJuWXpJcGxrTDFaLwp3QUx1Q1pwNHgzcDY4TW9ralFNR0x1ZER3SmtPWFFBTjJUYy9NVTNJNFl5MHF4Mi85M3gxVjl6NUZCQ2tKMnNNCkFLa1M1aGlqRncyZkFaSGRDZGtZdXhMNkpYR0hxeGhUSTRQRWRyNVloeVFxUmlEUHRyZGtyZ1d0dFgyaHhsb24Kb3lhL0tKYjNpZmcwQXpKQlNaMzNtVk1kRTZ6TVNhakpraWNPbEtZQmp3RUhLVnh4OEVUQzdOMEx6NFZMVXc5VQpjcmNxZ0RtS0VieGxwa2l3M044SzM1djY4OXJ2VWZCcVd6Smg2d0xWdWJhbmhHZDl6OUU4ankrbnJSbFV0K1lQCkFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUVLaFUwdXc4MVE4bERhL1c1Y2cvMjluYUYwSUUwcGwKbTRlZDllMHNHOTJMNnNzWHRZUG50RVFTQjZ0YU1LSHNEUkhHV2FRZjZUTVhGVXdUeEF6OHNNR2VzdjNZQ1B4bQpDdjdrMTdWZHV2aUdnMUM2YzlZTVRIaGJmN0ZhN2RnSExCcEdYZUUyMnY1L2U0N04wTkwwVStMRXF4SUpkUmJ1CkJMVDduRXRNYkJBQXlERGk5UUpOOWc0cVhTQjVkaExCd0U0QkJraVFmME9Jcmt1UUhQRllqaFczWlh4dVVBSlgKTXNjVkhOTkZVQ2FhQVJaVE9YNHJrMDc1VVE3VGlpcWFKV2VuU0tPVk9YL1R5SVJxdnR2NFFBTDRDMDJOSi9NNgppd0JxM00rRlU3ZG5pVTZ1VlRQSEFmT0JlSjJpZFRqU2FIQUd0eStKZlN2aDBEVmpzY05RdERFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

     

    # cat admin.key | base64 | tr -d '\n'
    LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdk5YNmJpWnZicFp1K3kwS29QbVhURDVyR1k0UGljK2FFVG1tVkdWMG1FNWxVbit4Clp4dnV0eUpoR1I2SndMYkh4b2l2RHdCY0RRWjJNeUtaWkM5V2Y4QUM3Z21hZU1kNmV2REtKSTBEQmk3blE4Q1oKRGwwQURkazNQekZOeU9HTXRLc2R2L2Q4ZFZmYytSUVFwQ2RyREFDcEV1WVlveGNObndHUjNRblpHTHNTK2lWeApoNnNZVXlPRHhIYStXSWNrS2tZZ3o3YTNaSzRGcmJWOW9jWmFKNk1tdnlpVzk0bjROQU15UVVtZDk1bFRIUk9zCnpFbW95WkluRHBTbUFZOEJCeWxjY2ZCRXd1emRDOCtGUzFNUFZISzNLb0E1aWhHOFphWklzTnpmQ3QrYit2UGEKNzFId2Fsc3lZZXNDMWJtMnA0Um5mYy9SUEk4dnA2MFpWTGZtRHdJREFRQUJBb0lCQUd2U0VjZkxFbDBtYi8vaQoyK3JHR0dsa1NtcXcvcFpGZjBiT2ovWW9aOFovSE1OYVBjVU40ZU9YTWJIU0NmMkxYODR3UlNSZnBDK2J2T205Cm02L2toNU14NDNwZTZXQ0ZKWjRFMXFiQURUWCttZlhKaHByNDV3c0JOOVpSSklUS3BEaVVhOHdzVjFmNDd5Y1oKR2RkZzJuSmovaVhZVThEcmJNK1phZHUzdjhtcUtxamEySnZxRGpOaXJOY1ZtWG42WVBHTml4Z1VzSXJidWdzVAovdWhSUUxHWmZCeXpEQThZZVBDUlFNQldhS2VoRlRSSlBtZUlpRk1id1ZiT250ajRTbGYvVkhwemRxem4wdGhhCjh3VjRuMnFXNDU3amZaVHRCWU1mMjN5Y2t6T3ZGUm10NjJWekREOFBvUFczTWVCMVc5bnRoUzcxdDBwaEZKcWgKRjBmd3JVRUNnWUVBOWQzR3ZWSTJ3azZJY0F6SVJPbHJ6TjNKV0Q3M0F4STJ1Qlp3cXZQcHRyZmU2OUI3QkpmZgpPRTJadkdxcXdCK0lJTm1FeUJxZ0hMczNZVEdVbTZEZDRJQnlDZXN4S3ZKeU0zWDgrb0NCMDVOeXQzM21qcHNYCkZpZ21rZDNCeDQyUm1WTUptcW1YTFpRRzJULysyM0ZrdXZ3Q01wUldoUkhQNS9qV0lTSmlJMmtDZ1lFQXhKNXoKKytMbk9qU0xHQ043cmxWaWVGc3hYR2hjQ1dBemtZVGttQWloY04renQ2V3pnVHFBYW1lUFdoUk9vQjgwVTgzWQpLSlRUcWJIak5rTG9hY1pwNW9tbyt0WXBCZkhmOFRMR3E0U1ozaXU4T2s0cmFTVWs0bU01MVZWYzd1N0ZuaHlOCk05cGNHQUtiWXorVnl1YURRVmlqVEE4K3Y1Q1I2bXpFMXozM0pyY0NnWUFoVkRNQXJ3aGxScWdRS0d0dGpBYmsKK3B1MHJyUmxZaTZ3dmJvQU1waFlBSXpqZ3B0cWYvdnVjZmFKS0J5RmRzREpVek5BdlBzL2FkR1VCMWlSMERqRAptaVhiV2xxTDY5bTNTQ05IQWV5WGRjRnpSa25ld3Y5YUxZOHM3dGE1Nm1MMldkOVBhL1htWTg2WjNLYjdzRjlyClNZWXl2UkVOY09DeUhYVkpnSk8xSVFLQmdRQzd3T3ZVbm9wYmhJMUdrYXVyZ0JHMkNLOE1KUU91V0ZVbUlwYi8KcjA0cXNSTzJ6TjZyQ3FoUjgyNXFnSFhNWmIvY1B2YXBXZVQ5YU1jbHE2S3dUeTRWWFdNbGxKZzcrd3IwRXA2dwpic0ZYN0wramxiM1NmQXZLdXlJZzI1RVJBS1ZwSks2WjAyeVo2UE5sUlBUUGVtdmdYTG9qQ1hQTURrdW9aaEZaCjBPeFA1UUtCZ1FEZ0NtSlhwbVVHNnhrb3J2bGc2OVJsSlp2Z1dDbkZ2elJISTNsekZhUFdtSnlFQVVaM3FGSnMKcURNcWsyTndOYzk0djMwc2Q3ajBIS0NjWk9EL0VXeGlOVVlXUFF2bUNkdHNIRkIrSnN4d01Hazd0SUh3Y3FvaQpyS0ZHVmQ5K01waVp3b3lRb21hVlhJaHh6RVlLd3QzYlNMVWFtZytxS3YzV0RrRHNCOG05ZkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

     

     

    As a non-root user, edit the file $HOME/.kube/config

     

    Carefully copy the text string generated from the admin.crt command above, and paste it in the "client-certificate-data" field, replacing the old text in that field.

    Carefully copy the text string generated from the admin.key command above and paste it in the "client-key-data" field, replacing the old text in that field.

     

    Here is an example of a completed config file after the replacement of the text.

     

    apiVersion: v1

    clusters:

    - cluster:

        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ESXdOVEU0TkRFd05sb1hEVE13TURJd01qRTROREV3Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUZICnhBbkNmMmV1Sjd1YkZadEIwb2pUSGpOMFhRd1RSV0wvbldSa3lYemxUbGRhLzMrZDV5SHZqalNtUTR2QmhFTmIKNzROcXBUaVl0dGVrTTNqTldiVHo3TU5Hb2V0WkdYdVArcnpRRS8vSEdVZFRZYi9yUjhYZGlmdE03U0VURzBkYQozZkN1SVB6VVArOFVwb1pUMVR4aWIxdUtwMENDdXVLdXJLY2sxYmh0UWQ0bHZ5TXV5NmZiM3d0ZEpUMUNBUlZHClR5YkZuQ3lENlZCdWlpL3ZMY0wzaW5Ia095b3V4b3B4ZXpUNnRxUEJtQ0Vmczk4UXBVWFJjeitiTGgzZyt3NXkKMitUUE5QNG5nUjVkTm5yS0VYRFF5Y3FIbW1La0lFQ3k4S1BXV2NXbUlvTnhyV3BLeHl3eHVxZ1VqOWtkdmc2agpXZWhCN1Nnam9CeEs2MTcyM044Q0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeUJWTk1XUXdwUlA0SlJ2eHJkQ0FKYXF2WFIKUDNrNjlpb3RMZEVwRENLNDlFdFF5MnM2VE5oZDJKYWhVTmRrangzRE9kZjFKZkovakEwQmxRYzdHRGp6V2F4cwo0TWk5cVowSGhBZVpJRjNFVE01RnBoYnRueDNncGk4SzcvemJUbnhWZHgvVldsa2o4RDc0ZTZDTURpN3RaSEJXCk5lMmN3UDRaWjhqT2NTOTJvcXlTRnQ0ekxzdU4yZ1l2alB4QUt1WnA2dWJlcktFTVRsbTBHNExoRWxhTTY2RDAKU0FZdGNMeG9rSkh4ZExzN1l1Q0hDdGczRzhiVUtKZ0k5TDVSYnR4TEQ0d2txRjd1bW05dU5BT3B5VStwcFJZMApiK1NDc0E4U3BxUDFRdGwzMzVpV2M5ZTIxalpEYTlOTWhnSi9hVGZOWVB6eTJQeml0bHFhaG9WV0lMMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

        server: https://192.168.99.100:6443

      name: kubernetes

    contexts:

    - context:

        cluster: kubernetes

        user: kubernetes-admin

      name: kubernetes-admin@kubernetes

    current-context: kubernetes-admin@kubernetes

    kind: Config

    preferences: {}

    users:

    - name: kubernetes-admin

      user:

        client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4VENDQWEwQ0NRRFdYc25nbVpQMGxEQU5CZ2txaGtpRzl3MEJBUXNGQURBVk1STXdFUVlEVlFRREV3cHIKZFdKbGNtNWxkR1Z6TUI0WERUSXdNREl3TlRJeE5ERXhNMW9YRFRJeU1URXdNVEl4TkRFeE0xb3dOREVaTUJjRwpBMVVFQXd3UWEzVmlaWEp1WlhSbGN5MWhaRzFwYmpFWE1CVUdBMVVFQ2d3T2MzbHpkR1Z0T20xaGMzUmxjbk13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUM4MWZwdUptOXVsbTc3TFFxZytaZE0KUG1zWmpnK0p6NW9ST2FaVVpYU1lUbVZTZjdGbkcrNjNJbUVaSG9uQXRzZkdpSzhQQUZ3TkJuWXpJcGxrTDFaLwp3QUx1Q1pwNHgzcDY4TW9ralFNR0x1ZER3SmtPWFFBTjJUYy9NVTNJNFl5MHF4Mi85M3gxVjl6NUZCQ2tKMnNNCkFLa1M1aGlqRncyZkFaSGRDZGtZdXhMNkpYR0hxeGhUSTRQRWRyNVloeVFxUmlEUHRyZGtyZ1d0dFgyaHhsb24Kb3lhL0tKYjNpZmcwQXpKQlNaMzNtVk1kRTZ6TVNhakpraWNPbEtZQmp3RUhLVnh4OEVUQzdOMEx6NFZMVXc5VQpjcmNxZ0RtS0VieGxwa2l3M044SzM1djY4OXJ2VWZCcVd6Smg2d0xWdWJhbmhHZDl6OUU4ankrbnJSbFV0K1lQCkFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUVLaFUwdXc4MVE4bERhL1c1Y2cvMjluYUYwSUUwcGwKbTRlZDllMHNHOTJMNnNzWHRZUG50RVFTQjZ0YU1LSHNEUkhHV2FRZjZUTVhGVXdUeEF6OHNNR2VzdjNZQ1B4bQpDdjdrMTdWZHV2aUdnMUM2YzlZTVRIaGJmN0ZhN2RnSExCcEdYZUUyMnY1L2U0N04wTkwwVStMRXF4SUpkUmJ1CkJMVDduRXRNYkJBQXlERGk5UUpOOWc0cVhTQjVkaExCd0U0QkJraVFmME9Jcmt1UUhQRllqaFczWlh4dVVBSlgKTXNjVkhOTkZVQ2FhQVJaVE9YNHJrMDc1VVE3VGlpcWFKV2VuU0tPVk9YL1R5SVJxdnR2NFFBTDRDMDJOSi9NNgppd0JxM00rRlU3ZG5pVTZ1VlRQSEFmT0JlSjJpZFRqU2FIQUd0eStKZlN2aDBEVmpzY05RdERFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

        client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdk5YNmJpWnZicFp1K3kwS29QbVhURDVyR1k0UGljK2FFVG1tVkdWMG1FNWxVbit4Clp4dnV0eUpoR1I2SndMYkh4b2l2RHdCY0RRWjJNeUtaWkM5V2Y4QUM3Z21hZU1kNmV2REtKSTBEQmk3blE4Q1oKRGwwQURkazNQekZOeU9HTXRLc2R2L2Q4ZFZmYytSUVFwQ2RyREFDcEV1WVlveGNObndHUjNRblpHTHNTK2lWeApoNnNZVXlPRHhIYStXSWNrS2tZZ3o3YTNaSzRGcmJWOW9jWmFKNk1tdnlpVzk0bjROQU15UVVtZDk1bFRIUk9zCnpFbW95WkluRHBTbUFZOEJCeWxjY2ZCRXd1emRDOCtGUzFNUFZISzNLb0E1aWhHOFphWklzTnpmQ3QrYit2UGEKNzFId2Fsc3lZZXNDMWJtMnA0Um5mYy9SUEk4dnA2MFpWTGZtRHdJREFRQUJBb0lCQUd2U0VjZkxFbDBtYi8vaQoyK3JHR0dsa1NtcXcvcFpGZjBiT2ovWW9aOFovSE1OYVBjVU40ZU9YTWJIU0NmMkxYODR3UlNSZnBDK2J2T205Cm02L2toNU14NDNwZTZXQ0ZKWjRFMXFiQURUWCttZlhKaHByNDV3c0JOOVpSSklUS3BEaVVhOHdzVjFmNDd5Y1oKR2RkZzJuSmovaVhZVThEcmJNK1phZHUzdjhtcUtxamEySnZxRGpOaXJOY1ZtWG42WVBHTml4Z1VzSXJidWdzVAovdWhSUUxHWmZCeXpEQThZZVBDUlFNQldhS2VoRlRSSlBtZUlpRk1id1ZiT250ajRTbGYvVkhwemRxem4wdGhhCjh3VjRuMnFXNDU3amZaVHRCWU1mMjN5Y2t6T3ZGUm10NjJWekREOFBvUFczTWVCMVc5bnRoUzcxdDBwaEZKcWgKRjBmd3JVRUNnWUVBOWQzR3ZWSTJ3azZJY0F6SVJPbHJ6TjNKV0Q3M0F4STJ1Qlp3cXZQcHRyZmU2OUI3QkpmZgpPRTJadkdxcXdCK0lJTm1FeUJxZ0hMczNZVEdVbTZEZDRJQnlDZXN4S3ZKeU0zWDgrb0NCMDVOeXQzM21qcHNYCkZpZ21rZDNCeDQyUm1WTUptcW1YTFpRRzJULysyM0ZrdXZ3Q01wUldoUkhQNS9qV0lTSmlJMmtDZ1lFQXhKNXoKKytMbk9qU0xHQ043cmxWaWVGc3hYR2hjQ1dBemtZVGttQWloY04renQ2V3pnVHFBYW1lUFdoUk9vQjgwVTgzWQpLSlRUcWJIak5rTG9hY1pwNW9tbyt0WXBCZkhmOFRMR3E0U1ozaXU4T2s0cmFTVWs0bU01MVZWYzd1N0ZuaHlOCk05cGNHQUtiWXorVnl1YURRVmlqVEE4K3Y1Q1I2bXpFMXozM0pyY0NnWUFoVkRNQXJ3aGxScWdRS0d0dGpBYmsKK3B1MHJyUmxZaTZ3dmJvQU1waFlBSXpqZ3B0cWYvdnVjZmFKS0J5RmRzREpVek5BdlBzL2FkR1VCMWlSMERqRAptaVhiV2xxTDY5bTNTQ05IQWV5WGRjRnpSa25ld3Y5YUxZOHM3dGE1Nm1MMldkOVBhL1htWTg2WjNLYjdzRjlyClNZWXl2UkVOY09DeUhYVkpnSk8xSVFLQmdRQzd3T3ZVbm9wYmhJMUdrYXVyZ0JHMkNLOE1KUU91V0ZVbUlwYi8KcjA0cXNSTzJ6TjZyQ3FoUjgyNXFnSFhNWmIvY1B2YXBXZVQ5YU1jbHE2S3dUeTRWWFdNbGxKZzcrd3IwRXA2dwpic0ZYN0wramxiM1NmQXZLdXlJZzI1RVJBS1ZwSks2WjAyeVo2UE5sUlBUUGVtdmdYTG9qQ1hQTURrdW9aaEZaCjBPeFA1UUtCZ1FEZ0NtSlhwbVVHNnhrb3J2bGc2OVJsSlp2Z1dDbkZ2elJISTNsekZhUFdtSnlFQVVaM3FGSnMKcURNcWsyTndOYzk0djMwc2Q3ajBIS0NjWk9EL0VXeGlOVVlXUFF2bUNkdHNIRkIrSnN4d01Hazd0SUh3Y3FvaQpyS0ZHVmQ5K01waVp3b3lRb21hVlhJaHh6RVlLd3QzYlNMVWFtZytxS3YzV0RrRHNCOG05ZkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

     

    9. Reboot the Master node as a final test

     

    Reboot the master node as a non-root user. Your kubectl command should still work correctly.

     

    $ kubectl get nodes
    NAME                  STATUS     ROLES      AGE       VERSION
    master.vagrant.vm     Ready      master    3h52m      v1.12.10+1.0.10.el7
    worker1.vagrant.vm    Ready     <none>     3h46m      v1.12.10+1.0.10.el7
    worker2.vagrant.vm    Ready     <none>     3h42m      v1.12.10+1.0.10.el7