container.training/slides/k8s/cluster-upgrade.md

Upgrading clusters

• It's recommended to run consistent versions across a cluster

  (mostly to have feature parity and latest security updates)

• It's not mandatory

  (otherwise, cluster upgrades would be a nightmare!)

• Components can be upgraded one at a time without problems

---

Checking what we're running

• It's easy to check the version for the API server

.exercise[

• Log into node test1

• Check the version of kubectl and of the API server:

  kubectl version
  

]

• In a HA setup with multiple API servers, they can have different versions

• Running the command above multiple times can return different values

---

Node versions

• It's also easy to check the version of kubelet

.exercise[

• Check node versions (includes kubelet, kernel, container engine):

  kubectl get nodes -o wide
  

]

• Different nodes can run different kubelet versions

• Different nodes can run different kernel versions

• Different nodes can run different container engines

---

Control plane versions

• If the control plane is self-hosted (running in pods), we can check it

.exercise[

• Show image versions for all pods in kube-system namespace:

    kubectl --namespace=kube-system get pods -o json \
            | jq -r '
              .items[]
              | [.spec.nodeName, .metadata.name]
                + 
                (.spec.containers[].image | split(":"))
              | @tsv
              ' \
            | column -t
  

]

---

What version are we running anyway?

• When I say, "I'm running Kubernetes 1.15", is that the version of:

  • kubectl

  • API server

  • kubelet

  • controller manager

  • something else?

---

Other versions that are important

• etcd

• kube-dns or CoreDNS

• CNI plugin(s)

• Network controller, network policy controller

• Container engine

• Linux kernel

---

General guidelines

• To update a component, use whatever was used to install it

• If it's a distro package, update that distro package

• If it's a container or pod, update that container or pod

• If you used configuration management, update with that

---

Know where your binaries come from

• Sometimes, we need to upgrade quickly

  (when a vulnerability is announced and patched)

• If we are using an installer, we should:

  • make sure it's using upstream packages

  • or make sure that whatever packages it uses are current

  • make sure we can tell it to pin specific component versions

---

Important questions

• Should we upgrade the control plane before or after the kubelets?

• Within the control plane, should we upgrade the API server first or last?

• How often should we upgrade?

• How long are versions maintained?

• All the answers are in the documentation about version skew policy!

• Let's review the key elements together ...

---

Kubernetes uses semantic versioning

• Kubernetes versions look like MAJOR.MINOR.PATCH; e.g. in 1.17.2:

  • MAJOR = 1
  • MINOR = 17
  • PATCH = 2

• It's always possible to mix and match different PATCH releases

  (e.g. 1.16.1 and 1.16.6 are compatible)

• It is recommended to run the latest PATCH release

  (but it's mandatory only when there is a security advisory)

---

Version skew

• API server must be more recent than its clients (kubelet and control plane)

• ... Which means it must always be upgraded first

• All components support a difference of one¹ MINOR version

• This allows live upgrades (since we can mix e.g. 1.15 and 1.16)

• It also means that going from 1.14 to 1.16 requires going through 1.15

.footnote[¹Except kubelet, which can be up to two MINOR behind API server, and kubectl, which can be one MINOR ahead or behind API server.]

---

Release cycle

• There is a new PATCH relese whenever necessary

  (every few weeks, or "ASAP" when there is a security vulnerability)

• There is a new MINOR release every 3 months (approximately)

• At any given time, three MINOR releases are maintained

• ... Which means that MINOR releases are maintained approximately 9 months

• We should expect to upgrade at least every 3 months (on average)

---

In practice

• We are going to update a few cluster components

• We will change the kubelet version on one node

• We will change the version of the API server

• We will work with cluster test (nodes test1, test2, test3)

---

Updating the API server

• This cluster has been deployed with kubeadm

• The control plane runs in static pods

• These pods are started automatically by kubelet

  (even when kubelet can't contact the API server)

• They are defined in YAML files in /etc/kubernetes/manifests

  (this path is set by a kubelet command-line flag)

• kubelet automatically updates the pods when the files are changed

---

Changing the API server version

• We will edit the YAML file to use a different image version

.exercise[

• Log into node test1

• Check API server version:

  kubectl version
  

• Edit the API server pod manifest:

  sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
  

• Look for the image: line, and update it to e.g. v1.16.0

]

---

Checking what we've done

• The API server will be briefly unavailable while kubelet restarts it

.exercise[

• Check the API server version:

  kubectl version
  

]

---

Was that a good idea?

--

No!

--

• Remember the guideline we gave earlier:

  To update a component, use whatever was used to install it.

• This control plane was deployed with kubeadm

• We should use kubeadm to upgrade it!

---

Updating the whole control plane

• Let's make it right, and use kubeadm to upgrade the entire control plane

  (note: this is possible only because the cluster was installed with kubeadm)

.exercise[

• Check what will be upgraded:

  sudo kubeadm upgrade plan
  

]

Note 1: kubeadm thinks that our cluster is running 1.16.0.
It is confused by our manual upgrade of the API server!

Note 2: kubeadm itself is still version 1.15.9.
It doesn't know how to upgrade do 1.16.X.

---

Upgrading kubeadm

• First things first: we need to upgrade kubeadm

.exercise[

• Upgrade kubeadm:

  sudo apt install kubeadm
  

• Check what kubeadm tells us:

  sudo kubeadm upgrade plan
  

]

Problem: kubeadm doesn't know know how to handle upgrades from version 1.15.

This is because we installed version 1.17 (or even later).

We need to install kubeadm version 1.16.X.

---

Downgrading kubeadm

• We need to go back to version 1.16.X (e.g. 1.16.6)

.exercise[

• View available versions for package kubeadm:

  apt show kubeadm -a | grep ^Version | grep 1.16
  

• Downgrade kubeadm:

  sudo apt install kubeadm=1.16.6-00
  

• Check what kubeadm tells us:

  sudo kubeadm upgrade plan
  

]

kubeadm should now agree to upgrade to 1.16.6.

---

Upgrading the cluster with kubeadm

• Ideally, we should revert our image: change

  (so that kubeadm executes the right migration steps)

• Or we can try the upgrade anyway

.exercise[

• Perform the upgrade:

  sudo kubeadm upgrade apply v1.16.6
  

]

---

Updating kubelet

• These nodes have been installed using the official Kubernetes packages

• We can therefore use apt or apt-get

.exercise[

• Log into node test3

• View available versions for package kubelet:

  apt show kubelet -a | grep ^Version
  

• Upgrade kubelet:

  sudo apt install kubelet=1.16.6-00
  

]

---

Checking what we've done

.exercise[

• Log into node test1

• Check node versions:

  kubectl get nodes -o wide
  

• Create a deployment and scale it to make sure that the node still works

]

---

Was that a good idea?

--

Almost!

--

• Yes, kubelet was installed with distribution packages

• However, kubeadm took care of configuring kubelet

  (when doing kubeadm join ...)

• We were supposed to run a special command before upgrading kubelet!

• That command should be executed on each node

• It will download the kubelet configuration generated by kubeadm

---

Upgrading kubelet the right way

• We need to upgrade kubeadm, upgrade kubelet config, then upgrade kubelet

  (after upgrading the control plane)

.exercise[

• Download the configuration on each node, and upgrade kubelet:

    for N in 1 2 3; do
      ssh test$N "
        sudo apt install kubeadm=1.16.6-00 &&
        sudo kubeadm upgrade node &&
        sudo apt install kubelet=1.16.6-00"
    done
  

]

---

Checking what we've done

• All our nodes should now be updated to version 1.16.6

.exercise[

• Check nodes versions:

  kubectl get nodes -o wide
  

]

---

class: extra-details

Skipping versions

• This example worked because we went from 1.15 to 1.16

• If you are upgrading from e.g. 1.14, you will have to go through 1.15 first

• This means upgrading kubeadm to 1.15.X, then using it to upgrade the cluster

• Then upgrading kubeadm to 1.16.X, etc.

• Make sure to read the release notes before upgrading!