From 1f77a52137d1db4677904bea98c5084df17fcf7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=B4me=20Petazzoni?= Date: Thu, 30 Nov 2023 16:45:11 -0600 Subject: [PATCH] =?UTF-8?q?=F0=9F=93=83=20Flesh=20out=20upgrade=20informat?= =?UTF-8?q?ion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the official policy (which is to drain nodes before upgrading), and give some explanations about when it may/may not be fine to upgrade without draining nodes. --- slides/k8s/cluster-upgrade.md | 80 +++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/slides/k8s/cluster-upgrade.md b/slides/k8s/cluster-upgrade.md index 5738e903..f65905dc 100644 --- a/slides/k8s/cluster-upgrade.md +++ b/slides/k8s/cluster-upgrade.md @@ -507,6 +507,86 @@ kubeadm should now agree to upgrade to 1.23.X. --- +## And now, was that a good idea? + +-- + +**Almost!** + +-- + +- The official recommendation is to *drain* a node before performing node maintenance + + (migrate all workloads off the node before upgrading it) + +- How do we do that? + +- Is it really necessary? + +- Let's see! + +--- + +## Draining a node + +- This can be achieved with the `kubectl drain` command, which will: + + - *cordon* the node (prevent new pods from being scheduled there) + + - *evict* all the pods running on the node (delete them gracefully) + + - the evicted pods will automatically be recreated somewhere else + + - evictions might be blocked in some cases (Pod Disruption Budgets, `emptyDir` volumes) + +- Once the node is drained, it can safely be upgraded, restarted... + +- Once it's ready, it can be put back in commission with `kubectl uncordon` + +--- + +## Is it necessary? + +- When upgrading kubelet from one patch-level version to another: + + - it's *probably fine* + +- When upgrading system packages: + + - it's *probably fine* + + - except [when it's not][datadog-systemd-outage] + +- When upgrading the kernel: + + - it's *probably fine* + + - ...as long as we can tolerate a restart of the containers on the node + + - ...and that they will be unavailable for a few minutes (during the reboot) + +[datadog-systemd-outage]: https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-impact/ + +--- + +## Is it necessary? + +- When upgrading kubelet from one minor version to another: + + - it *may or may not be fine* + + - in some cases (e.g. migrating from Docker to containerd) it *will not* + +- Here's what [the documentation][node-upgrade-docs] says: + + *Draining nodes before upgrading kubelet ensures that pods are re-admitted and containers are re-created, which may be necessary to resolve some security issues or other important bugs.* + +- Do it at your own risk, and if you do, test extensively in staging environments! + +[node-upgrade-docs]: https://kubernetes.io/docs/tasks/administer-cluster/cluster-upgrade/#manual-deployments + +--- + class: extra-details ## Skipping versions