diff --git a/prepare-vms/setup-admin-clusters.sh b/prepare-vms/setup-admin-clusters.sh index b50783d4..c4635740 100755 --- a/prepare-vms/setup-admin-clusters.sh +++ b/prepare-vms/setup-admin-clusters.sh @@ -1,9 +1,24 @@ #!/bin/sh set -e +retry () { + N=$1 + I=0 + shift + + while ! "$@"; do + I=$(($I+1)) + if [ $I -gt $N ]; then + echo "FAILED, ABORTING" + exit 1 + fi + echo "FAILED, RETRYING ($I/$N)" + done +} + export AWS_INSTANCE_TYPE=t3a.small -INFRA=infra/aws-us-west-2 +INFRA=infra/aws-eu-west-3 STUDENTS=2 @@ -17,9 +32,9 @@ TAG=$PREFIX-$SETTINGS --settings settings/$SETTINGS.yaml \ --count $STUDENTS -./workshopctl deploy $TAG -./workshopctl disabledocker $TAG -./workshopctl kubebins $TAG +retry 5 ./workshopctl deploy $TAG +retry 5 ./workshopctl disabledocker $TAG +retry 5 ./workshopctl kubebins $TAG ./workshopctl cards $TAG SETTINGS=admin-kubenet @@ -30,9 +45,9 @@ TAG=$PREFIX-$SETTINGS --settings settings/$SETTINGS.yaml \ --count $((3*$STUDENTS)) -./workshopctl disableaddrchecks $TAG -./workshopctl deploy $TAG -./workshopctl kubebins $TAG +retry 5 ./workshopctl disableaddrchecks $TAG +retry 5 ./workshopctl deploy $TAG +retry 5 ./workshopctl kubebins $TAG ./workshopctl cards $TAG SETTINGS=admin-kuberouter @@ -43,9 +58,9 @@ TAG=$PREFIX-$SETTINGS --settings settings/$SETTINGS.yaml \ --count $((3*$STUDENTS)) -./workshopctl disableaddrchecks $TAG -./workshopctl deploy $TAG -./workshopctl kubebins $TAG +retry 5 ./workshopctl disableaddrchecks $TAG +retry 5 ./workshopctl deploy $TAG +retry 5 ./workshopctl kubebins $TAG ./workshopctl cards $TAG #INFRA=infra/aws-us-west-1 @@ -60,7 +75,6 @@ TAG=$PREFIX-$SETTINGS --settings settings/$SETTINGS.yaml \ --count $((3*$STUDENTS)) -./workshopctl deploy $TAG -./workshopctl kube $TAG 1.16.6 +retry 5 ./workshopctl deploy $TAG +retry 5 ./workshopctl kube $TAG 1.15.9 ./workshopctl cards $TAG - diff --git a/slides/k8s/cluster-upgrade.md b/slides/k8s/cluster-upgrade.md index 20a22a22..bfae7535 100644 --- a/slides/k8s/cluster-upgrade.md +++ b/slides/k8s/cluster-upgrade.md @@ -81,7 +81,7 @@ ## What version are we running anyway? -- When I say, "I'm running Kubernetes 1.16", is that the version of: +- When I say, "I'm running Kubernetes 1.15", is that the version of: - kubectl @@ -254,7 +254,7 @@ and kubectl, which can be one MINOR ahead or behind API server.] sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml ``` -- Look for the `image:` line, and update it to e.g. `v1.17.0` +- Look for the `image:` line, and update it to e.g. `v1.16.0` ] @@ -308,11 +308,11 @@ and kubectl, which can be one MINOR ahead or behind API server.] ] -Note 1: kubeadm thinks that our cluster is running 1.17.0. +Note 1: kubeadm thinks that our cluster is running 1.16.0.
It is confused by our manual upgrade of the API server! -Note 2: kubeadm itself is still version 1.16.6. -
It doesn't know how to upgrade do 1.17.X. +Note 2: kubeadm itself is still version 1.15.9. +
It doesn't know how to upgrade do 1.16.X. --- @@ -334,8 +334,39 @@ Note 2: kubeadm itself is still version 1.16.6. ] -Note: kubeadm still thinks that our cluster is running 1.17.0. -
But at least it knows about version 1.17.X now. +Problem: kubeadm doesn't know know how to handle +upgrades from version 1.15. + +This is because we installed version 1.17 (or even later). + +We need to install kubeadm version 1.16.X. + +--- + +## Downgrading kubeadm + +- We need to go back to version 1.16.X (e.g. 1.16.6) + +.exercise[ + +- View available versions for package `kubeadm`: + ```bash + apt show kubeadm -a | grep ^Version | grep 1.16 + ``` + +- Downgrade kubeadm: + ``` + sudo apt install kubeadm=1.16.6-00 + ``` + +- Check what kubeadm tells us: + ``` + sudo kubeadm upgrade plan + ``` + +] + +kubeadm should now agree to upgrade to 1.16.6. --- @@ -351,7 +382,7 @@ Note: kubeadm still thinks that our cluster is running 1.17.0. - Perform the upgrade: ```bash - sudo kubeadm upgrade apply v1.17.2 + sudo kubeadm upgrade apply v1.16.6 ``` ] @@ -375,7 +406,7 @@ Note: kubeadm still thinks that our cluster is running 1.17.0. - Upgrade kubelet: ```bash - sudo apt install kubelet=1.17.2-00 + sudo apt install kubelet=1.16.6-00 ``` ] @@ -423,7 +454,7 @@ Note: kubeadm still thinks that our cluster is running 1.17.0. ## Upgrading kubelet the right way -- The command that we need to run was shown by kubeadm +- We need to upgrade kubeadm, upgrade kubelet config, then upgrade kubelet (after upgrading the control plane) @@ -432,8 +463,10 @@ Note: kubeadm still thinks that our cluster is running 1.17.0. - Download the configuration on each node, and upgrade kubelet: ```bash for N in 1 2 3; do - ssh test$N sudo kubeadm upgrade node config --kubelet-version v1.17.2 - ssh test$N sudo apt install kubelet=1.17.2-00 + ssh test$N " + sudo apt install kubeadm=1.16.6-00 && + sudo kubeadm upgrade node && + sudo apt install kubelet=1.16.6-00" done ``` ] @@ -442,7 +475,7 @@ Note: kubeadm still thinks that our cluster is running 1.17.0. ## Checking what we've done -- All our nodes should now be updated to version 1.17.2 +- All our nodes should now be updated to version 1.16.6 .exercise[ @@ -459,7 +492,7 @@ class: extra-details ## Skipping versions -- This example worked because we went from 1.16 to 1.17 +- This example worked because we went from 1.15 to 1.16 - If you are upgrading from e.g. 1.14, you will have to go through 1.15 first