mirror of
https://github.com/jpetazzo/container.training.git
synced 2026-07-19 04:49:19 +00:00
Add chapter about cluster upgrades + static pods
This commit is contained in:
305
slides/k8s/cluster-upgrade.md
Normal file
305
slides/k8s/cluster-upgrade.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Upgrading clusters
|
||||
|
||||
- It's *recommended* to run consistent versions across a cluster
|
||||
|
||||
(mostly to have feature parity and latest security updates)
|
||||
|
||||
- It's not *mandatory*
|
||||
|
||||
(otherwise, cluster upgrades would be a nightmare!)
|
||||
|
||||
- Components can be upgraded one at a time without problems
|
||||
|
||||
---
|
||||
|
||||
## Checking what we're running
|
||||
|
||||
- It's easy to check the version for the API server
|
||||
|
||||
.exercise[
|
||||
|
||||
- Check the version of kubectl and of the API server:
|
||||
```bash
|
||||
kubectl version
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
- In a HA setup with multiple API servers, they can have different versions
|
||||
|
||||
- Running the command above multiple times can return different values
|
||||
|
||||
---
|
||||
|
||||
## Node versions
|
||||
|
||||
- It's also easy to check the version of kubelet
|
||||
|
||||
.exercise[
|
||||
|
||||
- Check node versions (includes kubelet, kernel, container engine):
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
- Different nodes can run different kubelet versions
|
||||
|
||||
- Different nodes can run different kernel versions
|
||||
|
||||
- Different nodes can run different container engines
|
||||
|
||||
---
|
||||
|
||||
## Control plane versions
|
||||
|
||||
- If the control plane is self-hosted (running in pods), we can check it
|
||||
|
||||
.exercise[
|
||||
|
||||
- Show image versions for all pods in `kube-system` namespace:
|
||||
```bash
|
||||
kubectl --namespace=kube-system get pods -o json \
|
||||
| jq -r '
|
||||
.items[]
|
||||
| [.spec.nodeName, .metadata.name]
|
||||
+
|
||||
(.spec.containers[].image | split(":"))
|
||||
| @tsv
|
||||
' \
|
||||
| column -t
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## What version are we running anyway?
|
||||
|
||||
- When I say, "I'm Kubernetes 1.11", is that the version of:
|
||||
|
||||
- kubectl
|
||||
|
||||
- API server
|
||||
|
||||
- kubelet
|
||||
|
||||
- controller manager
|
||||
|
||||
- something else?
|
||||
|
||||
---
|
||||
|
||||
## Other versions that are important
|
||||
|
||||
- etcd
|
||||
|
||||
- kube-dns or CoreDNS
|
||||
|
||||
- CNI plugin(s)
|
||||
|
||||
- Network controller, network policy controller
|
||||
|
||||
- Linux kernel
|
||||
|
||||
---
|
||||
|
||||
## General guidelines
|
||||
|
||||
- To update a component, use whatever was used to install it
|
||||
|
||||
- If it's a distro package, update that distro package
|
||||
|
||||
- If it's a container or pod, update that container or pod
|
||||
|
||||
- If you used configuration management, update with that
|
||||
|
||||
---
|
||||
|
||||
## Know where your binaries come from
|
||||
|
||||
- Sometimes, we need to upgrade *quickly*
|
||||
|
||||
(when a vulnerability is announced and patched)
|
||||
|
||||
- If we are using an installer, we should:
|
||||
|
||||
- make sure it's using upstream packages
|
||||
|
||||
- or make sure that whatever packages it uses are current
|
||||
|
||||
- make sure we can tell it to pin specific component versions
|
||||
|
||||
---
|
||||
|
||||
## In practice
|
||||
|
||||
- We are going to update a few cluster components
|
||||
|
||||
- We will change the kubelet version on one node
|
||||
|
||||
- We will change the version of the API server
|
||||
|
||||
- We will work with cluster `test` (nodes `test1`, `test2`, `test3`)
|
||||
|
||||
---
|
||||
|
||||
## Updating kubelet
|
||||
|
||||
- These nodes have been installed using the official Kubernetes packages
|
||||
|
||||
- We can therefore use `apt` or `apt-get`
|
||||
|
||||
.exercise[
|
||||
|
||||
- Log into node `test3`
|
||||
|
||||
- View available versions for package `kubelet`:
|
||||
```bash
|
||||
apt show kubelet -a | grep ^Version
|
||||
```
|
||||
|
||||
- Upgrade kubelet:
|
||||
```bash
|
||||
apt install kubelet=1.14.1-00
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Checking what we've done
|
||||
|
||||
.exercise[
|
||||
|
||||
- Log into node `test1`
|
||||
|
||||
- Check node versions:
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
- Create a deployment and scale it to make sure that the node still works
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Updating the API server
|
||||
|
||||
- This cluster has been deployed with kubeadm
|
||||
|
||||
- The control plane runs in *static pods*
|
||||
|
||||
- These pods are started automatically by kubelet
|
||||
|
||||
(even when kubelet can't contact the API server)
|
||||
|
||||
- They are defined in YAML files in `/etc/kubernetes/manifests`
|
||||
|
||||
(this path is set by a kubelet command-line flag)
|
||||
|
||||
- kubelet automatically updates the pods when the files are changed
|
||||
|
||||
---
|
||||
|
||||
## Changing the API server version
|
||||
|
||||
- We will edit the YAML file to use a different image version
|
||||
|
||||
.exercise[
|
||||
|
||||
- Log into node `test1`
|
||||
|
||||
- Check API server version:
|
||||
```bash
|
||||
kubectl version
|
||||
```
|
||||
|
||||
- Edit the API server pod manifest:
|
||||
```bash
|
||||
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
|
||||
```
|
||||
|
||||
- Look for the `image:` line, and update it to e.g. `v1.14.0`
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Checking what we've done
|
||||
|
||||
- The API server will be shortly unavailable while kubelet restarts it
|
||||
|
||||
.exercise[
|
||||
|
||||
- Check the API server version:
|
||||
```bash
|
||||
kubectl version
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Updating the whole control plane
|
||||
|
||||
- As an example, we'll use kubeadm to upgrade the entire control plane
|
||||
|
||||
(note: this is possible only because the cluster was installed with kubeadm)
|
||||
|
||||
.exercise[
|
||||
|
||||
- Check what will be upgraded:
|
||||
```bash
|
||||
sudo kubeadm upgrade plan
|
||||
```
|
||||
|
||||
(Note: kubeadm is confused by our manual upgrade of the API server.
|
||||
<br/>It thinks the cluster is running 1.14.0!)
|
||||
|
||||
<!-- ##VERSION## -->
|
||||
|
||||
- Perform the upgrade:
|
||||
```bash
|
||||
sudo kubeadm upgrade apply v1.14.1
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Updating kubelets
|
||||
|
||||
- After updating the control plane, we need to update each kubelet
|
||||
|
||||
- This requires to run a special command on each node, to download the config
|
||||
|
||||
(this config is generated by kubeadm)
|
||||
|
||||
.exercise[
|
||||
|
||||
- Download the configuration on each node, and upgrade kubelet:
|
||||
```bash
|
||||
for N in 1 2 3; do
|
||||
ssh node$N sudo kubeadm upgrade node config --kubelet-version v1.14.1
|
||||
ssh node $N sudo apt install kubelet=1.14.1-00
|
||||
done
|
||||
```
|
||||
]
|
||||
|
||||
---
|
||||
|
||||
## Checking what we've done
|
||||
|
||||
- All our nodes should now be updated to version 1.14.1
|
||||
|
||||
.exercise[
|
||||
|
||||
- Check nodes versions:
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
]
|
||||
@@ -192,6 +192,8 @@ We should see YAML files corresponding to the pods of the control plane.
|
||||
|
||||
---
|
||||
|
||||
class: static-pods-exercise
|
||||
|
||||
## Running a static pod
|
||||
|
||||
- We are going to add a pod manifest to the directory, and kubelet will run it
|
||||
@@ -214,6 +216,8 @@ The output should include a pod named `hello-node1`.
|
||||
|
||||
---
|
||||
|
||||
class: static-pods-exercise
|
||||
|
||||
## Remarks
|
||||
|
||||
In the manifest, the pod was named `hello`.
|
||||
|
||||
@@ -12,6 +12,7 @@ slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
- static-pods-exercise
|
||||
|
||||
chapters:
|
||||
- shared/title.md
|
||||
@@ -30,7 +31,7 @@ chapters:
|
||||
- - k8s/setup-managed.md
|
||||
- k8s/setup-selfhosted.md
|
||||
- k8s/cluster-upgrade.md
|
||||
- k8s/cluster-configuration.md
|
||||
- k8s/staticpods.md
|
||||
- k8s/cluster-backup.md
|
||||
- k8s/cloud-controller-manager.md
|
||||
- k8s/bootstrap.md
|
||||
|
||||
Reference in New Issue
Block a user