diff --git a/slides/k8s/staticpods.md b/slides/k8s/staticpods.md index d572bea9..6dd7690c 100644 --- a/slides/k8s/staticpods.md +++ b/slides/k8s/staticpods.md @@ -1,147 +1,109 @@ # Static pods -- Hosting the Kubernetes control plane on Kubernetes has advantages: +Question: can we host the control plane of a cluster *on the cluster itself?* - - we can use Kubernetes' replication and scaling features for the control plane +- To create a Pod, we need to communicate with the API server - - we can leverage rolling updates to upgrade the control plane +- The API server needs etcd to be up -- However, there is a catch: +- Then the Pod needs to be bound to a node by the scheduler - - deploying on Kubernetes requires the API to be available +- So... all these things need to be running already! - - the API won't be available until the control plane is deployed +- Even if the Pod already exists, we still need API server and etcd -- How can we get out of that chicken-and-egg problem? + (so that kubelet can connect to the API server and "know" about the Pod) --- -## A possible approach +## Static pods -- Since each component of the control plane can be replicated... +Solution: run (parts of) the control plane in *static pods!* -- We could set up the control plane outside of the cluster +- Normally, kubelet queries the API server to know what pods to run -- Then, once the cluster is fully operational, create replicas running on the cluster +- Additionally, we can tell kubelet to run pods: -- Finally, remove the replicas that are running outside of the cluster + - by storing manifests in a directory (`--pod-manifest-path`) -*What could possibly go wrong?* + - by retrieving manifests from an HTTP server (`--manifest-url`) + +- These manifests should be normal pod manifests + + (make sure to include the namespace in the metadata block!) + +- kubelet will append the node name after the pod name --- -## Sawing off the branch you're sitting on +## How and when kubelet runs static pods -- What if anything goes wrong? +- kubelet runs static pods "no matter what" - (During the setup or at a later point) + (even if it can't connect to the API server, or if no API server is configured) -- Worst case scenario, we might need to: +- When there is no API server configuration, that's called "standalone mode" - - set up a new control plane (outside of the cluster) +- Almost nothing can prevent kubelet from running these pods - - restore a backup from the old control plane + (e.g. admission controllers, pod security settings... won't apply) - - move the new control plane to the cluster (again) +- kubelet monitors the manifest path (and/or the manifest URL) -- This doesn't sound like a great experience +- If manifests are deleted: their pods are destroyed + +- If manifests are modified: their pods are destroyed and recreated --- -## Static pods to the rescue +## Mirror pods -- Pods are started by kubelet (an agent running on every node) +- Static pods remain running even after API server connection is up -- To know which pods it should run, the kubelet queries the API server +- Once the API server is up, kubelet will create *mirror pods* -- The kubelet can also get a list of *static pods* from: +- Mirror pods represent the static pods that are running - - a directory containing one (or multiple) *manifests*, and/or +.warning[Deleting a mirror pod has no effect on the static pod!] - - a URL (serving a *manifest*) +- kubelet will immediately recreate the mirror pod if it is deleted -- These "manifests" are basically YAML definitions +.warning[Admission control can block the mirror pod, but not the static pod!] - (As produced by `kubectl get pod my-little-pod -o yaml`) +- Since kubelet runs the static pod even if there is no connection to the API server --- -## Static pods are dynamic +## Example -- Kubelet will periodically reload the manifests +- `kubeadm` leverages static pods to run the control plane -- It will start/stop pods accordingly + (etcd, API server, controller manager, scheduler) - (i.e. it is not necessary to restart the kubelet after updating the manifests) +- It "renders" a number of YAML manifests to `/etc/kubernetes/manifests` -- When connected to the Kubernetes API, the kubelet will create *mirror pods* +- This is the cluster boot sequence: -- Mirror pods are copies of the static pods + - machine boots - (so they can be seen with e.g. `kubectl get pods`) + - kubelet is started (typically by systemd) + + - kubelet reads static pod manifests and run them + + - control plane is up, yay! --- -## Bootstrapping a cluster with static pods +class: extra-details -- We can run control plane components with these static pods +## Pod checkpointer -- They can start without requiring access to the API server - -- Once they are up and running, the API becomes available - -- These pods are then visible through the API - - (We cannot upgrade them from the API, though) - -*This is how kubeadm has initialized our clusters.* - ---- - -## Static pods vs normal pods - -- The API only gives us read-only access to static pods - -- We can `kubectl delete` a static pod... - - ...But the kubelet will re-mirror it immediately - -- Static pods can be selected just like other pods - - (So they can receive service traffic) - -- A service can select a mixture of static and other pods - ---- - -## From static pods to normal pods - -- Once the control plane is up and running, it can be used to create normal pods - -- We can then set up a copy of the control plane in normal pods - -- Then the static pods can be removed - -- The scheduler and the controller manager use leader election - - (Only one is active at a time; removing an instance is seamless) - -- Each instance of the API server adds itself to the `kubernetes` service - -- Etcd will typically require more work! - ---- - -## From normal pods back to static pods - -- Alright, but what if the control plane is down and we need to fix it? - -- We restart it using static pods! - -- This can be done automatically with a “pod checkpointer” +- This pattern isn't used anymore, but perhaps it can provide inspiration - The pod checkpointer automatically generates manifests of running pods + (if they have specific labels/annotations) + - The manifests are used to restart these pods if API contact is lost - This pattern is implemented in [openshift/pod-checkpointer-operator] and [bootkube checkpointer] @@ -151,95 +113,6 @@ [openshift/pod-checkpointer-operator]: https://github.com/openshift/pod-checkpointer-operator [bootkube checkpointer]: https://github.com/kubernetes-retired/bootkube/blob/master/cmd/checkpoint/README.md ---- - -## Where should the control plane run? - -*Is it better to run the control plane in static pods, or normal pods?* - -- If I'm a *user* of the cluster: I don't care, it makes no difference to me - -- What if I'm an *admin*, i.e. the person who installs, upgrades, repairs... the cluster? - -- If I'm using a managed Kubernetes cluster (AKS, EKS, GKE...) it's not my problem - - (I'm not the one setting up and managing the control plane) - -- If I already picked a tool (kubeadm, kops...) to set up my cluster, the tool decides for me - -- What if I haven't picked a tool yet, or if I'm installing from scratch? - - - static pods = easier to set up, easier to troubleshoot, less risk of outage - - - normal pods = easier to upgrade, easier to move (if nodes need to be shut down) - ---- - -## Static pods in action - -- On our clusters, the `staticPodPath` is `/etc/kubernetes/manifests` - -.lab[ - -- Have a look at this directory: - ```bash - ls -l /etc/kubernetes/manifests - ``` - -] - -We should see YAML files corresponding to the pods of the control plane. - ---- - -class: static-pods-exercise - -## Running a static pod - -- We are going to add a pod manifest to the directory, and kubelet will run it - -.lab[ - -- Copy a manifest to the directory: - ```bash - sudo cp ~/container.training/k8s/just-a-pod.yaml /etc/kubernetes/manifests - ``` - -- Check that it's running: - ```bash - kubectl get pods - ``` - -] - -The output should include a pod named `hello-node1`. - ---- - -class: static-pods-exercise - -## Remarks - -In the manifest, the pod was named `hello`. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: hello - namespace: default -spec: - containers: - - name: hello - image: nginx -``` - -The `-node1` suffix was added automatically by kubelet. - -If we delete the pod (with `kubectl delete`), it will be recreated immediately. - -To delete the pod, we need to delete (or move) the manifest file. - ??? :EN:- Static pods