From abcc47b56395ee2ce765a10b83d3beba1c0f8d89 Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Wed, 28 Nov 2018 01:29:40 +0100 Subject: [PATCH 1/4] Add a section about static pods This was a request by @abuisine, so I'm flagging him for review :-) This section explains the challenges associated with self-hosting the control plane; and segues into static pods. It also mentions bootkube and the Pod Checkpointer. There is an exercise showing how to run a static pod. --- k8s/just-a-pod.yaml | 10 ++ slides/k8s/staticpods.md | 218 ++++++++++++++++++++++++++++++++++++++ slides/kube-selfpaced.yml | 1 + 3 files changed, 229 insertions(+) create mode 100644 k8s/just-a-pod.yaml create mode 100644 slides/k8s/staticpods.md diff --git a/k8s/just-a-pod.yaml b/k8s/just-a-pod.yaml new file mode 100644 index 00000000..346525de --- /dev/null +++ b/k8s/just-a-pod.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +Kind: Pod +metadata: + name: hello + namespace: default +spec: + containers: + - name: hello + image: nginx + diff --git a/slides/k8s/staticpods.md b/slides/k8s/staticpods.md new file mode 100644 index 00000000..a65f2f34 --- /dev/null +++ b/slides/k8s/staticpods.md @@ -0,0 +1,218 @@ +# Static pods + +- Pods are usually created indirectly, through another resource: + + Deployment, Daemon Set, Job, Stateful Set ... + +- They can also be created directly + +- This can be done by writing YAML and using `kubectl apply` or `kubectl create` + +- Some resources (not all of them) can be created with `kubectl run` + +- Creating a resource with `kubectl` requires the API to be up + +- If we want to run the API server (and its dependencies) on Kubernetes itself ... + + ... how can we create API pods (and other resources) when the API is not up yet? + +--- + +## In theory + +- Each component of the control plane can be replicated + +- We could set up the control plane outside of the cluster + +- Then, once the cluster is up, create replicas running on the cluster + +- Finally, remove the replicas that are running outside of the cluster + +*What could possibly go wrong?* + +--- + +## Sawing off the branch you're sitting on + +- What if anything goes wrong? + + (During the setup or at a later point) + +- Worst case scenario, we might need to: + + - set up a new control plane (outside of the cluster) + + - restore a backup from the old control plane + + - move the new control plane to the cluster (again) + +- This doesn't sound like a great experience + +--- + +## Static pods to the rescue + +- Pods are started by kubelet (an agent running on every node) + +- To know which pods it should run, the kubelet queries the API server + +- The kubelet can also get a list of *static pods* from: + + - a directory containing one (or multiple) *manifests*, and/or + + - a URL (serving a *manifest*) + +- These "manifests" are basically YAML definitions + + (As produced by `kubectl get pod my-little-pod -o yaml --export`) + +--- + +## Static pods are dynamic + +- Kubelet will periodically reload the manifests + +- It will start/stop pods accordingly + + (i.e. it is not necessary to restart the kubelet after updating the manifests) + +- When connected to the Kubernetes API, the kubelet will create *mirror pods* + +- Mirror pods are copies of the static pods + + (so they can be seen with e.g. `kubectl get pods`) + +--- + +## Bootstrapping a cluster with static pods + +- We can run control plane components with these static pods + +- They don't need the API to be up (just the kubelet) + +- Once they are up, the API becomes available + +- These pods are then visible through the API + + (We cannot upgrade them from the API, though) + +*This is how kubeadm has initialized our clusters.* + +--- + +## Static pods vs normal pods + +- The API only gives us a read-only access to static pods + +- We can `kubectl delete` a static pod ... + + ... But the kubelet will restart it immediately + +- Static pods can be selected just like other pods + + (So they can receive service traffic) + +- A service can select a mixture of static and other pods + +--- + +## From static pods to normal pods + +- Once the control plane is up and running, it can be used to create normal pods + +- We can then set up a copy of the control plane in normal pods + +- Then the static pods can be removed + +- The scheduler and the controller manager use leader election + + (Only one is active at a time; removing an instance is seamless) + +- Each instance of the API server adds itself to the `kubernetes` service + +- Etcd will typically require more work! + +--- + +## From normal pods back to static pods + +- Alright, but what if the control plane is down and we need to fix it? + +- We restart it using static pods! + +- This can be done automatically with the [Pod Checkpointer] + +- The Pod Checkpointer automatically generates manifests of running pods + +- The manifests are used to restart these pods if API contact is lost + + (More details in the [Pod Checkpointer] documentation page) + +- This technique is used by [bootkube] + +[Pod Checkpointer]: https://github.com/kubernetes-incubator/bootkube/blob/master/cmd/checkpoint/README.md +[bootkube]: https://github.com/kubernetes-incubator/bootkube + +--- + +## Static pods in action + +- On our clusters, the `staticPodPath` is `/etc/kubernetes/manifests` + +.exercise[ + +- Have a look at this directory: + ```bash + ls -l /etc/kubernetes/manifests + ``` + +] + +We should see YAML files corresponding to the pods of the control plane. + +--- + +## Running a static pod + +- We are going to add a pod manifest to the directory, and kubelet will run it + +.exercise[ + +- Copy a manifest to the directory: + ```bash + sudo cp ~/container.training/k8s/just-a-pod.yaml /etc/kubernetes/manifests + ``` + +- Check that it's running: + ```bash + kubectl get pods + ``` + +] + +The output should include a pod named `hello-node1`. + +--- + +## Remarks + +In the manifest, the pod was named `hello`. + +```yaml +apiVersion: v1 +Kind: Pod +metadata: + name: hello + namespace: default +spec: + containers: + - name: hello + image: nginx +``` + +The `-node1` suffix was added automatically by kubelet. + +If we delete the pod (with `kubectl delete`), it will be recreated immediately. + +To delete the pod, we need to delete (or move) the manifest file. + diff --git a/slides/kube-selfpaced.yml b/slides/kube-selfpaced.yml index 84da8835..5a162d5f 100644 --- a/slides/kube-selfpaced.yml +++ b/slides/kube-selfpaced.yml @@ -57,6 +57,7 @@ chapters: - - k8s/owners-and-dependents.md - k8s/statefulsets.md - k8s/portworx.md + - k8s/staticpods.md - - k8s/whatsnext.md - k8s/links.md - shared/thankyou.md From 40cd934118302d4cbec77e8bcf26ffcec8aea0bc Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Wed, 5 Dec 2018 14:25:19 -0600 Subject: [PATCH 2/4] Add a slide explaining tradeoffs between static/normal pods for control plane --- slides/k8s/staticpods.md | 23 ++++++++++++++++++++++- slides/kube-twodays.yml | 1 + 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/slides/k8s/staticpods.md b/slides/k8s/staticpods.md index a65f2f34..f1e179e3 100644 --- a/slides/k8s/staticpods.md +++ b/slides/k8s/staticpods.md @@ -155,6 +155,28 @@ --- +## Where should the control plane be? + +*Is it better to run the control plane in static pods, or normal pods?* + +- If I'm a *user* of the cluster: I don't care, it makes no difference to me + +- What if I'm an *admin*, i.e. the person who installs, upgraes, repairs... the cluster? + +- If I'm using a managed Kubernetes cluster (AKS, EKS, GKE...) it's not my problem + + (I'm not the one setting up and managing the control plane) + +- If I already picked a tool (kubeadm, kops...) to setup my cluster, the tool decides for me + +- What if I haven't picked a tool yet, or if I'm installing from scratch? + + - static pods = easier to set up, easier to troubleshoot, less risk of outage + + - normal pods = easier to upgrade, easier to move (if nodes need to be shutdown) + +--- + ## Static pods in action - On our clusters, the `staticPodPath` is `/etc/kubernetes/manifests` @@ -215,4 +237,3 @@ The `-node1` suffix was added automatically by kubelet. If we delete the pod (with `kubectl delete`), it will be recreated immediately. To delete the pod, we need to delete (or move) the manifest file. - diff --git a/slides/kube-twodays.yml b/slides/kube-twodays.yml index 0da90e1b..01eb1fa4 100644 --- a/slides/kube-twodays.yml +++ b/slides/kube-twodays.yml @@ -57,6 +57,7 @@ chapters: - - k8s/owners-and-dependents.md - k8s/statefulsets.md - k8s/portworx.md + - k8s/staticpods.md - - k8s/whatsnext.md - k8s/links.md - shared/thankyou.md From 995ea626db94d156ae589f10b1312928c8091331 Mon Sep 17 00:00:00 2001 From: Bridget Kromhout Date: Sun, 23 Dec 2018 16:07:03 -0600 Subject: [PATCH 3/4] Update staticpods.md Typo fixes --- slides/k8s/staticpods.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/slides/k8s/staticpods.md b/slides/k8s/staticpods.md index f1e179e3..09e1b0fe 100644 --- a/slides/k8s/staticpods.md +++ b/slides/k8s/staticpods.md @@ -161,19 +161,19 @@ - If I'm a *user* of the cluster: I don't care, it makes no difference to me -- What if I'm an *admin*, i.e. the person who installs, upgraes, repairs... the cluster? +- What if I'm an *admin*, i.e. the person who installs, upgrades, repairs... the cluster? - If I'm using a managed Kubernetes cluster (AKS, EKS, GKE...) it's not my problem (I'm not the one setting up and managing the control plane) -- If I already picked a tool (kubeadm, kops...) to setup my cluster, the tool decides for me +- If I already picked a tool (kubeadm, kops...) to set up my cluster, the tool decides for me - What if I haven't picked a tool yet, or if I'm installing from scratch? - static pods = easier to set up, easier to troubleshoot, less risk of outage - - normal pods = easier to upgrade, easier to move (if nodes need to be shutdown) + - normal pods = easier to upgrade, easier to move (if nodes need to be shut down) --- From 5afb37a3b9b47f822d6c3e83d99cfab34cccc2ed Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Mon, 24 Dec 2018 05:11:54 -0600 Subject: [PATCH 4/4] Updates after @bridgetkromhout's suggestions --- slides/k8s/staticpods.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/slides/k8s/staticpods.md b/slides/k8s/staticpods.md index 09e1b0fe..ae9be452 100644 --- a/slides/k8s/staticpods.md +++ b/slides/k8s/staticpods.md @@ -1,30 +1,28 @@ # Static pods -- Pods are usually created indirectly, through another resource: +- Hosting the Kubernetes control plane on Kubernetes has advantages: - Deployment, Daemon Set, Job, Stateful Set ... + - we can use Kubernetes' replication and scaling features for the control plane -- They can also be created directly + - we can leverage rolling updates to upgrade the control plane -- This can be done by writing YAML and using `kubectl apply` or `kubectl create` +- However, there is a catch: -- Some resources (not all of them) can be created with `kubectl run` + - deploying on Kubernetes requires the API to be available -- Creating a resource with `kubectl` requires the API to be up + - the API won't be available until the control plane is deployed -- If we want to run the API server (and its dependencies) on Kubernetes itself ... - - ... how can we create API pods (and other resources) when the API is not up yet? +- How can we get out of that chicken-and-egg problem? --- -## In theory +## A possible approach -- Each component of the control plane can be replicated +- Since each component of the control plane can be replicated ... - We could set up the control plane outside of the cluster -- Then, once the cluster is up, create replicas running on the cluster +- Then, once the cluster is fully operational, create replicas running on the cluster - Finally, remove the replicas that are running outside of the cluster @@ -88,9 +86,9 @@ - We can run control plane components with these static pods -- They don't need the API to be up (just the kubelet) +- They can start without requiring access to the API server -- Once they are up, the API becomes available +- Once they are up and running, the API becomes available - These pods are then visible through the API @@ -155,7 +153,7 @@ --- -## Where should the control plane be? +## Where should the control plane run? *Is it better to run the control plane in static pods, or normal pods?*