diff --git a/slides/k8s/alias-and-references.md b/slides/k8s/alias-and-references.md index cad46ad8..21179500 100644 --- a/slides/k8s/alias-and-references.md +++ b/slides/k8s/alias-and-references.md @@ -2,7 +2,7 @@ # External References && kubectl Aliases -Class Slides: https://2022-07-proofpoint.container.training/ +Class Slides: https://2022-08-nr.container.training/ Kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ diff --git a/slides/kube.yml b/slides/kube.yml index 97902919..f65deee6 100644 --- a/slides/kube.yml +++ b/slides/kube.yml @@ -66,17 +66,16 @@ content: - k8s/namespaces.md - k8s/healthchecks.md - k8s/kubenet.md - - k8s/netpol.md - - exercises/ingress-details.md - containers/software-deployment.md - k8s/volumes.md - k8s/configuration.md - k8s/secrets.md - - k8s/ingress.md - exercises/healthchecks-details.md - exercises/appconfig-details.md - # DAY 4 + - k8s/netpol.md + - k8s/ingress.md - k8s/statefulsets.md - k8s/pv-pvc-sc.md - k8s/volume-claim-templates.md @@ -84,10 +83,11 @@ content: - k8s/setup-devel.md - k8s/localkubeconfig.md - k8s/accessinternal.md - - k8s/kubectlproxy.md + - exercises/ingress-details.md - exercises/localcluster-details.md - # DAY 5 + - k8s/kubectlproxy.md - k8s/authn-authz.md - k8s/resource-limits.md - k8s/metrics-server.md diff --git a/slides/out.html b/slides/out.html index 028f4692..7cac5ec0 100644 --- a/slides/out.html +++ b/slides/out.html @@ -29,27 +29,13 @@ Kubernetes
Intermediate
Training
.debug[ ``` - M slides/containers/Getting_Inside.md + M slides/k8s/alias-and-references.md M slides/kube.yml - M slides/shared/prereqs.md -?? slides/b -?? slides/containers/High_Level_View.md -?? slides/containers/Macro_View.md -?? slides/containers/Training_Environment_And_Tmux.md -?? slides/containers/Understanding_Images.md -?? slides/containers/exploring-images/ -?? slides/containers/software-deployment.md -?? slides/containers/software-deployment/ -?? slides/k8s/alias-and-references.md -?? slides/k8s/kubectl-first.md -?? slides/k8s/kubectl-more.md -?? slides/kube.orig.yml -?? slides/logistics-gerry.md -?? slides/out.html + M slides/out.html ``` -These slides have been built from commit: b46dcd51 +These slides have been built from commit: d83a6232 [shared/title.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/shared/title.md)] @@ -706,8 +692,6 @@ name: toc-part-1 - [A Macroscopic View](#toc-a-macroscopic-view) -- [Container Based Software Deployment](#toc-container-based-software-deployment) - - [Kubernetes concepts](#toc-kubernetes-concepts) - [Declarative vs imperative](#toc-declarative-vs-imperative) @@ -725,16 +709,22 @@ name: toc-part-2 ## Part 2 -- [Revisiting `kubectl logs`](#toc-revisiting-kubectl-logs) -- [Accessing logs from the CLI](#toc-accessing-logs-from-the-cli) - [Labels and annotations](#toc-labels-and-annotations) + +- [Revisiting `kubectl logs`](#toc-revisiting-kubectl-logs) + +- [Accessing logs from the CLI](#toc-accessing-logs-from-the-cli) + - [Running our first containers on Kubernetes](#toc-running-our-first-containers-on-kubernetes) + - [More contact with `kubectl`](#toc-more-contact-with-kubectl) + +- [Exposing containers](#toc-exposing-containers) + - [Rolling updates](#toc-rolling-updates) + - [Deploying with YAML](#toc-deploying-with-yaml) -- [Namespaces](#toc-namespaces) -- [Daemon sets](#toc-daemon-sets) -- [Labels and selectors](#toc-labels-and-selectors) + .debug[(auto-generated TOC)] --- @@ -742,14 +732,15 @@ name: toc-part-3 ## Part 3 +- [Daemon sets](#toc-daemon-sets) +- [Labels and selectors](#toc-labels-and-selectors) +- [Namespaces](#toc-namespaces) - [Healthchecks](#toc-healthchecks) - [Kubernetes network model](#toc-kubernetes-network-model) -- [Exposing containers](#toc-exposing-containers) +- [Container Based Software Deployment](#toc-container-based-software-deployment) - [Volumes](#toc-volumes) - [Managing configuration](#toc-managing-configuration) - [Managing secrets](#toc-managing-secrets) -- [Exposing HTTP services with Ingress resources](#toc-exposing-http-services-with-ingress-resources) -- [Exercise — Ingress](#toc-exercise--ingress) - [Exercise — Healthchecks](#toc-exercise--healthchecks) - [Exercise — Application Configuration](#toc-exercise--application-configuration) @@ -759,30 +750,24 @@ name: toc-part-4 ## Part 4 +- [Network policies](#toc-network-policies) +- [Exposing HTTP services with Ingress resources](#toc-exposing-http-services-with-ingress-resources) - [Stateful sets](#toc-stateful-sets) - - [PV, PVC, and Storage Classes](#toc-pv-pvc-and-storage-classes) - - [Setting up Kubernetes](#toc-setting-up-kubernetes) - - [Running a local development cluster](#toc-running-a-local-development-cluster) - - [Controlling a Kubernetes cluster remotely](#toc-controlling-a-kubernetes-cluster-remotely) - - [Accessing internal services](#toc-accessing-internal-services) - -- [Accessing the API with `kubectl proxy`](#toc-accessing-the-api-with-kubectl-proxy) - +- [Exercise — Ingress](#toc-exercise--ingress) - [Exercise — Local Cluster](#toc-exercise--local-cluster) - .debug[(auto-generated TOC)] --- name: toc-part-5 ## Part 5 -- [Network policies](#toc-network-policies) +- [Accessing the API with `kubectl proxy`](#toc-accessing-the-api-with-kubectl-proxy) - [Authentication and authorization](#toc-authentication-and-authorization) - [Resource Limits](#toc-resource-limits) - [Defining min, max, and default resources](#toc-defining-min-max-and-default-resources) @@ -915,7 +900,7 @@ class: title | [Back to table of contents](#toc-part-1) | -[Next part](#toc-container-based-software-deployment) +[Next part](#toc-kubernetes-concepts) ] .debug[(automatically generated title slide)] @@ -1266,176 +1251,13 @@ class: pic --- -name: toc-container-based-software-deployment -class: title - - Container Based Software Deployment - -.nav[ -[Previous part](#toc-a-macroscopic-view) -| -[Back to table of contents](#toc-part-1) -| -[Next part](#toc-kubernetes-concepts) -] - -.debug[(automatically generated title slide)] - ---- -# Container Based Software Deployment - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-1.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-2.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-3.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-4.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-5.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-6.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-7.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-8.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-9.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-10.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-11.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-12.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-13.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-14.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-15.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-16.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - - -![dummmy](containers/software-deployment/slide-17.jpg) - -.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/aerial-view-of-containers.jpg)] - ---- - name: toc-kubernetes-concepts class: title Kubernetes concepts .nav[ -[Previous part](#toc-container-based-software-deployment) +[Previous part](#toc-a-macroscopic-view) | [Back to table of contents](#toc-part-1) | @@ -2014,7 +1836,7 @@ class: pic class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/blue-containers.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/aerial-view-of-containers.jpg)] --- @@ -2142,7 +1964,7 @@ specifying how to brew tea?] class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/blue-containers.jpg)] --- @@ -2430,7 +2252,7 @@ class: extra-details class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] --- @@ -2748,7 +2570,7 @@ Note: we don't need to specify `--validate=false` here. class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-housing.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] --- @@ -2762,7 +2584,7 @@ class: title | [Back to table of contents](#toc-part-1) | -[Next part](#toc-revisiting-kubectl-logs) +[Next part](#toc-labels-and-annotations) ] .debug[(automatically generated title slide)] @@ -2772,7 +2594,7 @@ class: title # External References && kubectl Aliases -Class Slides: https://2022-07-proofpoint.container.training/ +Class Slides: https://2022-08-nr.container.training/ Kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ @@ -2820,401 +2642,7 @@ sudo mv kx /usr/local/bin/kx class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/containers-by-the-water.jpg)] - ---- - -name: toc-revisiting-kubectl-logs -class: title - - Revisiting `kubectl logs` - -.nav[ -[Previous part](#toc-external-references--kubectl-aliases) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-accessing-logs-from-the-cli) -] - -.debug[(automatically generated title slide)] - ---- -# Revisiting `kubectl logs` - -- In this section, we assume that we have a Deployment with multiple Pods - - (e.g. `pingpong` that we scaled to at least 3 pods) - -- We will highlights some of the limitations of `kubectl logs` - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -## Streaming logs of multiple pods - -- By default, `kubectl logs` shows us the output of a single Pod - -.lab[ - -- Try to check the output of the Pods related to a Deployment: - ```bash - kubectl logs deploy/pingpong --tail 1 --follow - ``` - - - -] - -`kubectl logs` only shows us the logs of one of the Pods. - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -## Viewing logs of multiple pods - -- When we specify a deployment name, only one single pod's logs are shown - -- We can view the logs of multiple pods by specifying a *selector* - -- If we check the pods created by the deployment, they all have the label `app=pingpong` - - (this is just a default label that gets added when using `kubectl create deployment`) - -.lab[ - -- View the last line of log from all pods with the `app=pingpong` label: - ```bash - kubectl logs -l app=pingpong --tail 1 - ``` - -] - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -## Streaming logs of multiple pods - -- Can we stream the logs of all our `pingpong` pods? - -.lab[ - -- Combine `-l` and `-f` flags: - ```bash - kubectl logs -l app=pingpong --tail 1 -f - ``` - - - -] - -*Note: combining `-l` and `-f` is only possible since Kubernetes 1.14!* - -*Let's try to understand why ...* - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -class: extra-details - -## Streaming logs of many pods - -- Let's see what happens if we try to stream the logs for more than 5 pods - -.lab[ - -- Scale up our deployment: - ```bash - kubectl scale deployment pingpong --replicas=8 - ``` - -- Stream the logs: - ```bash - kubectl logs -l app=pingpong --tail 1 -f - ``` - - - -] - -We see a message like the following one: -``` -error: you are attempting to follow 8 log streams, -but maximum allowed concurency is 5, -use --max-log-requests to increase the limit -``` - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -class: extra-details - -## Why can't we stream the logs of many pods? - -- `kubectl` opens one connection to the API server per pod - -- For each pod, the API server opens one extra connection to the corresponding kubelet - -- If there are 1000 pods in our deployment, that's 1000 inbound + 1000 outbound connections on the API server - -- This could easily put a lot of stress on the API server - -- Prior Kubernetes 1.14, it was decided to *not* allow multiple connections - -- From Kubernetes 1.14, it is allowed, but limited to 5 connections - - (this can be changed with `--max-log-requests`) - -- For more details about the rationale, see - [PR #67573](https://github.com/kubernetes/kubernetes/pull/67573) - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -## Shortcomings of `kubectl logs` - -- We don't see which pod sent which log line - -- If pods are restarted / replaced, the log stream stops - -- If new pods are added, we don't see their logs - -- To stream the logs of multiple pods, we need to write a selector - -- There are external tools to address these shortcomings - - (e.g.: [Stern](https://github.com/wercker/stern)) - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -class: extra-details - -## `kubectl logs -l ... --tail N` - -- If we run this with Kubernetes 1.12, the last command shows multiple lines - -- This is a regression when `--tail` is used together with `-l`/`--selector` - -- It always shows the last 10 lines of output for each container - - (instead of the number of lines specified on the command line) - -- The problem was fixed in Kubernetes 1.13 - -*See [#70554](https://github.com/kubernetes/kubernetes/issues/70554) for details.* - -??? - -:EN:- Viewing logs with "kubectl logs" -:FR:- Consulter les logs avec "kubectl logs" - -.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/distillery-containers.jpg)] - ---- - -name: toc-accessing-logs-from-the-cli -class: title - - Accessing logs from the CLI - -.nav[ -[Previous part](#toc-revisiting-kubectl-logs) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-labels-and-annotations) -] - -.debug[(automatically generated title slide)] - ---- -# Accessing logs from the CLI - -- The `kubectl logs` command has limitations: - - - it cannot stream logs from multiple pods at a time - - - when showing logs from multiple pods, it mixes them all together - -- We are going to see how to do it better - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Doing it manually - -- We *could* (if we were so inclined) write a program or script that would: - - - take a selector as an argument - - - enumerate all pods matching that selector (with `kubectl get -l ...`) - - - fork one `kubectl logs --follow ...` command per container - - - annotate the logs (the output of each `kubectl logs ...` process) with their origin - - - preserve ordering by using `kubectl logs --timestamps ...` and merge the output - --- - -- We *could* do it, but thankfully, others did it for us already! - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Stern - -[Stern](https://github.com/stern/stern) is an open source project -originally by [Wercker](http://www.wercker.com/). - -From the README: - -*Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.* - -*The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.* - -Exactly what we need! - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Checking if Stern is installed - -- Run `stern` (without arguments) to check if it's installed: - - ``` - $ stern - Tail multiple pods and containers from Kubernetes - - Usage: - stern pod-query [flags] - ``` - -- If it's missing, let's see how to install it - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Installing Stern - -- Stern is written in Go - -- Go programs are usually very easy to install - - (no dependencies, extra libraries to install, etc) - -- Binary releases are available [here](https://github.com/stern/stern/releases) on GitHub - -- Stern is also available through most package managers - - (e.g. on macOS, we can `brew install stern` or `sudo port install stern`) - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Using Stern - -- There are two ways to specify the pods whose logs we want to see: - - - `-l` followed by a selector expression (like with many `kubectl` commands) - - - with a "pod query," i.e. a regex used to match pod names - -- These two ways can be combined if necessary - -.lab[ - -- View the logs for all the pingpong containers: - ```bash - stern pingpong - ``` - - - -] - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Stern convenient options - -- The `--tail N` flag shows the last `N` lines for each container - - (Instead of showing the logs since the creation of the container) - -- The `-t` / `--timestamps` flag shows timestamps - -- The `--all-namespaces` flag is self-explanatory - -.lab[ - -- View what's up with the `weave` system containers: - ```bash - stern --tail 1 --timestamps --all-namespaces weave - ``` - - - -] - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -## Using Stern with a selector - -- When specifying a selector, we can omit the value for a label - -- This will match all objects having that label (regardless of the value) - -- Everything created with `kubectl run` has a label `run` - -- Everything created with `kubectl create deployment` has a label `app` - -- We can use that property to view the logs of all the pods created with `kubectl create deployment` - -.lab[ - -- View the logs for all the things started with `kubectl create deployment`: - ```bash - stern -l app - ``` - - - -] - -??? - -:EN:- Viewing pod logs from the CLI -:FR:- Consulter les logs des pods depuis la CLI - -.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/lots-of-containers.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-housing.jpg)] --- @@ -3224,11 +2652,11 @@ class: title Labels and annotations .nav[ -[Previous part](#toc-accessing-logs-from-the-cli) +[Previous part](#toc-external-references--kubectl-aliases) | [Back to table of contents](#toc-part-2) | -[Next part](#toc-running-our-first-containers-on-kubernetes) +[Next part](#toc-revisiting-kubectl-logs) ] .debug[(automatically generated title slide)] @@ -3476,7 +2904,401 @@ class: extra-details class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/containers-by-the-water.jpg)] + +--- + +name: toc-revisiting-kubectl-logs +class: title + + Revisiting `kubectl logs` + +.nav[ +[Previous part](#toc-labels-and-annotations) +| +[Back to table of contents](#toc-part-2) +| +[Next part](#toc-accessing-logs-from-the-cli) +] + +.debug[(automatically generated title slide)] + +--- +# Revisiting `kubectl logs` + +- In this section, we assume that we have a Deployment with multiple Pods + + (e.g. `pingpong` that we scaled to at least 3 pods) + +- We will highlights some of the limitations of `kubectl logs` + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +## Streaming logs of multiple pods + +- By default, `kubectl logs` shows us the output of a single Pod + +.lab[ + +- Try to check the output of the Pods related to a Deployment: + ```bash + kubectl logs deploy/pingpong --tail 1 --follow + ``` + + + +] + +`kubectl logs` only shows us the logs of one of the Pods. + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +## Viewing logs of multiple pods + +- When we specify a deployment name, only one single pod's logs are shown + +- We can view the logs of multiple pods by specifying a *selector* + +- If we check the pods created by the deployment, they all have the label `app=pingpong` + + (this is just a default label that gets added when using `kubectl create deployment`) + +.lab[ + +- View the last line of log from all pods with the `app=pingpong` label: + ```bash + kubectl logs -l app=pingpong --tail 1 + ``` + +] + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +## Streaming logs of multiple pods + +- Can we stream the logs of all our `pingpong` pods? + +.lab[ + +- Combine `-l` and `-f` flags: + ```bash + kubectl logs -l app=pingpong --tail 1 -f + ``` + + + +] + +*Note: combining `-l` and `-f` is only possible since Kubernetes 1.14!* + +*Let's try to understand why ...* + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +class: extra-details + +## Streaming logs of many pods + +- Let's see what happens if we try to stream the logs for more than 5 pods + +.lab[ + +- Scale up our deployment: + ```bash + kubectl scale deployment pingpong --replicas=8 + ``` + +- Stream the logs: + ```bash + kubectl logs -l app=pingpong --tail 1 -f + ``` + + + +] + +We see a message like the following one: +``` +error: you are attempting to follow 8 log streams, +but maximum allowed concurency is 5, +use --max-log-requests to increase the limit +``` + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +class: extra-details + +## Why can't we stream the logs of many pods? + +- `kubectl` opens one connection to the API server per pod + +- For each pod, the API server opens one extra connection to the corresponding kubelet + +- If there are 1000 pods in our deployment, that's 1000 inbound + 1000 outbound connections on the API server + +- This could easily put a lot of stress on the API server + +- Prior Kubernetes 1.14, it was decided to *not* allow multiple connections + +- From Kubernetes 1.14, it is allowed, but limited to 5 connections + + (this can be changed with `--max-log-requests`) + +- For more details about the rationale, see + [PR #67573](https://github.com/kubernetes/kubernetes/pull/67573) + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +## Shortcomings of `kubectl logs` + +- We don't see which pod sent which log line + +- If pods are restarted / replaced, the log stream stops + +- If new pods are added, we don't see their logs + +- To stream the logs of multiple pods, we need to write a selector + +- There are external tools to address these shortcomings + + (e.g.: [Stern](https://github.com/wercker/stern)) + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +class: extra-details + +## `kubectl logs -l ... --tail N` + +- If we run this with Kubernetes 1.12, the last command shows multiple lines + +- This is a regression when `--tail` is used together with `-l`/`--selector` + +- It always shows the last 10 lines of output for each container + + (instead of the number of lines specified on the command line) + +- The problem was fixed in Kubernetes 1.13 + +*See [#70554](https://github.com/kubernetes/kubernetes/issues/70554) for details.* + +??? + +:EN:- Viewing logs with "kubectl logs" +:FR:- Consulter les logs avec "kubectl logs" + +.debug[[k8s/kubectl-logs.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubectl-logs.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/distillery-containers.jpg)] + +--- + +name: toc-accessing-logs-from-the-cli +class: title + + Accessing logs from the CLI + +.nav[ +[Previous part](#toc-revisiting-kubectl-logs) +| +[Back to table of contents](#toc-part-2) +| +[Next part](#toc-running-our-first-containers-on-kubernetes) +] + +.debug[(automatically generated title slide)] + +--- +# Accessing logs from the CLI + +- The `kubectl logs` command has limitations: + + - it cannot stream logs from multiple pods at a time + + - when showing logs from multiple pods, it mixes them all together + +- We are going to see how to do it better + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Doing it manually + +- We *could* (if we were so inclined) write a program or script that would: + + - take a selector as an argument + + - enumerate all pods matching that selector (with `kubectl get -l ...`) + + - fork one `kubectl logs --follow ...` command per container + + - annotate the logs (the output of each `kubectl logs ...` process) with their origin + + - preserve ordering by using `kubectl logs --timestamps ...` and merge the output + +-- + +- We *could* do it, but thankfully, others did it for us already! + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Stern + +[Stern](https://github.com/stern/stern) is an open source project +originally by [Wercker](http://www.wercker.com/). + +From the README: + +*Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.* + +*The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.* + +Exactly what we need! + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Checking if Stern is installed + +- Run `stern` (without arguments) to check if it's installed: + + ``` + $ stern + Tail multiple pods and containers from Kubernetes + + Usage: + stern pod-query [flags] + ``` + +- If it's missing, let's see how to install it + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Installing Stern + +- Stern is written in Go + +- Go programs are usually very easy to install + + (no dependencies, extra libraries to install, etc) + +- Binary releases are available [here](https://github.com/stern/stern/releases) on GitHub + +- Stern is also available through most package managers + + (e.g. on macOS, we can `brew install stern` or `sudo port install stern`) + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Using Stern + +- There are two ways to specify the pods whose logs we want to see: + + - `-l` followed by a selector expression (like with many `kubectl` commands) + + - with a "pod query," i.e. a regex used to match pod names + +- These two ways can be combined if necessary + +.lab[ + +- View the logs for all the pingpong containers: + ```bash + stern pingpong + ``` + + + +] + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Stern convenient options + +- The `--tail N` flag shows the last `N` lines for each container + + (Instead of showing the logs since the creation of the container) + +- The `-t` / `--timestamps` flag shows timestamps + +- The `--all-namespaces` flag is self-explanatory + +.lab[ + +- View what's up with the `weave` system containers: + ```bash + stern --tail 1 --timestamps --all-namespaces weave + ``` + + + +] + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +## Using Stern with a selector + +- When specifying a selector, we can omit the value for a label + +- This will match all objects having that label (regardless of the value) + +- Everything created with `kubectl run` has a label `run` + +- Everything created with `kubectl create deployment` has a label `app` + +- We can use that property to view the logs of all the pods created with `kubectl create deployment` + +.lab[ + +- View the logs for all the things started with `kubectl create deployment`: + ```bash + stern -l app + ``` + + + +] + +??? + +:EN:- Viewing pod logs from the CLI +:FR:- Consulter les logs des pods depuis la CLI + +.debug[[k8s/logs-cli.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/logs-cli.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/lots-of-containers.jpg)] --- @@ -3486,7 +3308,7 @@ class: title Running our first containers on Kubernetes .nav[ -[Previous part](#toc-labels-and-annotations) +[Previous part](#toc-accessing-logs-from-the-cli) | [Back to table of contents](#toc-part-2) | @@ -4037,7 +3859,7 @@ class: extra-details class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] --- @@ -4051,7 +3873,7 @@ class: title | [Back to table of contents](#toc-part-2) | -[Next part](#toc-rolling-updates) +[Next part](#toc-exposing-containers) ] .debug[(automatically generated title slide)] @@ -4419,2581 +4241,7 @@ $ curl -k https://10.96.0.1 class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-2.jpg)] - ---- - -name: toc-rolling-updates -class: title - - Rolling updates - -.nav[ -[Previous part](#toc-more-contact-with-kubectl) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-deploying-with-yaml) -] - -.debug[(automatically generated title slide)] - ---- -# Rolling updates - -- By default (without rolling updates), when a scaled resource is updated: - - - new pods are created - - - old pods are terminated - - - ... all at the same time - - - if something goes wrong, ¯\\\_(ツ)\_/¯ - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling updates - -- With rolling updates, when a Deployment is updated, it happens progressively - -- The Deployment controls multiple Replica Sets - -- Each Replica Set is a group of identical Pods - - (with the same image, arguments, parameters ...) - -- During the rolling update, we have at least two Replica Sets: - - - the "new" set (corresponding to the "target" version) - - - at least one "old" set - -- We can have multiple "old" sets - - (if we start another update before the first one is done) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Update strategy - -- Two parameters determine the pace of the rollout: `maxUnavailable` and `maxSurge` - -- They can be specified in absolute number of pods, or percentage of the `replicas` count - -- At any given time ... - - - there will always be at least `replicas`-`maxUnavailable` pods available - - - there will never be more than `replicas`+`maxSurge` pods in total - - - there will therefore be up to `maxUnavailable`+`maxSurge` pods being updated - -- We have the possibility of rolling back to the previous version -
(if the update fails or is unsatisfactory in any way) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Checking current rollout parameters - -- Recall how we build custom reports with `kubectl` and `jq`: - -.lab[ - -- Show the rollout plan for our deployments: - ```bash - kubectl get deploy -o json | - jq ".items[] | {name:.metadata.name} + .spec.strategy.rollingUpdate" - ``` - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling updates in practice - -- As of Kubernetes 1.8, we can do rolling updates with: - - `deployments`, `daemonsets`, `statefulsets` - -- Editing one of these resources will automatically result in a rolling update - -- Rolling updates can be monitored with the `kubectl rollout` subcommand - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling out the new `worker` service - -.lab[ - -- Let's monitor what's going on by opening a few terminals, and run: - ```bash - kubectl get pods -w - kubectl get replicasets -w - kubectl get deployments -w - ``` - - - -- Update `worker` either with `kubectl edit`, or by running: - ```bash - kubectl set image deploy worker worker=dockercoins/worker:v0.2 - ``` - -] - --- - -That rollout should be pretty quick. What shows in the web UI? - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Give it some time - -- At first, it looks like nothing is happening (the graph remains at the same level) - -- According to `kubectl get deploy -w`, the `deployment` was updated really quickly - -- But `kubectl get pods -w` tells a different story - -- The old `pods` are still here, and they stay in `Terminating` state for a while - -- Eventually, they are terminated; and then the graph decreases significantly - -- This delay is due to the fact that our worker doesn't handle signals - -- Kubernetes sends a "polite" shutdown request to the worker, which ignores it - -- After a grace period, Kubernetes gets impatient and kills the container - - (The grace period is 30 seconds, but [can be changed](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods) if needed) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling out something invalid - -- What happens if we make a mistake? - -.lab[ - -- Update `worker` by specifying a non-existent image: - ```bash - kubectl set image deploy worker worker=dockercoins/worker:v0.3 - ``` - -- Check what's going on: - ```bash - kubectl rollout status deploy worker - ``` - - - -] - --- - -Our rollout is stuck. However, the app is not dead. - -(After a minute, it will stabilize to be 20-25% slower.) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## What's going on with our rollout? - -- Why is our app a bit slower? - -- Because `MaxUnavailable=25%` - - ... So the rollout terminated 2 replicas out of 10 available - -- Okay, but why do we see 5 new replicas being rolled out? - -- Because `MaxSurge=25%` - - ... So in addition to replacing 2 replicas, the rollout is also starting 3 more - -- It rounded down the number of MaxUnavailable pods conservatively, -
- but the total number of pods being rolled out is allowed to be 25+25=50% - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -class: extra-details - -## The nitty-gritty details - -- We start with 10 pods running for the `worker` deployment - -- Current settings: MaxUnavailable=25% and MaxSurge=25% - -- When we start the rollout: - - - two replicas are taken down (as per MaxUnavailable=25%) - - two others are created (with the new version) to replace them - - three others are created (with the new version) per MaxSurge=25%) - -- Now we have 8 replicas up and running, and 5 being deployed - -- Our rollout is stuck at this point! - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Checking the dashboard during the bad rollout - -If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. - -.lab[ - -- Connect to the dashboard that we deployed earlier - -- Check that we have failures in Deployments, Pods, and Replica Sets - -- Can we see the reason for the failure? - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Recovering from a bad rollout - -- We could push some `v0.3` image - - (the pod retry logic will eventually catch it and the rollout will proceed) - -- Or we could invoke a manual rollback - -.lab[ - - - -- Cancel the deployment and wait for the dust to settle: - ```bash - kubectl rollout undo deploy worker - kubectl rollout status deploy worker - ``` - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling back to an older version - -- We reverted to `v0.2` - -- But this version still has a performance problem - -- How can we get back to the previous version? - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Multiple "undos" - -- What happens if we try `kubectl rollout undo` again? - -.lab[ - -- Try it: - ```bash - kubectl rollout undo deployment worker - ``` - -- Check the web UI, the list of pods ... - -] - -🤔 That didn't work. - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Multiple "undos" don't work - -- If we see successive versions as a stack: - - - `kubectl rollout undo` doesn't "pop" the last element from the stack - - - it copies the N-1th element to the top - -- Multiple "undos" just swap back and forth between the last two versions! - -.lab[ - -- Go back to v0.2 again: - ```bash - kubectl rollout undo deployment worker - ``` - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## In this specific scenario - -- Our version numbers are easy to guess - -- What if we had used git hashes? - -- What if we had changed other parameters in the Pod spec? - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Listing versions - -- We can list successive versions of a Deployment with `kubectl rollout history` - -.lab[ - -- Look at our successive versions: - ```bash - kubectl rollout history deployment worker - ``` - -] - -We don't see *all* revisions. - -We might see something like 1, 4, 5. - -(Depending on how many "undos" we did before.) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Explaining deployment revisions - -- These revisions correspond to our Replica Sets - -- This information is stored in the Replica Set annotations - -.lab[ - -- Check the annotations for our replica sets: - ```bash - kubectl describe replicasets -l app=worker | grep -A3 ^Annotations - ``` - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -class: extra-details - -## What about the missing revisions? - -- The missing revisions are stored in another annotation: - - `deployment.kubernetes.io/revision-history` - -- These are not shown in `kubectl rollout history` - -- We could easily reconstruct the full list with a script - - (if we wanted to!) - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -## Rolling back to an older version - -- `kubectl rollout undo` can work with a revision number - -.lab[ - -- Roll back to the "known good" deployment version: - ```bash - kubectl rollout undo deployment worker --to-revision=1 - ``` - -- Check the web UI or the list of pods - -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -class: extra-details - -## Changing rollout parameters - -- We want to: - - - revert to `v0.1` - - be conservative on availability (always have desired number of available workers) - - go slow on rollout speed (update only one pod at a time) - - give some time to our workers to "warm up" before starting more - -The corresponding changes can be expressed in the following YAML snippet: - -.small[ -```yaml -spec: - template: - spec: - containers: - - name: worker - image: dockercoins/worker:v0.1 - strategy: - rollingUpdate: - maxUnavailable: 0 - maxSurge: 1 - minReadySeconds: 10 -``` -] - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -class: extra-details - -## Applying changes through a YAML patch - -- We could use `kubectl edit deployment worker` - -- But we could also use `kubectl patch` with the exact YAML shown before - -.lab[ - -.small[ - -- Apply all our changes and wait for them to take effect: - ```bash - kubectl patch deployment worker -p " - spec: - template: - spec: - containers: - - name: worker - image: dockercoins/worker:v0.1 - strategy: - rollingUpdate: - maxUnavailable: 0 - maxSurge: 1 - minReadySeconds: 10 - " - kubectl rollout status deployment worker - kubectl get deploy -o json worker | - jq "{name:.metadata.name} + .spec.strategy.rollingUpdate" - ``` - ] - -] - -??? - -:EN:- Rolling updates -:EN:- Rolling back a bad deployment - -:FR:- Mettre à jour un déploiement -:FR:- Concept de *rolling update* et *rollback* -:FR:- Paramétrer la vitesse de déploiement - -.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/two-containers-on-a-truck.jpg)] - ---- - -name: toc-deploying-with-yaml -class: title - - Deploying with YAML - -.nav[ -[Previous part](#toc-rolling-updates) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-namespaces) -] - -.debug[(automatically generated title slide)] - ---- -# Deploying with YAML - -- So far, we created resources with the following commands: - - - `kubectl run` - - - `kubectl create deployment` - - - `kubectl expose` - -- We can also create resources directly with YAML manifests - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## `kubectl apply` vs `create` - -- `kubectl create -f whatever.yaml` - - - creates resources if they don't exist - - - if resources already exist, don't alter them -
(and display error message) - -- `kubectl apply -f whatever.yaml` - - - creates resources if they don't exist - - - if resources already exist, update them -
(to match the definition provided by the YAML file) - - - stores the manifest as an *annotation* in the resource - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## Creating multiple resources - -- The manifest can contain multiple resources separated by `---` - -```yaml - kind: ... - apiVersion: ... - metadata: ... - name: ... - ... - --- - kind: ... - apiVersion: ... - metadata: ... - name: ... - ... -``` - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## Creating multiple resources - -- The manifest can also contain a list of resources - -```yaml - apiVersion: v1 - kind: List - items: - - kind: ... - apiVersion: ... - ... - - kind: ... - apiVersion: ... - ... -``` - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## Deploying dockercoins with YAML - -- We provide a YAML manifest with all the resources for Dockercoins - - (Deployments and Services) - -- We can use it if we need to deploy or redeploy Dockercoins - -.lab[ - -- Deploy or redeploy Dockercoins: - ```bash - kubectl apply -f ~/container.training/k8s/dockercoins.yaml - ``` - -] - -(If we deployed Dockercoins earlier, we will see warning messages, -because the resources that we created lack the necessary annotation. -We can safely ignore them.) - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## Deleting resources - -- We can also use a YAML file to *delete* resources - -- `kubectl delete -f ...` will delete all the resources mentioned in a YAML file - - (useful to clean up everything that was created by `kubectl apply -f ...`) - -- The definitions of the resources don't matter - - (just their `kind`, `apiVersion`, and `name`) - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## Pruning¹ resources - -- We can also tell `kubectl` to remove old resources - -- This is done with `kubectl apply -f ... --prune` - -- It will remove resources that don't exist in the YAML file(s) - -- But only if they were created with `kubectl apply` in the first place - - (technically, if they have an annotation `kubectl.kubernetes.io/last-applied-configuration`) - -.footnote[¹If English is not your first language: *to prune* means to remove dead or overgrown branches in a tree, to help it to grow.] - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -## YAML as source of truth - -- Imagine the following workflow: - - - do not use `kubectl run`, `kubectl create deployment`, `kubectl expose` ... - - - define everything with YAML - - - `kubectl apply -f ... --prune --all` that YAML - - - keep that YAML under version control - - - enforce all changes to go through that YAML (e.g. with pull requests) - -- Our version control system now has a full history of what we deploy - -- Compares to "Infrastructure-as-Code", but for app deployments - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -class: extra-details - -## Specifying the namespace - -- When creating resources from YAML manifests, the namespace is optional - -- If we specify a namespace: - - - resources are created in the specified namespace - - - this is typical for things deployed only once per cluster - - - example: system components, cluster add-ons ... - -- If we don't specify a namespace: - - - resources are created in the current namespace - - - this is typical for things that may be deployed multiple times - - - example: applications (production, staging, feature branches ...) - -??? - -:EN:- Deploying with YAML manifests -:FR:- Déployer avec des *manifests* YAML - -.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/wall-of-containers.jpeg)] - ---- - -name: toc-namespaces -class: title - - Namespaces - -.nav[ -[Previous part](#toc-deploying-with-yaml) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-daemon-sets) -] - -.debug[(automatically generated title slide)] - ---- -# Namespaces - -- We would like to deploy another copy of DockerCoins on our cluster - -- We could rename all our deployments and services: - - hasher → hasher2, redis → redis2, rng → rng2, etc. - -- That would require updating the code - -- There has to be a better way! - --- - -- As hinted by the title of this section, we will use *namespaces* - -.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] ---- - -## Identifying a resource - -- We cannot have two resources with the same name - - (or can we...?) - --- - -- We cannot have two resources *of the same kind* with the same name - - (but it's OK to have an `rng` service, an `rng` deployment, and an `rng` daemon set) - --- - -- We cannot have two resources of the same kind with the same name *in the same namespace* - - (but it's OK to have e.g. two `rng` services in different namespaces) - --- - -- Except for resources that exist at the *cluster scope* - - (these do not belong to a namespace) - -.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] ---- - -## Uniquely identifying a resource - -- For *namespaced* resources: - - the tuple *(kind, name, namespace)* needs to be unique - -- For resources at the *cluster scope*: - - the tuple *(kind, name)* needs to be unique - -.lab[ - -- List resource types again, and check the NAMESPACED column: - ```bash - kubectl api-resources - ``` - -] - -.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] ---- - -## Pre-existing namespaces - -- If we deploy a cluster with `kubeadm`, we have three or four namespaces: - - - `default` (for our applications) - - - `kube-system` (for the control plane) - - - `kube-public` (contains one ConfigMap for cluster discovery) - - - `kube-node-lease` (in Kubernetes 1.14 and later; contains Lease objects) - -- If we deploy differently, we may have different namespaces - -.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] ---- - -## Creating namespaces - -- Let's see two identical methods to create a namespace - -.lab[ - -- We can use `kubectl create namespace`: - ```bash - kubectl create namespace blue - ``` - -- Or we can construct a very minimal YAML snippet: - ```bash - kubectl apply -f- < - -- Unfortunately, as of Kubernetes 1.19, the CLI cannot create daemon sets - --- - -- More precisely: it doesn't have a subcommand to create a daemon set - --- - -- But any kind of resource can always be created by providing a YAML description: - ```bash - kubectl apply -f foo.yaml - ``` - --- - -- How do we create the YAML file for our daemon set? - --- - - - option 1: [read the docs](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#create-a-daemonset) - --- - - - option 2: `vi` our way out of it - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Creating the YAML file for our daemon set - -- Let's start with the YAML file for the current `rng` resource - -.lab[ - -- Dump the `rng` resource in YAML: - ```bash - kubectl get deploy/rng -o yaml >rng.yml - ``` - -- Edit `rng.yml` - -] - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## "Casting" a resource to another - -- What if we just changed the `kind` field? - - (It can't be that easy, right?) - -.lab[ - -- Change `kind: Deployment` to `kind: DaemonSet` - - - -- Save, quit - -- Try to create our new resource: - ```bash - kubectl apply -f rng.yml - ``` - - - -] - --- - -We all knew this couldn't be that easy, right! - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Understanding the problem - -- The core of the error is: - ``` - error validating data: - [ValidationError(DaemonSet.spec): - unknown field "replicas" in io.k8s.api.extensions.v1beta1.DaemonSetSpec, - ... - ``` - --- - -- *Obviously,* it doesn't make sense to specify a number of replicas for a daemon set - --- - -- Workaround: fix the YAML - - - remove the `replicas` field - - remove the `strategy` field (which defines the rollout mechanism for a deployment) - - remove the `progressDeadlineSeconds` field (also used by the rollout mechanism) - - remove the `status: {}` line at the end - --- - -- Or, we could also ... - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Use the `--force`, Luke - -- We could also tell Kubernetes to ignore these errors and try anyway - -- The `--force` flag's actual name is `--validate=false` - -.lab[ - -- Try to load our YAML file and ignore errors: - ```bash - kubectl apply -f rng.yml --validate=false - ``` - -] - --- - -🎩✨🐇 - --- - -Wait ... Now, can it be *that* easy? - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Checking what we've done - -- Did we transform our `deployment` into a `daemonset`? - -.lab[ - -- Look at the resources that we have now: - ```bash - kubectl get all - ``` - -] - --- - -We have two resources called `rng`: - -- the *deployment* that was existing before - -- the *daemon set* that we just created - -We also have one too many pods. -
-(The pod corresponding to the *deployment* still exists.) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## `deploy/rng` and `ds/rng` - -- You can have different resource types with the same name - - (i.e. a *deployment* and a *daemon set* both named `rng`) - -- We still have the old `rng` *deployment* - - ``` -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -deployment.apps/rng 1 1 1 1 18m - ``` - -- But now we have the new `rng` *daemon set* as well - - ``` -NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE -daemonset.apps/rng 2 2 2 2 2 9s - ``` - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Too many pods - -- If we check with `kubectl get pods`, we see: - - - *one pod* for the deployment (named `rng-xxxxxxxxxx-yyyyy`) - - - *one pod per node* for the daemon set (named `rng-zzzzz`) - - ``` - NAME READY STATUS RESTARTS AGE - rng-54f57d4d49-7pt82 1/1 Running 0 11m - rng-b85tm 1/1 Running 0 25s - rng-hfbrr 1/1 Running 0 25s - [...] - ``` - --- - -The daemon set created one pod per node, except on the master node. - -The master node has [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) preventing pods from running there. - -(To schedule a pod on this node anyway, the pod will require appropriate [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/).) - -.footnote[(Off by one? We don't run these pods on the node hosting the control plane.)] - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Is this working? - -- Look at the web UI - --- - -- The graph should now go above 10 hashes per second! - --- - -- It looks like the newly created pods are serving traffic correctly - -- How and why did this happen? - - (We didn't do anything special to add them to the `rng` service load balancer!) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/ShippingContainerSFBay.jpg)] - ---- - -name: toc-labels-and-selectors -class: title - - Labels and selectors - -.nav[ -[Previous part](#toc-daemon-sets) -| -[Back to table of contents](#toc-part-2) -| -[Next part](#toc-healthchecks) -] - -.debug[(automatically generated title slide)] - ---- - -# Labels and selectors - -- The `rng` *service* is load balancing requests to a set of pods - -- That set of pods is defined by the *selector* of the `rng` service - -.lab[ - -- Check the *selector* in the `rng` service definition: - ```bash - kubectl describe service rng - ``` - -] - -- The selector is `app=rng` - -- It means "all the pods having the label `app=rng`" - - (They can have additional labels as well, that's OK!) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Selector evaluation - -- We can use selectors with many `kubectl` commands - -- For instance, with `kubectl get`, `kubectl logs`, `kubectl delete` ... and more - -.lab[ - -- Get the list of pods matching selector `app=rng`: - ```bash - kubectl get pods -l app=rng - kubectl get pods --selector app=rng - ``` - -] - -But ... why do these pods (in particular, the *new* ones) have this `app=rng` label? - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Where do labels come from? - -- When we create a deployment with `kubectl create deployment rng`, -
this deployment gets the label `app=rng` - -- The replica sets created by this deployment also get the label `app=rng` - -- The pods created by these replica sets also get the label `app=rng` - -- When we created the daemon set from the deployment, we re-used the same spec - -- Therefore, the pods created by the daemon set get the same labels - -.footnote[Note: when we use `kubectl run stuff`, the label is `run=stuff` instead.] - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Updating load balancer configuration - -- We would like to remove a pod from the load balancer - -- What would happen if we removed that pod, with `kubectl delete pod ...`? - --- - - It would be re-created immediately (by the replica set or the daemon set) - --- - -- What would happen if we removed the `app=rng` label from that pod? - --- - - It would *also* be re-created immediately - --- - - Why?!? - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Selectors for replica sets and daemon sets - -- The "mission" of a replica set is: - - "Make sure that there is the right number of pods matching this spec!" - -- The "mission" of a daemon set is: - - "Make sure that there is a pod matching this spec on each node!" - --- - -- *In fact,* replica sets and daemon sets do not check pod specifications - -- They merely have a *selector*, and they look for pods matching that selector - -- Yes, we can fool them by manually creating pods with the "right" labels - -- Bottom line: if we remove our `app=rng` label ... - - ... The pod "disappears" for its parent, which re-creates another pod to replace it - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Isolation of replica sets and daemon sets - -- Since both the `rng` daemon set and the `rng` replica set use `app=rng` ... - - ... Why don't they "find" each other's pods? - --- - -- *Replica sets* have a more specific selector, visible with `kubectl describe` - - (It looks like `app=rng,pod-template-hash=abcd1234`) - -- *Daemon sets* also have a more specific selector, but it's invisible - - (It looks like `app=rng,controller-revision-hash=abcd1234`) - -- As a result, each controller only "sees" the pods it manages - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Removing a pod from the load balancer - -- Currently, the `rng` service is defined by the `app=rng` selector - -- The only way to remove a pod is to remove or change the `app` label - -- ... But that will cause another pod to be created instead! - -- What's the solution? - --- - -- We need to change the selector of the `rng` service! - -- Let's add another label to that selector (e.g. `active=yes`) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Selectors with multiple labels - -- If a selector specifies multiple labels, they are understood as a logical *AND* - - (in other words: the pods must match all the labels) - -- We cannot have a logical *OR* - - (e.g. `app=api AND (release=prod OR release=preprod)`) - -- We can, however, apply as many extra labels as we want to our pods: - - - use selector `app=api AND prod-or-preprod=yes` - - - add `prod-or-preprod=yes` to both sets of pods - -- We will see later that in other places, we can use more advanced selectors - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## The plan - -1. Add the label `active=yes` to all our `rng` pods - -2. Update the selector for the `rng` service to also include `active=yes` - -3. Toggle traffic to a pod by manually adding/removing the `active` label - -4. Profit! - -*Note: if we swap steps 1 and 2, it will cause a short -service disruption, because there will be a period of time -during which the service selector won't match any pod. -During that time, requests to the service will time out. -By doing things in the order above, we guarantee that there won't -be any interruption.* - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Adding labels to pods - -- We want to add the label `active=yes` to all pods that have `app=rng` - -- We could edit each pod one by one with `kubectl edit` ... - -- ... Or we could use `kubectl label` to label them all - -- `kubectl label` can use selectors itself - -.lab[ - -- Add `active=yes` to all pods that have `app=rng`: - ```bash - kubectl label pods -l app=rng active=yes - ``` - -] - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Updating the service selector - -- We need to edit the service specification - -- Reminder: in the service definition, we will see `app: rng` in two places - - - the label of the service itself (we don't need to touch that one) - - - the selector of the service (that's the one we want to change) - -.lab[ - -- Update the service to add `active: yes` to its selector: - ```bash - kubectl edit service rng - ``` - - - -] - --- - -... And then we get *the weirdest error ever.* Why? - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## When the YAML parser is being too smart - -- YAML parsers try to help us: - - - `xyz` is the string `"xyz"` - - - `42` is the integer `42` - - - `yes` is the boolean value `true` - -- If we want the string `"42"` or the string `"yes"`, we have to quote them - -- So we have to use `active: "yes"` - -.footnote[For a good laugh: if we had used "ja", "oui", "si" ... as the value, it would have worked!] - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Updating the service selector, take 2 - -.lab[ - -- Update the YAML manifest of the service - -- Add `active: "yes"` to its selector - - - -] - -This time it should work! - -If we did everything correctly, the web UI shouldn't show any change. - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Updating labels - -- We want to disable the pod that was created by the deployment - -- All we have to do, is remove the `active` label from that pod - -- To identify that pod, we can use its name - -- ... Or rely on the fact that it's the only one with a `pod-template-hash` label - -- Good to know: - - - `kubectl label ... foo=` doesn't remove a label (it sets it to an empty string) - - - to remove label `foo`, use `kubectl label ... foo-` - - - to change an existing label, we would need to add `--overwrite` - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Removing a pod from the load balancer - -.lab[ - -- In one window, check the logs of that pod: - ```bash - POD=$(kubectl get pod -l app=rng,pod-template-hash -o name) - kubectl logs --tail 1 --follow $POD - ``` - (We should see a steady stream of HTTP logs) - - - -- In another window, remove the label from the pod: - ```bash - kubectl label pod -l app=rng,pod-template-hash active- - ``` - (The stream of HTTP logs should stop immediately) - - - -] - -There might be a slight change in the web UI (since we removed a bit -of capacity from the `rng` service). If we remove more pods, -the effect should be more visible. - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Updating the daemon set - -- If we scale up our cluster by adding new nodes, the daemon set will create more pods - -- These pods won't have the `active=yes` label - -- If we want these pods to have that label, we need to edit the daemon set spec - -- We can do that with e.g. `kubectl edit daemonset rng` - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## We've put resources in your resources - -- Reminder: a daemon set is a resource that creates more resources! - -- There is a difference between: - - - the label(s) of a resource (in the `metadata` block in the beginning) - - - the selector of a resource (in the `spec` block) - - - the label(s) of the resource(s) created by the first resource (in the `template` block) - -- We would need to update the selector and the template - - (metadata labels are not mandatory) - -- The template must match the selector - - (i.e. the resource will refuse to create resources that it will not select) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Labels and debugging - -- When a pod is misbehaving, we can delete it: another one will be recreated - -- But we can also change its labels - -- It will be removed from the load balancer (it won't receive traffic anymore) - -- Another pod will be recreated immediately - -- But the problematic pod is still here, and we can inspect and debug it - -- We can even re-add it to the rotation if necessary - - (Very useful to troubleshoot intermittent and elusive bugs) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -## Labels and advanced rollout control - -- Conversely, we can add pods matching a service's selector - -- These pods will then receive requests and serve traffic - -- Examples: - - - one-shot pod with all debug flags enabled, to collect logs - - - pods created automatically, but added to rotation in a second step -
- (by setting their label accordingly) - -- This gives us building blocks for canary and blue/green deployments - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Advanced label selectors - -- As indicated earlier, service selectors are limited to a `AND` - -- But in many other places in the Kubernetes API, we can use complex selectors - - (e.g. Deployment, ReplicaSet, DaemonSet, NetworkPolicy ...) - -- These allow extra operations; specifically: - - - checking for presence (or absence) of a label - - - checking if a label is (or is not) in a given set - -- Relevant documentation: - - [Service spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#servicespec-v1-core), - [LabelSelector spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#labelselector-v1-meta), - [label selector doc](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Example of advanced selector - -```yaml - theSelector: - matchLabels: - app: portal - component: api - matchExpressions: - - key: release - operator: In - values: [ production, preproduction ] - - key: signed-off-by - operator: Exists -``` - -This selector matches pods that meet *all* the indicated conditions. - -`operator` can be `In`, `NotIn`, `Exists`, `DoesNotExist`. - -A `nil` selector matches *nothing*, a `{}` selector matches *everything*. -
-(Because that means "match all pods that meet at least zero condition".) - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Services and Endpoints - -- Each Service has a corresponding Endpoints resource - - (see `kubectl get endpoints` or `kubectl get ep`) - -- That Endpoints resource is used by various controllers - - (e.g. `kube-proxy` when setting up `iptables` rules for ClusterIP services) - -- These Endpoints are populated (and updated) with the Service selector - -- We can update the Endpoints manually, but our changes will get overwritten - -- ... Except if the Service selector is empty! - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: extra-details - -## Empty Service selector - -- If a service selector is empty, Endpoints don't get updated automatically - - (but we can still set them manually) - -- This lets us create Services pointing to arbitrary destinations - - (potentially outside the cluster; or things that are not in pods) - -- Another use-case: the `kubernetes` service in the `default` namespace - - (its Endpoints are maintained automatically by the API server) - -??? - -:EN:- Scaling with Daemon Sets -:FR:- Utilisation de Daemon Sets - -.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/aerial-view-of-containers.jpg)] - ---- - -name: toc-healthchecks -class: title - - Healthchecks - -.nav[ -[Previous part](#toc-labels-and-selectors) -| -[Back to table of contents](#toc-part-3) -| -[Next part](#toc-kubernetes-network-model) -] - -.debug[(automatically generated title slide)] - ---- -# Healthchecks - -- Containers can have *healthchecks* - -- There are three kinds of healthchecks, corresponding to very different use-cases: - - - liveness = detect when a container is "dead" and needs to be restarted - - - readiness = detect when a container is ready to serve traffic - - - startup = detect if a container has finished to boot - -- These healthchecks are optional (we can use none, all, or some of them) - -- Different probes are available (HTTP request, TCP connection, program execution) - -- Let's see the difference and how to use them! - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Liveness probe - -*This container is dead, we don't know how to fix it, other than restarting it.* - -- Indicates if the container is dead or alive - -- A dead container cannot come back to life - -- If the liveness probe fails, the container is killed (destroyed) - - (to make really sure that it's really dead; no zombies or undeads!) - -- What happens next depends on the pod's `restartPolicy`: - - - `Never`: the container is not restarted - - - `OnFailure` or `Always`: the container is restarted - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## When to use a liveness probe - -- To indicate failures that can't be recovered - - - deadlocks (causing all requests to time out) - - - internal corruption (causing all requests to error) - -- Anything where our incident response would be "just restart/reboot it" - -.warning[**Do not** use liveness probes for problems that can't be fixed by a restart] - -- Otherwise we just restart our pods for no reason, creating useless load - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Readiness probe (1) - -*Make sure that a container is ready before continuing a rolling update.* - -- Indicates if the container is ready to handle traffic - -- When doing a rolling update, the Deployment controller waits for Pods to be ready - - (a Pod is ready when all the containers in the Pod are ready) - -- Improves reliability and safety of rolling updates: - - - don't roll out a broken version (that doesn't pass readiness checks) - - - don't lose processing capacity during a rolling update - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Readiness probe (2) - -*Temporarily remove a container (overloaded or otherwise) from a Service load balancer.* - -- A container can mark itself "not ready" temporarily - - (e.g. if it's overloaded or needs to reload/restart/garbage collect...) - -- If a container becomes "unready" it might be ready again soon - -- If the readiness probe fails: - - - the container is *not* killed - - - if the pod is a member of a service, it is temporarily removed - - - it is re-added as soon as the readiness probe passes again - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## When to use a readiness probe - -- To indicate failure due to an external cause - - - database is down or unreachable - - - mandatory auth or other backend service unavailable - -- To indicate temporary failure or unavailability - - - application can only service *N* parallel connections - - - runtime is busy doing garbage collection or initial data load - -- To redirect new connections to other Pods - - (e.g. fail the readiness probe when the Pod's load is too high) - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Dependencies - -- If a web server depends on a database to function, and the database is down: - - - the web server's liveness probe should succeed - - - the web server's readiness probe should fail - -- Same thing for any hard dependency (without which the container can't work) - -.warning[**Do not** fail liveness probes for problems that are external to the container] - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Timing and thresholds - -- Probes are executed at intervals of `periodSeconds` (default: 10) - -- The timeout for a probe is set with `timeoutSeconds` (default: 1) - -.warning[If a probe takes longer than that, it is considered as a FAIL] - -- A probe is considered successful after `successThreshold` successes (default: 1) - -- A probe is considered failing after `failureThreshold` failures (default: 3) - -- A probe can have an `initialDelaySeconds` parameter (default: 0) - -- Kubernetes will wait that amount of time before running the probe for the first time - - (this is important to avoid killing services that take a long time to start) - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Startup probe - -*The container takes too long to start, and is killed by the liveness probe!* - -- By default, probes (including liveness) start immediately - -- With the default probe interval and failure threshold: - - *a container must respond in less than 30 seconds, or it will be killed!* - -- There are two ways to avoid that: - - - set `initialDelaySeconds` (a fixed, rigid delay) - - - use a `startupProbe` - -- Kubernetes will run only the startup probe, and when it succeeds, run the other probes - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## When to use a startup probe - -- For containers that take a long time to start - - (more than 30 seconds) - -- Especially if that time can vary a lot - - (e.g. fast in dev, slow in prod, or the other way around) - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Different types of probes - -- HTTP request - - - specify URL of the request (and optional headers) - - - any status code between 200 and 399 indicates success - -- TCP connection - - - the probe succeeds if the TCP port is open - -- arbitrary exec - - - a command is executed in the container - - - exit status of zero indicates success - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Benefits of using probes - -- Rolling updates proceed when containers are *actually ready* - - (as opposed to merely started) - -- Containers in a broken state get killed and restarted - - (instead of serving errors or timeouts) - -- Unavailable backends get removed from load balancer rotation - - (thus improving response times across the board) - -- If a probe is not defined, it's as if there was an "always successful" probe - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Example: HTTP probe - -Here is a pod template for the `rng` web service of the DockerCoins app: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: healthy-app -spec: - containers: - - name: myapp - image: myregistry.io/myapp:v1.0 - livenessProbe: - httpGet: - path: /health - port: 80 - periodSeconds: 5 -``` - -If the backend serves an error, or takes longer than 1s, 3 times in a row, it gets killed. - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Example: exec probe - -Here is a pod template for a Redis server: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: redis-with-liveness -spec: - containers: - - name: redis - image: redis - livenessProbe: - exec: - command: ["redis-cli", "ping"] -``` - -If the Redis process becomes unresponsive, it will be killed. - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Questions to ask before adding healthchecks - -- Do we want liveness, readiness, both? - - (sometimes, we can use the same check, but with different failure thresholds) - -- Do we have existing HTTP endpoints that we can use? - -- Do we need to add new endpoints, or perhaps use something else? - -- Are our healthchecks likely to use resources and/or slow down the app? - -- Do they depend on additional services? - - (this can be particularly tricky, see next slide) - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Healthchecks and dependencies - -- Liveness checks should not be influenced by the state of external services - -- All checks should reply quickly (by default, less than 1 second) - -- Otherwise, they are considered to fail - -- This might require to check the health of dependencies asynchronously - - (e.g. if a database or API might be healthy but still take more than - 1 second to reply, we should check the status asynchronously and report - a cached status) - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -## Healthchecks for workers - -(In that context, worker = process that doesn't accept connections) - -- Readiness is useful mostly for rolling updates - - (because workers aren't backends for a service) - -- Liveness may help us restart a broken worker, but how can we check it? - -- Embedding an HTTP server is a (potentially expensive) option - -- Using a "lease" file can be relatively easy: - - - touch a file during each iteration of the main loop - - - check the timestamp of that file from an exec probe - -- Writing logs (and checking them from the probe) also works - -??? - -:EN:- Using healthchecks to improve availability -:FR:- Utiliser des *healthchecks* pour améliorer la disponibilité - -.debug[[k8s/healthchecks.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/healthchecks.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/blue-containers.jpg)] - ---- - -name: toc-kubernetes-network-model -class: title - - Kubernetes network model - -.nav[ -[Previous part](#toc-healthchecks) -| -[Back to table of contents](#toc-part-3) -| -[Next part](#toc-exposing-containers) -] - -.debug[(automatically generated title slide)] - ---- -# Kubernetes network model - -- TL,DR: - - *Our cluster (nodes and pods) is one big flat IP network.* - --- - -- In detail: - - - all nodes must be able to reach each other, without NAT - - - all pods must be able to reach each other, without NAT - - - pods and nodes must be able to reach each other, without NAT - - - each pod is aware of its IP address (no NAT) - - - pod IP addresses are assigned by the network implementation - -- Kubernetes doesn't mandate any particular implementation - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -## Kubernetes network model: the good - -- Everything can reach everything - -- No address translation - -- No port translation - -- No new protocol - -- The network implementation can decide how to allocate addresses - -- IP addresses don't have to be "portable" from a node to another - - (We can use e.g. a subnet per node and use a simple routed topology) - -- The specification is simple enough to allow many various implementations - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -## Kubernetes network model: the less good - -- Everything can reach everything - - - if you want security, you need to add network policies - - - the network implementation that you use needs to support them - -- There are literally dozens of implementations out there - - (https://github.com/containernetworking/cni/ lists more than 25 plugins) - -- Pods have level 3 (IP) connectivity, but *services* are level 4 (TCP or UDP) - - (Services map to a single UDP or TCP port; no port ranges or arbitrary IP packets) - -- `kube-proxy` is on the data path when connecting to a pod or container, -
and it's not particularly fast (relies on userland proxying or iptables) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -## Kubernetes network model: in practice - -- The nodes that we are using have been set up to use [Weave](https://github.com/weaveworks/weave) - -- We don't endorse Weave in a particular way, it just Works For Us - -- Don't worry about the warning about `kube-proxy` performance - -- Unless you: - - - routinely saturate 10G network interfaces - - count packet rates in millions per second - - run high-traffic VOIP or gaming platforms - - do weird things that involve millions of simultaneous connections -
(in which case you're already familiar with kernel tuning) - -- If necessary, there are alternatives to `kube-proxy`; e.g. - [`kube-router`](https://www.kube-router.io) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: extra-details - -## The Container Network Interface (CNI) - -- Most Kubernetes clusters use CNI "plugins" to implement networking - -- When a pod is created, Kubernetes delegates the network setup to these plugins - - (it can be a single plugin, or a combination of plugins, each doing one task) - -- Typically, CNI plugins will: - - - allocate an IP address (by calling an IPAM plugin) - - - add a network interface into the pod's network namespace - - - configure the interface as well as required routes etc. - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: extra-details - -## Multiple moving parts - -- The "pod-to-pod network" or "pod network": - - - provides communication between pods and nodes - - - is generally implemented with CNI plugins - -- The "pod-to-service network": - - - provides internal communication and load balancing - - - is generally implemented with kube-proxy (or e.g. kube-router) - -- Network policies: - - - provide firewalling and isolation - - - can be bundled with the "pod network" or provided by another component - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -![Overview of the three Kubernetes network layers](images/k8s-net-0-overview.svg) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -![Pod-to-pod network](images/k8s-net-1-pod-to-pod.svg) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -![Pod-to-service network](images/k8s-net-2-pod-to-svc.svg) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -![Network policies](images/k8s-net-3-netpol.svg) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -![View with all the layers again](images/k8s-net-4-overview.svg) - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: extra-details - -## Even more moving parts - -- Inbound traffic can be handled by multiple components: - - - something like kube-proxy or kube-router (for NodePort services) - - - load balancers (ideally, connected to the pod network) - -- It is possible to use multiple pod networks in parallel - - (with "meta-plugins" like CNI-Genie or Multus) - -- Some solutions can fill multiple roles - - (e.g. kube-router can be set up to provide the pod network and/or network policies and/or replace kube-proxy) - -??? - -:EN:- The Kubernetes network model -:FR:- Le modèle réseau de Kubernetes - -.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] --- @@ -7003,11 +4251,11 @@ class: title Exposing containers .nav[ -[Previous part](#toc-kubernetes-network-model) +[Previous part](#toc-more-contact-with-kubectl) | -[Back to table of contents](#toc-part-3) +[Back to table of contents](#toc-part-2) | -[Next part](#toc-volumes) +[Next part](#toc-rolling-updates) ] .debug[(automatically generated title slide)] @@ -7626,6 +4874,2743 @@ class: pic class: pic +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-2.jpg)] + +--- + +name: toc-rolling-updates +class: title + + Rolling updates + +.nav[ +[Previous part](#toc-exposing-containers) +| +[Back to table of contents](#toc-part-2) +| +[Next part](#toc-deploying-with-yaml) +] + +.debug[(automatically generated title slide)] + +--- +# Rolling updates + +- By default (without rolling updates), when a scaled resource is updated: + + - new pods are created + + - old pods are terminated + + - ... all at the same time + + - if something goes wrong, ¯\\\_(ツ)\_/¯ + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling updates + +- With rolling updates, when a Deployment is updated, it happens progressively + +- The Deployment controls multiple Replica Sets + +- Each Replica Set is a group of identical Pods + + (with the same image, arguments, parameters ...) + +- During the rolling update, we have at least two Replica Sets: + + - the "new" set (corresponding to the "target" version) + + - at least one "old" set + +- We can have multiple "old" sets + + (if we start another update before the first one is done) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Update strategy + +- Two parameters determine the pace of the rollout: `maxUnavailable` and `maxSurge` + +- They can be specified in absolute number of pods, or percentage of the `replicas` count + +- At any given time ... + + - there will always be at least `replicas`-`maxUnavailable` pods available + + - there will never be more than `replicas`+`maxSurge` pods in total + + - there will therefore be up to `maxUnavailable`+`maxSurge` pods being updated + +- We have the possibility of rolling back to the previous version +
(if the update fails or is unsatisfactory in any way) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Checking current rollout parameters + +- Recall how we build custom reports with `kubectl` and `jq`: + +.lab[ + +- Show the rollout plan for our deployments: + ```bash + kubectl get deploy -o json | + jq ".items[] | {name:.metadata.name} + .spec.strategy.rollingUpdate" + ``` + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling updates in practice + +- As of Kubernetes 1.8, we can do rolling updates with: + + `deployments`, `daemonsets`, `statefulsets` + +- Editing one of these resources will automatically result in a rolling update + +- Rolling updates can be monitored with the `kubectl rollout` subcommand + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling out the new `worker` service + +.lab[ + +- Let's monitor what's going on by opening a few terminals, and run: + ```bash + kubectl get pods -w + kubectl get replicasets -w + kubectl get deployments -w + ``` + + + +- Update `worker` either with `kubectl edit`, or by running: + ```bash + kubectl set image deploy worker worker=dockercoins/worker:v0.2 + ``` + +] + +-- + +That rollout should be pretty quick. What shows in the web UI? + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Give it some time + +- At first, it looks like nothing is happening (the graph remains at the same level) + +- According to `kubectl get deploy -w`, the `deployment` was updated really quickly + +- But `kubectl get pods -w` tells a different story + +- The old `pods` are still here, and they stay in `Terminating` state for a while + +- Eventually, they are terminated; and then the graph decreases significantly + +- This delay is due to the fact that our worker doesn't handle signals + +- Kubernetes sends a "polite" shutdown request to the worker, which ignores it + +- After a grace period, Kubernetes gets impatient and kills the container + + (The grace period is 30 seconds, but [can be changed](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods) if needed) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling out something invalid + +- What happens if we make a mistake? + +.lab[ + +- Update `worker` by specifying a non-existent image: + ```bash + kubectl set image deploy worker worker=dockercoins/worker:v0.3 + ``` + +- Check what's going on: + ```bash + kubectl rollout status deploy worker + ``` + + + +] + +-- + +Our rollout is stuck. However, the app is not dead. + +(After a minute, it will stabilize to be 20-25% slower.) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## What's going on with our rollout? + +- Why is our app a bit slower? + +- Because `MaxUnavailable=25%` + + ... So the rollout terminated 2 replicas out of 10 available + +- Okay, but why do we see 5 new replicas being rolled out? + +- Because `MaxSurge=25%` + + ... So in addition to replacing 2 replicas, the rollout is also starting 3 more + +- It rounded down the number of MaxUnavailable pods conservatively, +
+ but the total number of pods being rolled out is allowed to be 25+25=50% + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +class: extra-details + +## The nitty-gritty details + +- We start with 10 pods running for the `worker` deployment + +- Current settings: MaxUnavailable=25% and MaxSurge=25% + +- When we start the rollout: + + - two replicas are taken down (as per MaxUnavailable=25%) + - two others are created (with the new version) to replace them + - three others are created (with the new version) per MaxSurge=25%) + +- Now we have 8 replicas up and running, and 5 being deployed + +- Our rollout is stuck at this point! + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Checking the dashboard during the bad rollout + +If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. + +.lab[ + +- Connect to the dashboard that we deployed earlier + +- Check that we have failures in Deployments, Pods, and Replica Sets + +- Can we see the reason for the failure? + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Recovering from a bad rollout + +- We could push some `v0.3` image + + (the pod retry logic will eventually catch it and the rollout will proceed) + +- Or we could invoke a manual rollback + +.lab[ + + + +- Cancel the deployment and wait for the dust to settle: + ```bash + kubectl rollout undo deploy worker + kubectl rollout status deploy worker + ``` + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling back to an older version + +- We reverted to `v0.2` + +- But this version still has a performance problem + +- How can we get back to the previous version? + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Multiple "undos" + +- What happens if we try `kubectl rollout undo` again? + +.lab[ + +- Try it: + ```bash + kubectl rollout undo deployment worker + ``` + +- Check the web UI, the list of pods ... + +] + +🤔 That didn't work. + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Multiple "undos" don't work + +- If we see successive versions as a stack: + + - `kubectl rollout undo` doesn't "pop" the last element from the stack + + - it copies the N-1th element to the top + +- Multiple "undos" just swap back and forth between the last two versions! + +.lab[ + +- Go back to v0.2 again: + ```bash + kubectl rollout undo deployment worker + ``` + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## In this specific scenario + +- Our version numbers are easy to guess + +- What if we had used git hashes? + +- What if we had changed other parameters in the Pod spec? + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Listing versions + +- We can list successive versions of a Deployment with `kubectl rollout history` + +.lab[ + +- Look at our successive versions: + ```bash + kubectl rollout history deployment worker + ``` + +] + +We don't see *all* revisions. + +We might see something like 1, 4, 5. + +(Depending on how many "undos" we did before.) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Explaining deployment revisions + +- These revisions correspond to our Replica Sets + +- This information is stored in the Replica Set annotations + +.lab[ + +- Check the annotations for our replica sets: + ```bash + kubectl describe replicasets -l app=worker | grep -A3 ^Annotations + ``` + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +class: extra-details + +## What about the missing revisions? + +- The missing revisions are stored in another annotation: + + `deployment.kubernetes.io/revision-history` + +- These are not shown in `kubectl rollout history` + +- We could easily reconstruct the full list with a script + + (if we wanted to!) + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +## Rolling back to an older version + +- `kubectl rollout undo` can work with a revision number + +.lab[ + +- Roll back to the "known good" deployment version: + ```bash + kubectl rollout undo deployment worker --to-revision=1 + ``` + +- Check the web UI or the list of pods + +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +class: extra-details + +## Changing rollout parameters + +- We want to: + + - revert to `v0.1` + - be conservative on availability (always have desired number of available workers) + - go slow on rollout speed (update only one pod at a time) + - give some time to our workers to "warm up" before starting more + +The corresponding changes can be expressed in the following YAML snippet: + +.small[ +```yaml +spec: + template: + spec: + containers: + - name: worker + image: dockercoins/worker:v0.1 + strategy: + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + minReadySeconds: 10 +``` +] + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +class: extra-details + +## Applying changes through a YAML patch + +- We could use `kubectl edit deployment worker` + +- But we could also use `kubectl patch` with the exact YAML shown before + +.lab[ + +.small[ + +- Apply all our changes and wait for them to take effect: + ```bash + kubectl patch deployment worker -p " + spec: + template: + spec: + containers: + - name: worker + image: dockercoins/worker:v0.1 + strategy: + rollingUpdate: + maxUnavailable: 0 + maxSurge: 1 + minReadySeconds: 10 + " + kubectl rollout status deployment worker + kubectl get deploy -o json worker | + jq "{name:.metadata.name} + .spec.strategy.rollingUpdate" + ``` + ] + +] + +??? + +:EN:- Rolling updates +:EN:- Rolling back a bad deployment + +:FR:- Mettre à jour un déploiement +:FR:- Concept de *rolling update* et *rollback* +:FR:- Paramétrer la vitesse de déploiement + +.debug[[k8s/rollout.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/rollout.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/two-containers-on-a-truck.jpg)] + +--- + +name: toc-deploying-with-yaml +class: title + + Deploying with YAML + +.nav[ +[Previous part](#toc-rolling-updates) +| +[Back to table of contents](#toc-part-2) +| +[Next part](#toc-daemon-sets) +] + +.debug[(automatically generated title slide)] + +--- +# Deploying with YAML + +- So far, we created resources with the following commands: + + - `kubectl run` + + - `kubectl create deployment` + + - `kubectl expose` + +- We can also create resources directly with YAML manifests + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## `kubectl apply` vs `create` + +- `kubectl create -f whatever.yaml` + + - creates resources if they don't exist + + - if resources already exist, don't alter them +
(and display error message) + +- `kubectl apply -f whatever.yaml` + + - creates resources if they don't exist + + - if resources already exist, update them +
(to match the definition provided by the YAML file) + + - stores the manifest as an *annotation* in the resource + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## Creating multiple resources + +- The manifest can contain multiple resources separated by `---` + +```yaml + kind: ... + apiVersion: ... + metadata: ... + name: ... + ... + --- + kind: ... + apiVersion: ... + metadata: ... + name: ... + ... +``` + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## Creating multiple resources + +- The manifest can also contain a list of resources + +```yaml + apiVersion: v1 + kind: List + items: + - kind: ... + apiVersion: ... + ... + - kind: ... + apiVersion: ... + ... +``` + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## Deploying dockercoins with YAML + +- We provide a YAML manifest with all the resources for Dockercoins + + (Deployments and Services) + +- We can use it if we need to deploy or redeploy Dockercoins + +.lab[ + +- Deploy or redeploy Dockercoins: + ```bash + kubectl apply -f ~/container.training/k8s/dockercoins.yaml + ``` + +] + +(If we deployed Dockercoins earlier, we will see warning messages, +because the resources that we created lack the necessary annotation. +We can safely ignore them.) + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## Deleting resources + +- We can also use a YAML file to *delete* resources + +- `kubectl delete -f ...` will delete all the resources mentioned in a YAML file + + (useful to clean up everything that was created by `kubectl apply -f ...`) + +- The definitions of the resources don't matter + + (just their `kind`, `apiVersion`, and `name`) + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## Pruning¹ resources + +- We can also tell `kubectl` to remove old resources + +- This is done with `kubectl apply -f ... --prune` + +- It will remove resources that don't exist in the YAML file(s) + +- But only if they were created with `kubectl apply` in the first place + + (technically, if they have an annotation `kubectl.kubernetes.io/last-applied-configuration`) + +.footnote[¹If English is not your first language: *to prune* means to remove dead or overgrown branches in a tree, to help it to grow.] + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +## YAML as source of truth + +- Imagine the following workflow: + + - do not use `kubectl run`, `kubectl create deployment`, `kubectl expose` ... + + - define everything with YAML + + - `kubectl apply -f ... --prune --all` that YAML + + - keep that YAML under version control + + - enforce all changes to go through that YAML (e.g. with pull requests) + +- Our version control system now has a full history of what we deploy + +- Compares to "Infrastructure-as-Code", but for app deployments + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +class: extra-details + +## Specifying the namespace + +- When creating resources from YAML manifests, the namespace is optional + +- If we specify a namespace: + + - resources are created in the specified namespace + + - this is typical for things deployed only once per cluster + + - example: system components, cluster add-ons ... + +- If we don't specify a namespace: + + - resources are created in the current namespace + + - this is typical for things that may be deployed multiple times + + - example: applications (production, staging, feature branches ...) + +??? + +:EN:- Deploying with YAML manifests +:FR:- Déployer avec des *manifests* YAML + +.debug[[k8s/yamldeploy.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/yamldeploy.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/wall-of-containers.jpeg)] + +--- + +name: toc-daemon-sets +class: title + + Daemon sets + +.nav[ +[Previous part](#toc-deploying-with-yaml) +| +[Back to table of contents](#toc-part-3) +| +[Next part](#toc-labels-and-selectors) +] + +.debug[(automatically generated title slide)] + +--- +# Daemon sets + +- We want to scale `rng` in a way that is different from how we scaled `worker` + +- We want one (and exactly one) instance of `rng` per node + +- We *do not want* two instances of `rng` on the same node + +- We will do that with a *daemon set* + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Why not a deployment? + +- Can't we just do `kubectl scale deployment rng --replicas=...`? + +-- + +- Nothing guarantees that the `rng` containers will be distributed evenly + +- If we add nodes later, they will not automatically run a copy of `rng` + +- If we remove (or reboot) a node, one `rng` container will restart elsewhere + + (and we will end up with two instances `rng` on the same node) + +- By contrast, a daemon set will start one pod per node and keep it that way + + (as nodes are added or removed) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Daemon sets in practice + +- Daemon sets are great for cluster-wide, per-node processes: + + - `kube-proxy` + + - `weave` (our overlay network) + + - monitoring agents + + - hardware management tools (e.g. SCSI/FC HBA agents) + + - etc. + +- They can also be restricted to run [only on some nodes](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#running-pods-on-only-some-nodes) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Creating a daemon set + + + +- Unfortunately, as of Kubernetes 1.19, the CLI cannot create daemon sets + +-- + +- More precisely: it doesn't have a subcommand to create a daemon set + +-- + +- But any kind of resource can always be created by providing a YAML description: + ```bash + kubectl apply -f foo.yaml + ``` + +-- + +- How do we create the YAML file for our daemon set? + +-- + + - option 1: [read the docs](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#create-a-daemonset) + +-- + + - option 2: `vi` our way out of it + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Creating the YAML file for our daemon set + +- Let's start with the YAML file for the current `rng` resource + +.lab[ + +- Dump the `rng` resource in YAML: + ```bash + kubectl get deploy/rng -o yaml >rng.yml + ``` + +- Edit `rng.yml` + +] + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## "Casting" a resource to another + +- What if we just changed the `kind` field? + + (It can't be that easy, right?) + +.lab[ + +- Change `kind: Deployment` to `kind: DaemonSet` + + + +- Save, quit + +- Try to create our new resource: + ```bash + kubectl apply -f rng.yml + ``` + + + +] + +-- + +We all knew this couldn't be that easy, right! + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Understanding the problem + +- The core of the error is: + ``` + error validating data: + [ValidationError(DaemonSet.spec): + unknown field "replicas" in io.k8s.api.extensions.v1beta1.DaemonSetSpec, + ... + ``` + +-- + +- *Obviously,* it doesn't make sense to specify a number of replicas for a daemon set + +-- + +- Workaround: fix the YAML + + - remove the `replicas` field + - remove the `strategy` field (which defines the rollout mechanism for a deployment) + - remove the `progressDeadlineSeconds` field (also used by the rollout mechanism) + - remove the `status: {}` line at the end + +-- + +- Or, we could also ... + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Use the `--force`, Luke + +- We could also tell Kubernetes to ignore these errors and try anyway + +- The `--force` flag's actual name is `--validate=false` + +.lab[ + +- Try to load our YAML file and ignore errors: + ```bash + kubectl apply -f rng.yml --validate=false + ``` + +] + +-- + +🎩✨🐇 + +-- + +Wait ... Now, can it be *that* easy? + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Checking what we've done + +- Did we transform our `deployment` into a `daemonset`? + +.lab[ + +- Look at the resources that we have now: + ```bash + kubectl get all + ``` + +] + +-- + +We have two resources called `rng`: + +- the *deployment* that was existing before + +- the *daemon set* that we just created + +We also have one too many pods. +
+(The pod corresponding to the *deployment* still exists.) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## `deploy/rng` and `ds/rng` + +- You can have different resource types with the same name + + (i.e. a *deployment* and a *daemon set* both named `rng`) + +- We still have the old `rng` *deployment* + + ``` +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +deployment.apps/rng 1 1 1 1 18m + ``` + +- But now we have the new `rng` *daemon set* as well + + ``` +NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE +daemonset.apps/rng 2 2 2 2 2 9s + ``` + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Too many pods + +- If we check with `kubectl get pods`, we see: + + - *one pod* for the deployment (named `rng-xxxxxxxxxx-yyyyy`) + + - *one pod per node* for the daemon set (named `rng-zzzzz`) + + ``` + NAME READY STATUS RESTARTS AGE + rng-54f57d4d49-7pt82 1/1 Running 0 11m + rng-b85tm 1/1 Running 0 25s + rng-hfbrr 1/1 Running 0 25s + [...] + ``` + +-- + +The daemon set created one pod per node, except on the master node. + +The master node has [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) preventing pods from running there. + +(To schedule a pod on this node anyway, the pod will require appropriate [tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/).) + +.footnote[(Off by one? We don't run these pods on the node hosting the control plane.)] + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Is this working? + +- Look at the web UI + +-- + +- The graph should now go above 10 hashes per second! + +-- + +- It looks like the newly created pods are serving traffic correctly + +- How and why did this happen? + + (We didn't do anything special to add them to the `rng` service load balancer!) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/Container-Ship-Freighter-Navigation-Elbe-Romance-1782991.jpg)] + +--- + +name: toc-labels-and-selectors +class: title + + Labels and selectors + +.nav[ +[Previous part](#toc-daemon-sets) +| +[Back to table of contents](#toc-part-3) +| +[Next part](#toc-namespaces) +] + +.debug[(automatically generated title slide)] + +--- + +# Labels and selectors + +- The `rng` *service* is load balancing requests to a set of pods + +- That set of pods is defined by the *selector* of the `rng` service + +.lab[ + +- Check the *selector* in the `rng` service definition: + ```bash + kubectl describe service rng + ``` + +] + +- The selector is `app=rng` + +- It means "all the pods having the label `app=rng`" + + (They can have additional labels as well, that's OK!) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Selector evaluation + +- We can use selectors with many `kubectl` commands + +- For instance, with `kubectl get`, `kubectl logs`, `kubectl delete` ... and more + +.lab[ + +- Get the list of pods matching selector `app=rng`: + ```bash + kubectl get pods -l app=rng + kubectl get pods --selector app=rng + ``` + +] + +But ... why do these pods (in particular, the *new* ones) have this `app=rng` label? + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Where do labels come from? + +- When we create a deployment with `kubectl create deployment rng`, +
this deployment gets the label `app=rng` + +- The replica sets created by this deployment also get the label `app=rng` + +- The pods created by these replica sets also get the label `app=rng` + +- When we created the daemon set from the deployment, we re-used the same spec + +- Therefore, the pods created by the daemon set get the same labels + +.footnote[Note: when we use `kubectl run stuff`, the label is `run=stuff` instead.] + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Updating load balancer configuration + +- We would like to remove a pod from the load balancer + +- What would happen if we removed that pod, with `kubectl delete pod ...`? + +-- + + It would be re-created immediately (by the replica set or the daemon set) + +-- + +- What would happen if we removed the `app=rng` label from that pod? + +-- + + It would *also* be re-created immediately + +-- + + Why?!? + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Selectors for replica sets and daemon sets + +- The "mission" of a replica set is: + + "Make sure that there is the right number of pods matching this spec!" + +- The "mission" of a daemon set is: + + "Make sure that there is a pod matching this spec on each node!" + +-- + +- *In fact,* replica sets and daemon sets do not check pod specifications + +- They merely have a *selector*, and they look for pods matching that selector + +- Yes, we can fool them by manually creating pods with the "right" labels + +- Bottom line: if we remove our `app=rng` label ... + + ... The pod "disappears" for its parent, which re-creates another pod to replace it + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Isolation of replica sets and daemon sets + +- Since both the `rng` daemon set and the `rng` replica set use `app=rng` ... + + ... Why don't they "find" each other's pods? + +-- + +- *Replica sets* have a more specific selector, visible with `kubectl describe` + + (It looks like `app=rng,pod-template-hash=abcd1234`) + +- *Daemon sets* also have a more specific selector, but it's invisible + + (It looks like `app=rng,controller-revision-hash=abcd1234`) + +- As a result, each controller only "sees" the pods it manages + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Removing a pod from the load balancer + +- Currently, the `rng` service is defined by the `app=rng` selector + +- The only way to remove a pod is to remove or change the `app` label + +- ... But that will cause another pod to be created instead! + +- What's the solution? + +-- + +- We need to change the selector of the `rng` service! + +- Let's add another label to that selector (e.g. `active=yes`) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Selectors with multiple labels + +- If a selector specifies multiple labels, they are understood as a logical *AND* + + (in other words: the pods must match all the labels) + +- We cannot have a logical *OR* + + (e.g. `app=api AND (release=prod OR release=preprod)`) + +- We can, however, apply as many extra labels as we want to our pods: + + - use selector `app=api AND prod-or-preprod=yes` + + - add `prod-or-preprod=yes` to both sets of pods + +- We will see later that in other places, we can use more advanced selectors + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## The plan + +1. Add the label `active=yes` to all our `rng` pods + +2. Update the selector for the `rng` service to also include `active=yes` + +3. Toggle traffic to a pod by manually adding/removing the `active` label + +4. Profit! + +*Note: if we swap steps 1 and 2, it will cause a short +service disruption, because there will be a period of time +during which the service selector won't match any pod. +During that time, requests to the service will time out. +By doing things in the order above, we guarantee that there won't +be any interruption.* + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Adding labels to pods + +- We want to add the label `active=yes` to all pods that have `app=rng` + +- We could edit each pod one by one with `kubectl edit` ... + +- ... Or we could use `kubectl label` to label them all + +- `kubectl label` can use selectors itself + +.lab[ + +- Add `active=yes` to all pods that have `app=rng`: + ```bash + kubectl label pods -l app=rng active=yes + ``` + +] + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Updating the service selector + +- We need to edit the service specification + +- Reminder: in the service definition, we will see `app: rng` in two places + + - the label of the service itself (we don't need to touch that one) + + - the selector of the service (that's the one we want to change) + +.lab[ + +- Update the service to add `active: yes` to its selector: + ```bash + kubectl edit service rng + ``` + + + +] + +-- + +... And then we get *the weirdest error ever.* Why? + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## When the YAML parser is being too smart + +- YAML parsers try to help us: + + - `xyz` is the string `"xyz"` + + - `42` is the integer `42` + + - `yes` is the boolean value `true` + +- If we want the string `"42"` or the string `"yes"`, we have to quote them + +- So we have to use `active: "yes"` + +.footnote[For a good laugh: if we had used "ja", "oui", "si" ... as the value, it would have worked!] + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Updating the service selector, take 2 + +.lab[ + +- Update the YAML manifest of the service + +- Add `active: "yes"` to its selector + + + +] + +This time it should work! + +If we did everything correctly, the web UI shouldn't show any change. + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Updating labels + +- We want to disable the pod that was created by the deployment + +- All we have to do, is remove the `active` label from that pod + +- To identify that pod, we can use its name + +- ... Or rely on the fact that it's the only one with a `pod-template-hash` label + +- Good to know: + + - `kubectl label ... foo=` doesn't remove a label (it sets it to an empty string) + + - to remove label `foo`, use `kubectl label ... foo-` + + - to change an existing label, we would need to add `--overwrite` + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Removing a pod from the load balancer + +.lab[ + +- In one window, check the logs of that pod: + ```bash + POD=$(kubectl get pod -l app=rng,pod-template-hash -o name) + kubectl logs --tail 1 --follow $POD + ``` + (We should see a steady stream of HTTP logs) + + + +- In another window, remove the label from the pod: + ```bash + kubectl label pod -l app=rng,pod-template-hash active- + ``` + (The stream of HTTP logs should stop immediately) + + + +] + +There might be a slight change in the web UI (since we removed a bit +of capacity from the `rng` service). If we remove more pods, +the effect should be more visible. + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Updating the daemon set + +- If we scale up our cluster by adding new nodes, the daemon set will create more pods + +- These pods won't have the `active=yes` label + +- If we want these pods to have that label, we need to edit the daemon set spec + +- We can do that with e.g. `kubectl edit daemonset rng` + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## We've put resources in your resources + +- Reminder: a daemon set is a resource that creates more resources! + +- There is a difference between: + + - the label(s) of a resource (in the `metadata` block in the beginning) + + - the selector of a resource (in the `spec` block) + + - the label(s) of the resource(s) created by the first resource (in the `template` block) + +- We would need to update the selector and the template + + (metadata labels are not mandatory) + +- The template must match the selector + + (i.e. the resource will refuse to create resources that it will not select) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Labels and debugging + +- When a pod is misbehaving, we can delete it: another one will be recreated + +- But we can also change its labels + +- It will be removed from the load balancer (it won't receive traffic anymore) + +- Another pod will be recreated immediately + +- But the problematic pod is still here, and we can inspect and debug it + +- We can even re-add it to the rotation if necessary + + (Very useful to troubleshoot intermittent and elusive bugs) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +## Labels and advanced rollout control + +- Conversely, we can add pods matching a service's selector + +- These pods will then receive requests and serve traffic + +- Examples: + + - one-shot pod with all debug flags enabled, to collect logs + + - pods created automatically, but added to rotation in a second step +
+ (by setting their label accordingly) + +- This gives us building blocks for canary and blue/green deployments + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Advanced label selectors + +- As indicated earlier, service selectors are limited to a `AND` + +- But in many other places in the Kubernetes API, we can use complex selectors + + (e.g. Deployment, ReplicaSet, DaemonSet, NetworkPolicy ...) + +- These allow extra operations; specifically: + + - checking for presence (or absence) of a label + + - checking if a label is (or is not) in a given set + +- Relevant documentation: + + [Service spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#servicespec-v1-core), + [LabelSelector spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#labelselector-v1-meta), + [label selector doc](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Example of advanced selector + +```yaml + theSelector: + matchLabels: + app: portal + component: api + matchExpressions: + - key: release + operator: In + values: [ production, preproduction ] + - key: signed-off-by + operator: Exists +``` + +This selector matches pods that meet *all* the indicated conditions. + +`operator` can be `In`, `NotIn`, `Exists`, `DoesNotExist`. + +A `nil` selector matches *nothing*, a `{}` selector matches *everything*. +
+(Because that means "match all pods that meet at least zero condition".) + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Services and Endpoints + +- Each Service has a corresponding Endpoints resource + + (see `kubectl get endpoints` or `kubectl get ep`) + +- That Endpoints resource is used by various controllers + + (e.g. `kube-proxy` when setting up `iptables` rules for ClusterIP services) + +- These Endpoints are populated (and updated) with the Service selector + +- We can update the Endpoints manually, but our changes will get overwritten + +- ... Except if the Service selector is empty! + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: extra-details + +## Empty Service selector + +- If a service selector is empty, Endpoints don't get updated automatically + + (but we can still set them manually) + +- This lets us create Services pointing to arbitrary destinations + + (potentially outside the cluster; or things that are not in pods) + +- Another use-case: the `kubernetes` service in the `default` namespace + + (its Endpoints are maintained automatically by the API server) + +??? + +:EN:- Scaling with Daemon Sets +:FR:- Utilisation de Daemon Sets + +.debug[[k8s/daemonset.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/daemonset.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/ShippingContainerSFBay.jpg)] + +--- + +name: toc-namespaces +class: title + + Namespaces + +.nav[ +[Previous part](#toc-labels-and-selectors) +| +[Back to table of contents](#toc-part-3) +| +[Next part](#toc-healthchecks) +] + +.debug[(automatically generated title slide)] + +--- +# Namespaces + +- We would like to deploy another copy of DockerCoins on our cluster + +- We could rename all our deployments and services: + + hasher → hasher2, redis → redis2, rng → rng2, etc. + +- That would require updating the code + +- There has to be a better way! + +-- + +- As hinted by the title of this section, we will use *namespaces* + +.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] +--- + +## Identifying a resource + +- We cannot have two resources with the same name + + (or can we...?) + +-- + +- We cannot have two resources *of the same kind* with the same name + + (but it's OK to have an `rng` service, an `rng` deployment, and an `rng` daemon set) + +-- + +- We cannot have two resources of the same kind with the same name *in the same namespace* + + (but it's OK to have e.g. two `rng` services in different namespaces) + +-- + +- Except for resources that exist at the *cluster scope* + + (these do not belong to a namespace) + +.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] +--- + +## Uniquely identifying a resource + +- For *namespaced* resources: + + the tuple *(kind, name, namespace)* needs to be unique + +- For resources at the *cluster scope*: + + the tuple *(kind, name)* needs to be unique + +.lab[ + +- List resource types again, and check the NAMESPACED column: + ```bash + kubectl api-resources + ``` + +] + +.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] +--- + +## Pre-existing namespaces + +- If we deploy a cluster with `kubeadm`, we have three or four namespaces: + + - `default` (for our applications) + + - `kube-system` (for the control plane) + + - `kube-public` (contains one ConfigMap for cluster discovery) + + - `kube-node-lease` (in Kubernetes 1.14 and later; contains Lease objects) + +- If we deploy differently, we may have different namespaces + +.debug[[k8s/namespaces.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/namespaces.md)] +--- + +## Creating namespaces + +- Let's see two identical methods to create a namespace + +.lab[ + +- We can use `kubectl create namespace`: + ```bash + kubectl create namespace blue + ``` + +- Or we can construct a very minimal YAML snippet: + ```bash + kubectl apply -f- <and it's not particularly fast (relies on userland proxying or iptables) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +## Kubernetes network model: in practice + +- The nodes that we are using have been set up to use [Weave](https://github.com/weaveworks/weave) + +- We don't endorse Weave in a particular way, it just Works For Us + +- Don't worry about the warning about `kube-proxy` performance + +- Unless you: + + - routinely saturate 10G network interfaces + - count packet rates in millions per second + - run high-traffic VOIP or gaming platforms + - do weird things that involve millions of simultaneous connections +
(in which case you're already familiar with kernel tuning) + +- If necessary, there are alternatives to `kube-proxy`; e.g. + [`kube-router`](https://www.kube-router.io) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: extra-details + +## The Container Network Interface (CNI) + +- Most Kubernetes clusters use CNI "plugins" to implement networking + +- When a pod is created, Kubernetes delegates the network setup to these plugins + + (it can be a single plugin, or a combination of plugins, each doing one task) + +- Typically, CNI plugins will: + + - allocate an IP address (by calling an IPAM plugin) + + - add a network interface into the pod's network namespace + + - configure the interface as well as required routes etc. + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: extra-details + +## Multiple moving parts + +- The "pod-to-pod network" or "pod network": + + - provides communication between pods and nodes + + - is generally implemented with CNI plugins + +- The "pod-to-service network": + + - provides internal communication and load balancing + + - is generally implemented with kube-proxy (or e.g. kube-router) + +- Network policies: + + - provide firewalling and isolation + + - can be bundled with the "pod network" or provided by another component + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +![Overview of the three Kubernetes network layers](images/k8s-net-0-overview.svg) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +![Pod-to-pod network](images/k8s-net-1-pod-to-pod.svg) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +![Pod-to-service network](images/k8s-net-2-pod-to-svc.svg) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +![Network policies](images/k8s-net-3-netpol.svg) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +![View with all the layers again](images/k8s-net-4-overview.svg) + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: extra-details + +## Even more moving parts + +- Inbound traffic can be handled by multiple components: + + - something like kube-proxy or kube-router (for NodePort services) + + - load balancers (ideally, connected to the pod network) + +- It is possible to use multiple pod networks in parallel + + (with "meta-plugins" like CNI-Genie or Multus) + +- Some solutions can fill multiple roles + + (e.g. kube-router can be set up to provide the pod network and/or network policies and/or replace kube-proxy) + +??? + +:EN:- The Kubernetes network model +:FR:- Le modèle réseau de Kubernetes + +.debug[[k8s/kubenet.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/kubenet.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] + +--- + +name: toc-container-based-software-deployment +class: title + + Container Based Software Deployment + +.nav[ +[Previous part](#toc-kubernetes-network-model) +| +[Back to table of contents](#toc-part-3) +| +[Next part](#toc-volumes) +] + +.debug[(automatically generated title slide)] + +--- +# Container Based Software Deployment + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-1.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-2.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-3.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-4.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-5.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-6.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-7.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-8.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-9.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-10.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-11.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-12.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-13.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-14.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-15.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-16.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + + +![dummmy](containers/software-deployment/slide-17.jpg) + +.debug[[containers/software-deployment.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/containers/software-deployment.md)] +--- + +class: pic + .interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] --- @@ -7636,7 +7621,7 @@ class: title Volumes .nav[ -[Previous part](#toc-exposing-containers) +[Previous part](#toc-container-based-software-deployment) | [Back to table of contents](#toc-part-3) | @@ -8840,7 +8825,7 @@ class: title | [Back to table of contents](#toc-part-3) | -[Next part](#toc-exposing-http-services-with-ingress-resources) +[Next part](#toc-exercise--healthchecks) ] .debug[(automatically generated title slide)] @@ -9158,17 +9143,766 @@ class: pic --- -name: toc-exposing-http-services-with-ingress-resources +name: toc-exercise--healthchecks class: title - Exposing HTTP services with Ingress resources + Exercise — Healthchecks .nav[ [Previous part](#toc-managing-secrets) | [Back to table of contents](#toc-part-3) | -[Next part](#toc-exercise--ingress) +[Next part](#toc-exercise--application-configuration) +] + +.debug[(automatically generated title slide)] + +--- +# Exercise — Healthchecks + +- We want to add healthchecks to the `rng` service in dockercoins + +- The `rng` service exhibits an interesting behavior under load: + + *its latency increases (which will cause probes to time out!)* + +- We want to see: + + - what happens when the readiness probe fails + + - what happens when the liveness probe fails + + - how to set "appropriate" probes and probe parameters + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +## Setup + +- First, deploy a new copy of dockercoins + + (for instance, in a brand new namespace) + +- Pro tip #1: ping (e.g. with `httping`) the `rng` service at all times + + - it should initially show a few milliseconds latency + + - that will increase when we scale up + + - it will also let us detect when the service goes "boom" + +- Pro tip #2: also keep an eye on the web UI + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +## Readiness + +- Add a readiness probe to `rng` + + - this requires editing the pod template in the Deployment manifest + + - use a simple HTTP check on the `/` route of the service + + - keep all other parameters (timeouts, thresholds...) at their default values + +- Check what happens when deploying an invalid image for `rng` (e.g. `alpine`) + +*(If the probe was set up correctly, the app will continue to work, +because Kubernetes won't switch over the traffic to the `alpine` containers, +because they don't pass the readiness probe.)* + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +## Readiness under load + +- Then roll back `rng` to the original image + +- Check what happens when we scale up the `worker` Deployment to 15+ workers + + (get the latency above 1 second) + +*(We should now observe intermittent unavailability of the service, i.e. every +30 seconds it will be unreachable for a bit, then come back, then go away again, etc.)* + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +## Liveness + +- Now replace the readiness probe with a liveness probe + +- What happens now? + +*(At first the behavior looks the same as with the readiness probe: +service becomes unreachable, then reachable again, etc.; but there is +a significant difference behind the scenes. What is it?)* + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +## Readiness and liveness + +- Bonus questions! + +- What happens if we enable both probes at the same time? + +- What strategies can we use so that both probes are useful? + +.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/lots-of-containers.jpg)] + +--- + +name: toc-exercise--application-configuration +class: title + + Exercise — Application Configuration + +.nav[ +[Previous part](#toc-exercise--healthchecks) +| +[Back to table of contents](#toc-part-3) +| +[Next part](#toc-network-policies) +] + +.debug[(automatically generated title slide)] + +--- +# Exercise — Application Configuration + +- We want to configure an application with a ConfigMap + +- We will use the "rainbow" example shown previously + + (HAProxy load balancing traffic to services in multiple namespaces) + +- We won't provide the HAProxy configuration file + +- Instead, we will provide a list of namespaces + + (e.g. as a space-delimited list in a ConfigMap) + +- Our Pod should generate the HAProxy configuration using the ConfigMap + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +## Setup + +- Let's say that we have the "rainbow" app deployed: + ```bash + kubectl apply -f ~/container.training/k8s/rainbow.yaml + ``` + +- And a ConfigMap like the following one: + ```bash + kubectl create configmap rainbow --from-literal=namespaces="blue green" + ``` + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +## Goal 1 + +- We want a Deployment and a Service called `rainbow` + +- The `rainbow` Service should load balance across Namespaces `blue` and `green` + + (i.e. to the Services called `color` in both these Namespaces) + +- We want to be able to update the configuration: + + - update the ConfigMap to put `blue green red` + + - what should we do so that HAproxy picks up the change? + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +## Goal 2 + +- Check what happens if we specify a backend that doesn't exist + + (e.g. add `purple` to the list of namespaces) + +- If we specify invalid backends to HAProxy, it won't start! + +- Implement a workaround among these two: + + - remove invalid backends from the list before starting HAProxy + + - wait until all backends are valid before starting HAProxy + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +## Goal 3 + +- We'd like HAProxy to pick up ConfigMap updates automatically + +- How can we do that? + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +## Hints + +- Check the following slides if you need help! + +-- + +- We want to generate the HAProxy configuration in an `initContainer` + +-- + +- The `namespaces` entry of the `rainbow` ConfigMap should be exposed to the `initContainer` + +-- + +- The HAProxy configuration should be in a volume shared with HAProxy + +.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] + +--- + +name: toc-network-policies +class: title + + Network policies + +.nav[ +[Previous part](#toc-exercise--application-configuration) +| +[Back to table of contents](#toc-part-4) +| +[Next part](#toc-exposing-http-services-with-ingress-resources) +] + +.debug[(automatically generated title slide)] + +--- +# Network policies + +- Namespaces help us to *organize* resources + +- Namespaces do not provide isolation + +- By default, every pod can contact every other pod + +- By default, every service accepts traffic from anyone + +- If we want this to be different, we need *network policies* + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## What's a network policy? + +A network policy is defined by the following things. + +- A *pod selector* indicating which pods it applies to + + e.g.: "all pods in namespace `blue` with the label `zone=internal`" + +- A list of *ingress rules* indicating which inbound traffic is allowed + + e.g.: "TCP connections to ports 8000 and 8080 coming from pods with label `zone=dmz`, + and from the external subnet 4.42.6.0/24, except 4.42.6.5" + +- A list of *egress rules* indicating which outbound traffic is allowed + +A network policy can provide ingress rules, egress rules, or both. + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## How do network policies apply? + +- A pod can be "selected" by any number of network policies + +- If a pod isn't selected by any network policy, then its traffic is unrestricted + + (In other words: in the absence of network policies, all traffic is allowed) + +- If a pod is selected by at least one network policy, then all traffic is blocked ... + + ... unless it is explicitly allowed by one of these network policies + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +class: extra-details + +## Traffic filtering is flow-oriented + +- Network policies deal with *connections*, not individual packets + +- Example: to allow HTTP (80/tcp) connections to pod A, you only need an ingress rule + + (You do not need a matching egress rule to allow response traffic to go through) + +- This also applies for UDP traffic + + (Allowing DNS traffic can be done with a single rule) + +- Network policy implementations use stateful connection tracking + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Pod-to-pod traffic + +- Connections from pod A to pod B have to be allowed by both pods: + + - pod A has to be unrestricted, or allow the connection as an *egress* rule + + - pod B has to be unrestricted, or allow the connection as an *ingress* rule + +- As a consequence: if a network policy restricts traffic going from/to a pod, +
+ the restriction cannot be overridden by a network policy selecting another pod + +- This prevents an entity managing network policies in namespace A + (but without permission to do so in namespace B) + from adding network policies giving them access to namespace B + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## The rationale for network policies + +- In network security, it is generally considered better to "deny all, then allow selectively" + + (The other approach, "allow all, then block selectively" makes it too easy to leave holes) + +- As soon as one network policy selects a pod, the pod enters this "deny all" logic + +- Further network policies can open additional access + +- Good network policies should be scoped as precisely as possible + +- In particular: make sure that the selector is not too broad + + (Otherwise, you end up affecting pods that were otherwise well secured) + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Our first network policy + +This is our game plan: + +- run a web server in a pod + +- create a network policy to block all access to the web server + +- create another network policy to allow access only from specific pods + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Running our test web server + +.lab[ + +- Let's use the `nginx` image: + ```bash + kubectl create deployment testweb --image=nginx + ``` + + + +- Find out the IP address of the pod with one of these two commands: + ```bash + kubectl get pods -o wide -l app=testweb + IP=$(kubectl get pods -l app=testweb -o json | jq -r .items[0].status.podIP) + ``` + +- Check that we can connect to the server: + ```bash + curl $IP + ``` +] + +The `curl` command should show us the "Welcome to nginx!" page. + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Adding a very restrictive network policy + +- The policy will select pods with the label `app=testweb` + +- It will specify an empty list of ingress rules (matching nothing) + +.lab[ + +- Apply the policy in this YAML file: + ```bash + kubectl apply -f ~/container.training/k8s/netpol-deny-all-for-testweb.yaml + ``` + +- Check if we can still access the server: + ```bash + curl $IP + ``` + + + +] + +The `curl` command should now time out. + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Looking at the network policy + +This is the file that we applied: + +```yaml +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: deny-all-for-testweb +spec: + podSelector: + matchLabels: + app: testweb + ingress: [] + + +``` + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Allowing connections only from specific pods + +- We want to allow traffic from pods with the label `run=testcurl` + +- Reminder: this label is automatically applied when we do `kubectl run testcurl ...` + +.lab[ + +- Apply another policy: + ```bash + kubectl apply -f ~/container.training/k8s/netpol-allow-testcurl-for-testweb.yaml + ``` + +] + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Looking at the network policy + +This is the second file that we applied: + +```yaml +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: allow-testcurl-for-testweb +spec: + podSelector: + matchLabels: + app: testweb + ingress: + - from: + - podSelector: + matchLabels: + run: testcurl + + +``` + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Testing the network policy + +- Let's create pods with, and without, the required label + +.lab[ + +- Try to connect to testweb from a pod with the `run=testcurl` label: + ```bash + kubectl run testcurl --rm -i --image=centos -- curl -m3 $IP + ``` + +- Try to connect to testweb with a different label: + ```bash + kubectl run testkurl --rm -i --image=centos -- curl -m3 $IP + ``` + +] + +The first command will work (and show the "Welcome to nginx!" page). + +The second command will fail and time out after 3 seconds. + +(The timeout is obtained with the `-m3` option.) + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## An important warning + +- Some network plugins only have partial support for network policies + +- For instance, Weave added support for egress rules [in version 2.4](https://github.com/weaveworks/weave/pull/3313) (released in July 2018) + +- But only recently added support for ipBlock [in version 2.5](https://github.com/weaveworks/weave/pull/3367) (released in Nov 2018) + +- Unsupported features might be silently ignored + + (Making you believe that you are secure, when you're not) + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Network policies, pods, and services + +- Network policies apply to *pods* + +- A *service* can select multiple pods + + (And load balance traffic across them) + +- It is possible that we can connect to some pods, but not some others + + (Because of how network policies have been defined for these pods) + +- In that case, connections to the service will randomly pass or fail + + (Depending on whether the connection was sent to a pod that we have access to or not) + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Network policies and namespaces + +- A good strategy is to isolate a namespace, so that: + + - all the pods in the namespace can communicate together + + - other namespaces cannot access the pods + + - external access has to be enabled explicitly + +- Let's see what this would look like for the DockerCoins app! + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Network policies for DockerCoins + +- We are going to apply two policies + +- The first policy will prevent traffic from other namespaces + +- The second policy will allow traffic to the `webui` pods + +- That's all we need for that app! + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Blocking traffic from other namespaces + +This policy selects all pods in the current namespace. + +It allows traffic only from pods in the current namespace. + +(An empty `podSelector` means "all pods.") + +```yaml +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: deny-from-other-namespaces +spec: + podSelector: {} + ingress: + - from: + - podSelector: {} +``` + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Allowing traffic to `webui` pods + +This policy selects all pods with label `app=webui`. + +It allows traffic from any source. + +(An empty `from` field means "all sources.") + +```yaml +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: allow-webui +spec: + podSelector: + matchLabels: + app: webui + ingress: + - from: [] +``` + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Applying both network policies + +- Both network policies are declared in the file [k8s/netpol-dockercoins.yaml](https://github.com/jpetazzo/container.training/tree/master/k8s/netpol-dockercoins.yaml) + +.lab[ + +- Apply the network policies: + ```bash + kubectl apply -f ~/container.training/k8s/netpol-dockercoins.yaml + ``` + +- Check that we can still access the web UI from outside +
+ (and that the app is still working correctly!) + +- Check that we can't connect anymore to `rng` or `hasher` through their ClusterIP + +] + +Note: using `kubectl proxy` or `kubectl port-forward` allows us to connect +regardless of existing network policies. This allows us to debug and +troubleshoot easily, without having to poke holes in our firewall. + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Cleaning up our network policies + +- The network policies that we have installed block all traffic to the default namespace + +- We should remove them, otherwise further demos and exercises will fail! + +.lab[ + +- Remove all network policies: + ```bash + kubectl delete networkpolicies --all + ``` + +] + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Protecting the control plane + +- Should we add network policies to block unauthorized access to the control plane? + + (etcd, API server, etc.) + +-- + +- At first, it seems like a good idea ... + +-- + +- But it *shouldn't* be necessary: + + - not all network plugins support network policies + + - the control plane is secured by other methods (mutual TLS, mostly) + + - the code running in our pods can reasonably expect to contact the API +
+ (and it can do so safely thanks to the API permission model) + +- If we block access to the control plane, we might disrupt legitimate code + +- ...Without necessarily improving security + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Tools and resources + +- [Cilium Network Policy Editor](https://editor.cilium.io/) + +- [Tufin Network Policy Viewer](https://orca.tufin.io/netpol/) + +- Two resources by [Ahmet Alp Balkan](https://ahmet.im/): + + - a [very good talk about network policies](https://www.youtube.com/watch?list=PLj6h78yzYM2P-3-xqvmWaZbbI1sW-ulZb&v=3gGpMmYeEO8) at KubeCon North America 2017 + + - a repository of [ready-to-use recipes](https://github.com/ahmetb/kubernetes-network-policy-recipes) for network policies + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +## Documentation + +- As always, the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) is a good starting point + +- The API documentation has a lot of detail about the format of various objects: + + - [NetworkPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicy-v1-networking-k8s-io) + + - [NetworkPolicySpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicyspec-v1-networking-k8s-io) + + - [NetworkPolicyIngressRule](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicyingressrule-v1-networking-k8s-io) + + - etc. + +??? + +:EN:- Isolating workloads with Network Policies +:FR:- Isolation réseau avec les *network policies* + +.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] + +--- + +name: toc-exposing-http-services-with-ingress-resources +class: title + + Exposing HTTP services with Ingress resources + +.nav[ +[Previous part](#toc-network-policies) +| +[Back to table of contents](#toc-part-4) +| +[Next part](#toc-stateful-sets) ] .debug[(automatically generated title slide)] @@ -10018,318 +10752,6 @@ For inspiration, check [flagger by Weave](https://github.com/weaveworks/flagger) class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/lots-of-containers.jpg)] - ---- - -name: toc-exercise--ingress -class: title - - Exercise — Ingress - -.nav[ -[Previous part](#toc-exposing-http-services-with-ingress-resources) -| -[Back to table of contents](#toc-part-3) -| -[Next part](#toc-exercise--healthchecks) -] - -.debug[(automatically generated title slide)] - ---- -# Exercise — Ingress - -- We want to expose a web app through an ingress controller - -- This will require: - - - the web app itself (dockercoins, NGINX, whatever we want) - - - an ingress controller - - - a domain name (`use \*.nip.io` or `\*.localdev.me`) - - - an ingress resource - -.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] ---- - -## Goal - -- We want to be able to access the web app using a URL like: - - http://webapp.localdev.me - - *or* - - http://webapp.A.B.C.D.nip.io - - (where A.B.C.D is the IP address of one of our nodes) - -.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] ---- - -## Hints - -- For the ingress controller, we can use: - - - [ingress-nginx](https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md) - - - the [Traefik Helm chart](https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-helm-chart) - - - the container.training [Traefik DaemonSet](https://raw.githubusercontent.com/jpetazzo/container.training/main/k8s/traefik-v2.yaml) - -- If our cluster supports LoadBalancer Services: easy - - (nothing special to do) - -- For local clusters, things can be more difficult; two options: - - - map localhost:80 to e.g. a NodePort service, and use `\*.localdev.me` - - - use hostNetwork, or ExternalIP, and use `\*.nip.io` - -.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/plastic-containers.JPG)] - ---- - -name: toc-exercise--healthchecks -class: title - - Exercise — Healthchecks - -.nav[ -[Previous part](#toc-exercise--ingress) -| -[Back to table of contents](#toc-part-3) -| -[Next part](#toc-exercise--application-configuration) -] - -.debug[(automatically generated title slide)] - ---- -# Exercise — Healthchecks - -- We want to add healthchecks to the `rng` service in dockercoins - -- The `rng` service exhibits an interesting behavior under load: - - *its latency increases (which will cause probes to time out!)* - -- We want to see: - - - what happens when the readiness probe fails - - - what happens when the liveness probe fails - - - how to set "appropriate" probes and probe parameters - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -## Setup - -- First, deploy a new copy of dockercoins - - (for instance, in a brand new namespace) - -- Pro tip #1: ping (e.g. with `httping`) the `rng` service at all times - - - it should initially show a few milliseconds latency - - - that will increase when we scale up - - - it will also let us detect when the service goes "boom" - -- Pro tip #2: also keep an eye on the web UI - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -## Readiness - -- Add a readiness probe to `rng` - - - this requires editing the pod template in the Deployment manifest - - - use a simple HTTP check on the `/` route of the service - - - keep all other parameters (timeouts, thresholds...) at their default values - -- Check what happens when deploying an invalid image for `rng` (e.g. `alpine`) - -*(If the probe was set up correctly, the app will continue to work, -because Kubernetes won't switch over the traffic to the `alpine` containers, -because they don't pass the readiness probe.)* - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -## Readiness under load - -- Then roll back `rng` to the original image - -- Check what happens when we scale up the `worker` Deployment to 15+ workers - - (get the latency above 1 second) - -*(We should now observe intermittent unavailability of the service, i.e. every -30 seconds it will be unreachable for a bit, then come back, then go away again, etc.)* - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -## Liveness - -- Now replace the readiness probe with a liveness probe - -- What happens now? - -*(At first the behavior looks the same as with the readiness probe: -service becomes unreachable, then reachable again, etc.; but there is -a significant difference behind the scenes. What is it?)* - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -## Readiness and liveness - -- Bonus questions! - -- What happens if we enable both probes at the same time? - -- What strategies can we use so that both probes are useful? - -.debug[[exercises/healthchecks-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/healthchecks-details.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-1.jpg)] - ---- - -name: toc-exercise--application-configuration -class: title - - Exercise — Application Configuration - -.nav[ -[Previous part](#toc-exercise--healthchecks) -| -[Back to table of contents](#toc-part-3) -| -[Next part](#toc-stateful-sets) -] - -.debug[(automatically generated title slide)] - ---- -# Exercise — Application Configuration - -- We want to configure an application with a ConfigMap - -- We will use the "rainbow" example shown previously - - (HAProxy load balancing traffic to services in multiple namespaces) - -- We won't provide the HAProxy configuration file - -- Instead, we will provide a list of namespaces - - (e.g. as a space-delimited list in a ConfigMap) - -- Our Pod should generate the HAProxy configuration using the ConfigMap - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -## Setup - -- Let's say that we have the "rainbow" app deployed: - ```bash - kubectl apply -f ~/container.training/k8s/rainbow.yaml - ``` - -- And a ConfigMap like the following one: - ```bash - kubectl create configmap rainbow --from-literal=namespaces="blue green" - ``` - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -## Goal 1 - -- We want a Deployment and a Service called `rainbow` - -- The `rainbow` Service should load balance across Namespaces `blue` and `green` - - (i.e. to the Services called `color` in both these Namespaces) - -- We want to be able to update the configuration: - - - update the ConfigMap to put `blue green red` - - - what should we do so that HAproxy picks up the change? - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -## Goal 2 - -- Check what happens if we specify a backend that doesn't exist - - (e.g. add `purple` to the list of namespaces) - -- If we specify invalid backends to HAProxy, it won't start! - -- Implement a workaround among these two: - - - remove invalid backends from the list before starting HAProxy - - - wait until all backends are valid before starting HAProxy - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -## Goal 3 - -- We'd like HAProxy to pick up ConfigMap updates automatically - -- How can we do that? - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -## Hints - -- Check the following slides if you need help! - --- - -- We want to generate the HAProxy configuration in an `initContainer` - --- - -- The `namespaces` entry of the `rainbow` ConfigMap should be exposed to the `initContainer` - --- - -- The HAProxy configuration should be in a volume shared with HAProxy - -.debug[[exercises/appconfig-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/appconfig-details.md)] ---- - -class: pic - .interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/train-of-containers-2.jpg)] --- @@ -10340,7 +10762,7 @@ class: title Stateful sets .nav[ -[Previous part](#toc-exercise--application-configuration) +[Previous part](#toc-exposing-http-services-with-ingress-resources) | [Back to table of contents](#toc-part-4) | @@ -11999,7 +12421,7 @@ class: title | [Back to table of contents](#toc-part-4) | -[Next part](#toc-accessing-the-api-with-kubectl-proxy) +[Next part](#toc-exercise--ingress) ] .debug[(automatically generated title slide)] @@ -12170,10 +12592,10 @@ class: pic --- -name: toc-accessing-the-api-with-kubectl-proxy +name: toc-exercise--ingress class: title - Accessing the API with `kubectl proxy` + Exercise — Ingress .nav[ [Previous part](#toc-accessing-internal-services) @@ -12185,6 +12607,155 @@ class: title .debug[(automatically generated title slide)] +--- +# Exercise — Ingress + +- We want to expose a web app through an ingress controller + +- This will require: + + - the web app itself (dockercoins, NGINX, whatever we want) + + - an ingress controller + + - a domain name (`use \*.nip.io` or `\*.localdev.me`) + + - an ingress resource + +.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] +--- + +## Goal + +- We want to be able to access the web app using a URL like: + + http://webapp.localdev.me + + *or* + + http://webapp.A.B.C.D.nip.io + + (where A.B.C.D is the IP address of one of our nodes) + +.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] +--- + +## Hints + +- For the ingress controller, we can use: + + - [ingress-nginx](https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md) + + - the [Traefik Helm chart](https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-helm-chart) + + - the container.training [Traefik DaemonSet](https://raw.githubusercontent.com/jpetazzo/container.training/main/k8s/traefik-v2.yaml) + +- If our cluster supports LoadBalancer Services: easy + + (nothing special to do) + +- For local clusters, things can be more difficult; two options: + + - map localhost:80 to e.g. a NodePort service, and use `\*.localdev.me` + + - use hostNetwork, or ExternalIP, and use `\*.nip.io` + +.debug[[exercises/ingress-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/ingress-details.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] + +--- + +name: toc-exercise--local-cluster +class: title + + Exercise — Local Cluster + +.nav[ +[Previous part](#toc-exercise--ingress) +| +[Back to table of contents](#toc-part-4) +| +[Next part](#toc-accessing-the-api-with-kubectl-proxy) +] + +.debug[(automatically generated title slide)] + +--- +# Exercise — Local Cluster + +- We want to have our own local Kubernetes cluster + + (we can use Docker Desktop, KinD, minikube... anything will do!) + +- Then we want to run a copy of dockercoins on that cluster + +- We want to be able to connect to the web UI + + (we can expose the port, or use port-forward, or whatever) + +.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] +--- + +## Goal + +- Be able to see the dockercoins web UI running on our local cluster + +.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] +--- + +## Hints + +- On a Mac or Windows machine: + + the easiest solution is probably Docker Desktop + +- On a Linux machine: + + the easiest solution is probably KinD or k3d + +- To connect to the web UI: + + `kubectl port-forward` is probably the easiest solution + +.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] +--- + +## Bonus + +- If you already have a local Kubernetes cluster: + + try to run another one! + +- Try to use another method than `kubectl port-forward` + +.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] +--- + +class: pic + +.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] + +--- + +name: toc-accessing-the-api-with-kubectl-proxy +class: title + + Accessing the API with `kubectl proxy` + +.nav[ +[Previous part](#toc-exercise--local-cluster) +| +[Back to table of contents](#toc-part-5) +| +[Next part](#toc-authentication-and-authorization) +] + +.debug[(automatically generated title slide)] + --- # Accessing the API with `kubectl proxy` @@ -12405,592 +12976,6 @@ class: extra-details class: pic -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/chinook-helicopter-container.jpg)] - ---- - -name: toc-exercise--local-cluster -class: title - - Exercise — Local Cluster - -.nav[ -[Previous part](#toc-accessing-the-api-with-kubectl-proxy) -| -[Back to table of contents](#toc-part-4) -| -[Next part](#toc-network-policies) -] - -.debug[(automatically generated title slide)] - ---- -# Exercise — Local Cluster - -- We want to have our own local Kubernetes cluster - - (we can use Docker Desktop, KinD, minikube... anything will do!) - -- Then we want to run a copy of dockercoins on that cluster - -- We want to be able to connect to the web UI - - (we can expose the port, or use port-forward, or whatever) - -.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] ---- - -## Goal - -- Be able to see the dockercoins web UI running on our local cluster - -.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] ---- - -## Hints - -- On a Mac or Windows machine: - - the easiest solution is probably Docker Desktop - -- On a Linux machine: - - the easiest solution is probably KinD or k3d - -- To connect to the web UI: - - `kubectl port-forward` is probably the easiest solution - -.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] ---- - -## Bonus - -- If you already have a local Kubernetes cluster: - - try to run another one! - -- Try to use another method than `kubectl port-forward` - -.debug[[exercises/localcluster-details.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/exercises/localcluster-details.md)] ---- - -class: pic - -.interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-cranes.jpg)] - ---- - -name: toc-network-policies -class: title - - Network policies - -.nav[ -[Previous part](#toc-exercise--local-cluster) -| -[Back to table of contents](#toc-part-5) -| -[Next part](#toc-authentication-and-authorization) -] - -.debug[(automatically generated title slide)] - ---- -# Network policies - -- Namespaces help us to *organize* resources - -- Namespaces do not provide isolation - -- By default, every pod can contact every other pod - -- By default, every service accepts traffic from anyone - -- If we want this to be different, we need *network policies* - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## What's a network policy? - -A network policy is defined by the following things. - -- A *pod selector* indicating which pods it applies to - - e.g.: "all pods in namespace `blue` with the label `zone=internal`" - -- A list of *ingress rules* indicating which inbound traffic is allowed - - e.g.: "TCP connections to ports 8000 and 8080 coming from pods with label `zone=dmz`, - and from the external subnet 4.42.6.0/24, except 4.42.6.5" - -- A list of *egress rules* indicating which outbound traffic is allowed - -A network policy can provide ingress rules, egress rules, or both. - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## How do network policies apply? - -- A pod can be "selected" by any number of network policies - -- If a pod isn't selected by any network policy, then its traffic is unrestricted - - (In other words: in the absence of network policies, all traffic is allowed) - -- If a pod is selected by at least one network policy, then all traffic is blocked ... - - ... unless it is explicitly allowed by one of these network policies - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -class: extra-details - -## Traffic filtering is flow-oriented - -- Network policies deal with *connections*, not individual packets - -- Example: to allow HTTP (80/tcp) connections to pod A, you only need an ingress rule - - (You do not need a matching egress rule to allow response traffic to go through) - -- This also applies for UDP traffic - - (Allowing DNS traffic can be done with a single rule) - -- Network policy implementations use stateful connection tracking - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Pod-to-pod traffic - -- Connections from pod A to pod B have to be allowed by both pods: - - - pod A has to be unrestricted, or allow the connection as an *egress* rule - - - pod B has to be unrestricted, or allow the connection as an *ingress* rule - -- As a consequence: if a network policy restricts traffic going from/to a pod, -
- the restriction cannot be overridden by a network policy selecting another pod - -- This prevents an entity managing network policies in namespace A - (but without permission to do so in namespace B) - from adding network policies giving them access to namespace B - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## The rationale for network policies - -- In network security, it is generally considered better to "deny all, then allow selectively" - - (The other approach, "allow all, then block selectively" makes it too easy to leave holes) - -- As soon as one network policy selects a pod, the pod enters this "deny all" logic - -- Further network policies can open additional access - -- Good network policies should be scoped as precisely as possible - -- In particular: make sure that the selector is not too broad - - (Otherwise, you end up affecting pods that were otherwise well secured) - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Our first network policy - -This is our game plan: - -- run a web server in a pod - -- create a network policy to block all access to the web server - -- create another network policy to allow access only from specific pods - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Running our test web server - -.lab[ - -- Let's use the `nginx` image: - ```bash - kubectl create deployment testweb --image=nginx - ``` - - - -- Find out the IP address of the pod with one of these two commands: - ```bash - kubectl get pods -o wide -l app=testweb - IP=$(kubectl get pods -l app=testweb -o json | jq -r .items[0].status.podIP) - ``` - -- Check that we can connect to the server: - ```bash - curl $IP - ``` -] - -The `curl` command should show us the "Welcome to nginx!" page. - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Adding a very restrictive network policy - -- The policy will select pods with the label `app=testweb` - -- It will specify an empty list of ingress rules (matching nothing) - -.lab[ - -- Apply the policy in this YAML file: - ```bash - kubectl apply -f ~/container.training/k8s/netpol-deny-all-for-testweb.yaml - ``` - -- Check if we can still access the server: - ```bash - curl $IP - ``` - - - -] - -The `curl` command should now time out. - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Looking at the network policy - -This is the file that we applied: - -```yaml -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: deny-all-for-testweb -spec: - podSelector: - matchLabels: - app: testweb - ingress: [] - - -``` - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Allowing connections only from specific pods - -- We want to allow traffic from pods with the label `run=testcurl` - -- Reminder: this label is automatically applied when we do `kubectl run testcurl ...` - -.lab[ - -- Apply another policy: - ```bash - kubectl apply -f ~/container.training/k8s/netpol-allow-testcurl-for-testweb.yaml - ``` - -] - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Looking at the network policy - -This is the second file that we applied: - -```yaml -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: allow-testcurl-for-testweb -spec: - podSelector: - matchLabels: - app: testweb - ingress: - - from: - - podSelector: - matchLabels: - run: testcurl - - -``` - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Testing the network policy - -- Let's create pods with, and without, the required label - -.lab[ - -- Try to connect to testweb from a pod with the `run=testcurl` label: - ```bash - kubectl run testcurl --rm -i --image=centos -- curl -m3 $IP - ``` - -- Try to connect to testweb with a different label: - ```bash - kubectl run testkurl --rm -i --image=centos -- curl -m3 $IP - ``` - -] - -The first command will work (and show the "Welcome to nginx!" page). - -The second command will fail and time out after 3 seconds. - -(The timeout is obtained with the `-m3` option.) - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## An important warning - -- Some network plugins only have partial support for network policies - -- For instance, Weave added support for egress rules [in version 2.4](https://github.com/weaveworks/weave/pull/3313) (released in July 2018) - -- But only recently added support for ipBlock [in version 2.5](https://github.com/weaveworks/weave/pull/3367) (released in Nov 2018) - -- Unsupported features might be silently ignored - - (Making you believe that you are secure, when you're not) - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Network policies, pods, and services - -- Network policies apply to *pods* - -- A *service* can select multiple pods - - (And load balance traffic across them) - -- It is possible that we can connect to some pods, but not some others - - (Because of how network policies have been defined for these pods) - -- In that case, connections to the service will randomly pass or fail - - (Depending on whether the connection was sent to a pod that we have access to or not) - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Network policies and namespaces - -- A good strategy is to isolate a namespace, so that: - - - all the pods in the namespace can communicate together - - - other namespaces cannot access the pods - - - external access has to be enabled explicitly - -- Let's see what this would look like for the DockerCoins app! - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Network policies for DockerCoins - -- We are going to apply two policies - -- The first policy will prevent traffic from other namespaces - -- The second policy will allow traffic to the `webui` pods - -- That's all we need for that app! - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Blocking traffic from other namespaces - -This policy selects all pods in the current namespace. - -It allows traffic only from pods in the current namespace. - -(An empty `podSelector` means "all pods.") - -```yaml -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: deny-from-other-namespaces -spec: - podSelector: {} - ingress: - - from: - - podSelector: {} -``` - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Allowing traffic to `webui` pods - -This policy selects all pods with label `app=webui`. - -It allows traffic from any source. - -(An empty `from` field means "all sources.") - -```yaml -kind: NetworkPolicy -apiVersion: networking.k8s.io/v1 -metadata: - name: allow-webui -spec: - podSelector: - matchLabels: - app: webui - ingress: - - from: [] -``` - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Applying both network policies - -- Both network policies are declared in the file [k8s/netpol-dockercoins.yaml](https://github.com/jpetazzo/container.training/tree/master/k8s/netpol-dockercoins.yaml) - -.lab[ - -- Apply the network policies: - ```bash - kubectl apply -f ~/container.training/k8s/netpol-dockercoins.yaml - ``` - -- Check that we can still access the web UI from outside -
- (and that the app is still working correctly!) - -- Check that we can't connect anymore to `rng` or `hasher` through their ClusterIP - -] - -Note: using `kubectl proxy` or `kubectl port-forward` allows us to connect -regardless of existing network policies. This allows us to debug and -troubleshoot easily, without having to poke holes in our firewall. - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Cleaning up our network policies - -- The network policies that we have installed block all traffic to the default namespace - -- We should remove them, otherwise further demos and exercises will fail! - -.lab[ - -- Remove all network policies: - ```bash - kubectl delete networkpolicies --all - ``` - -] - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Protecting the control plane - -- Should we add network policies to block unauthorized access to the control plane? - - (etcd, API server, etc.) - --- - -- At first, it seems like a good idea ... - --- - -- But it *shouldn't* be necessary: - - - not all network plugins support network policies - - - the control plane is secured by other methods (mutual TLS, mostly) - - - the code running in our pods can reasonably expect to contact the API -
- (and it can do so safely thanks to the API permission model) - -- If we block access to the control plane, we might disrupt legitimate code - -- ...Without necessarily improving security - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Tools and resources - -- [Cilium Network Policy Editor](https://editor.cilium.io/) - -- [Tufin Network Policy Viewer](https://orca.tufin.io/netpol/) - -- Two resources by [Ahmet Alp Balkan](https://ahmet.im/): - - - a [very good talk about network policies](https://www.youtube.com/watch?list=PLj6h78yzYM2P-3-xqvmWaZbbI1sW-ulZb&v=3gGpMmYeEO8) at KubeCon North America 2017 - - - a repository of [ready-to-use recipes](https://github.com/ahmetb/kubernetes-network-policy-recipes) for network policies - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -## Documentation - -- As always, the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) is a good starting point - -- The API documentation has a lot of detail about the format of various objects: - - - [NetworkPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicy-v1-networking-k8s-io) - - - [NetworkPolicySpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicyspec-v1-networking-k8s-io) - - - [NetworkPolicyIngressRule](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#networkpolicyingressrule-v1-networking-k8s-io) - - - etc. - -??? - -:EN:- Isolating workloads with Network Policies -:FR:- Isolation réseau avec les *network policies* - -.debug[[k8s/netpol.md](https://github.com/jpetazzo/container.training.git/tree/2022-08-nr/slides/k8s/netpol.md)] ---- - -class: pic - .interstitial[![Image separating from the next part](https://gallant-turing-d0d520.netlify.com/containers/container-housing.jpg)] --- @@ -13001,7 +12986,7 @@ class: title Authentication and authorization .nav[ -[Previous part](#toc-network-policies) +[Previous part](#toc-accessing-the-api-with-kubectl-proxy) | [Back to table of contents](#toc-part-5) |