From 38a40d56a07ce72b2ff714f0224a6cece39f762d Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Tue, 10 Apr 2018 06:04:17 -0500 Subject: [PATCH] Label use-cases and rollouts This adds a few realistic examples of label usage. It also adds explanations about why deploying a new version of the worker doesn't seem to be effective immediately (the worker doesn't handle signals). --- slides/kube/daemonset.md | 36 ++++++++++++++++++++++++++++++++++++ slides/kube/rollout.md | 20 ++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/slides/kube/daemonset.md b/slides/kube/daemonset.md index 93945f4b..b502fec3 100644 --- a/slides/kube/daemonset.md +++ b/slides/kube/daemonset.md @@ -460,3 +460,39 @@ The timestamps should give us a hint about how many pods are currently receiving kubectl get pods -l run=rng -o name | xargs kubectl patch -p "$PATCH" ``` + +--- + +## Labels and debugging + +- When a pod is misbehaving, we can delete it: another one will be recreated + +- But we can also change its labels + +- It will be removed from the load balancer (it won't receive traffic anymore) + +- Another pod will be recreated immediately + +- But the problematic pod is still here, and we can inspect and debug it + +- We can even re-add it to the rotation if necessary + + (Very useful to troubleshoot intermittent and elusive bugs) + +--- + +## Labels and advanced rollout control + +- Conversely, we can add pods matching a service's selector + +- These pods will then receive requests and serve traffic + +- Examples: + + - one-shot pod with all debug flags enabled, to collect logs + + - pods created automatically, but added to rotation in a second step +
+ (by setting their label accordingly) + +- This gives us building blocks for canary and blue/green deployments diff --git a/slides/kube/rollout.md b/slides/kube/rollout.md index 10a49913..1b479415 100644 --- a/slides/kube/rollout.md +++ b/slides/kube/rollout.md @@ -94,6 +94,26 @@ That rollout should be pretty quick. What shows in the web UI? --- +## Give it some time + +- At first, it looks like nothing is happening (the graph remains at the same level) + +- According to `kubectl get deploy -w`, the `deployment` was updated really quickly + +- But `kubectl get pods -w` tells a different story + +- The old `pods` are still here, and they stay in `Terminating` state for a while + +- Eventually, they are terminated; and then the graph decreases significantly + +- This delay is due to the fact that our worker doesn't handle signals + +- Kubernetes sends a "polite" shutdown request to the worker, which ignores it + +- Eventually, Kubernetes gets impatient and kills the container + +--- + ## Rolling out a boo-boo - What happens if we make a mistake?