From e569388618871a8042cd8d28baf18e951d943d10 Mon Sep 17 00:00:00 2001 From: Julien Girardin Date: Wed, 29 Jan 2020 18:04:03 +0100 Subject: [PATCH] Advanced rollout and security --- slides/3.yml | 6 +- slides/k8s/advanced-rollout.md | 76 +++++++++++++++++++++++ slides/k8s/devs-and-ops-joined-topics.md | 10 +++ slides/k8s/kubernetes-security.md | 78 ++++++++++++++++++++++++ slides/k8s/monitoring-intro.md | 6 -- slides/k8s/prometheus-endpoint.md | 14 +++-- slides/k8s/rollout.md | 41 +++++++------ 7 files changed, 200 insertions(+), 31 deletions(-) create mode 100644 slides/k8s/advanced-rollout.md create mode 100644 slides/k8s/devs-and-ops-joined-topics.md create mode 100644 slides/k8s/kubernetes-security.md delete mode 100644 slides/k8s/monitoring-intro.md diff --git a/slides/3.yml b/slides/3.yml index 05549e0e..93876681 100644 --- a/slides/3.yml +++ b/slides/3.yml @@ -44,9 +44,11 @@ chapters: - k8s/exercise-ci-build.md - k8s/kaniko.md - k8s/exercise-ci-kaniko.md + - k8s/rollout.md + - k8s/advanced-rollout.md + - k8s/devs-and-ops-joined-topics.md - - - k8s/monitoring-intro.md #- k8s/prometheus-intro.md - k8s/prometheus-endpoint.md - k8s/exercise-prometheus.md @@ -54,6 +56,8 @@ chapters: - k8s/exercise-opentelemetry.md +- + - k8s/kubernetes-security.md #- # - k8s/rollout.md # - k8s/blue-green.md diff --git a/slides/k8s/advanced-rollout.md b/slides/k8s/advanced-rollout.md new file mode 100644 index 00000000..30b91137 --- /dev/null +++ b/slides/k8s/advanced-rollout.md @@ -0,0 +1,76 @@ +# Advanced Rollout + +- In some cases the built-in mechanism of kubernetes is not enough. + +- You want more control on the rollout, include a feedback of the monitoring, deploying +on multiple clusters, etc + +- Two "main" strategies exist here: + + - canary deployment + - blue/green deployment + +--- +## Canary deployment + +- focus on one component of the stack + + - deploy a new version of the component close to the production + - redirect some portion of prod traffic to new version + - scale up new version, redirect more traffic, checking everything is ok + - scale down old version + - move component to component with the same procedure + +- That's what kubernetes does by default, but does every components at the same time + +- Could be paired with `kubectl wait --for` and applying component sequentially, + for hand made canary deployement + +--- +## Blue/Green deployment + +- focus on all stack + + - deploy en new stack + - check the new stack work as espected + - put traffic on new stack, rollback if any goes wrong + - garbage collect the previous infra structure + +- there is nothing like that by default in kubernetes + +- helm chart with multiple releases is the closest one + +- could be paired with ingress feature like `nginx.ingress.kubernetes.io/canary-*` + +--- +## Not hand-made ? + +There is a few additionnal controllers that help achieving those kind of rollout behaviours + +They leverage kubernetes API at different levels to achieve this goal. + +--- +## Spinnaker + +- https://www.spinnaker.io + +- Help to deploy the same app on multiple cluster. + +- Is able to analyse rollout status (canary analysis) and correlate it to monitoring + +- Rollback if anything goes wrong + +- also support Blue/Green + +- Configuration done via UI + +--- +## Argo-rollout + +- https://github.com/argoproj/argo-rollouts + +- Replace your deployments with CRD (Custom Resource Definition) "deployment-like" + +- Full control via CRDs + +- BlueGreen and Canary deployment diff --git a/slides/k8s/devs-and-ops-joined-topics.md b/slides/k8s/devs-and-ops-joined-topics.md new file mode 100644 index 00000000..fb4e0759 --- /dev/null +++ b/slides/k8s/devs-and-ops-joined-topics.md @@ -0,0 +1,10 @@ +## We are done, what else ? + +We have seen what means developping an application on kubernetes. + +There still few subjects to tackle that are not purely relevant for developers + +They have *some involvement* for developers: + +- Monitoring +- Security diff --git a/slides/k8s/kubernetes-security.md b/slides/k8s/kubernetes-security.md new file mode 100644 index 00000000..a68e6f72 --- /dev/null +++ b/slides/k8s/kubernetes-security.md @@ -0,0 +1,78 @@ +# Security and kubernetes + +There are many mechanisms in kubernetes to ensure the security. +Obviously the more you constrain your app, the better. + +There is also mechanism to forbid "unsafe" application to be launched on +kubernetes, but that's more for ops-guys 😈 (more on that next days) + +Let's focus on what can we do on the developer latop, to make app +compatible with secure system, enforced or not (it's always a good practice) + +--- +## No container in privileged mode + +- risks: + - If one privileged container get compromised, + we basically get full access to the node from within a container + (not need to tamper auth logs, alter binary). + + - Sniffing networks allow often to get access to the entire cluster. + +- how to avoid: + ``` + [...] + spec: + containers: + - name: foo + securityContext: + privileged: false + ``` + + Luckily that's the default ! + +--- +## No container run as "root" + +- risks: + - bind mounting a directory like /usr/bin allow to change node system core +
ex: copy a tampered version of "ping", wait for an admin to login + and to issue a ping command and bingo ! + +- how to avoid: + + ``` + [...] + spec: + containers: + - name: foo + securityContext: + runAsUser: 1000 + runAsGroup: 100 + ``` + - The default is to use the image default + + - If your writing your own Dockerfile, don't forget about the `USER` instruction +--- +## Capabilities + +- You can give capabilities one-by-one to a container + +- It's useful if you need more capabilities (for some reason), but not grating 'root' privileged + +- risks: no risks whatsoever, except by granting a big list of capabilities + +- how to use: + ``` + [...] + spec: + containers: + - name: foo + securityContext: + capabilities: + add: ["NET_ADMIN", "SYS_TIME"] + drop: [] + ``` + The default use the container runtime defaults + +- and we can also drop default capabilities granted by the container runtime ! diff --git a/slides/k8s/monitoring-intro.md b/slides/k8s/monitoring-intro.md deleted file mode 100644 index 26077479..00000000 --- a/slides/k8s/monitoring-intro.md +++ /dev/null @@ -1,6 +0,0 @@ -# Monitoring - -We have shipped our application to kubernetes, now we need to tackle -the last item of the development cycle: - -- monitoring diff --git a/slides/k8s/prometheus-endpoint.md b/slides/k8s/prometheus-endpoint.md index 4b398416..8e00e806 100644 --- a/slides/k8s/prometheus-endpoint.md +++ b/slides/k8s/prometheus-endpoint.md @@ -1,13 +1,19 @@ # Prometheus -Prometheus is monitoring system with small storage io footprint. It's quite ubiquitous +Prometheus is a monitoring system with small storage I/O footprint -in the kubernetes world. This section is not a description +It's quite ubiquitous in the kubernetes world. +This section is not a description of prometheus. +*Note: More on prometheus next day* + +--- ## Prometheus endpoint -The goal here is to expose an HTTP endoint for prometheus. Sample response: +- Prometheus "pull" metrics from different HTTP endpoint + +- The goal for the developper is to expose an HTTP endoint for prometheus. Sample response: .small[ ``` @@ -21,7 +27,6 @@ The goal here is to expose an HTTP endoint for prometheus. Sample response: ``` ] - To achieve this multiple strategies could be used: - developping in the application itself (especialy if it's already an httpserver) @@ -46,7 +51,6 @@ Links (do you see a pattern ?): - https://github.com/prometheus/client_ruby - https://github.com/prometheus/client_golang - --- ## Add sidecar Exporter diff --git a/slides/k8s/rollout.md b/slides/k8s/rollout.md index 5e8c1a9f..634e5c52 100644 --- a/slides/k8s/rollout.md +++ b/slides/k8s/rollout.md @@ -79,7 +79,8 @@ - Rolling updates can be monitored with the `kubectl rollout` subcommand ---- + +/--> ] @@ -161,7 +162,7 @@ Our rollout is stuck. However, the app is not dead. (After a minute, it will stabilize to be 20-25% slower.) ---- +/--- ## What's going on with our rollout? @@ -181,7 +182,7 @@ Our rollout is stuck. However, the app is not dead.
but the total number of pods being rolled out is allowed to be 25+25=50% ---- +/--- class: extra-details @@ -201,7 +202,7 @@ class: extra-details - Our rollout is stuck at this point! ---- +/--- ## Checking the dashboard during the bad rollout @@ -217,7 +218,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. ] ---- +/--- ## Recovering from a bad rollout @@ -239,7 +240,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. ] ---- +/--- ## Rolling back to an older version @@ -249,7 +250,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. - How can we get back to the previous version? ---- +/--- ## Multiple "undos" @@ -268,7 +269,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. 🤔 That didn't work. ---- +/--- ## Multiple "undos" don't work @@ -289,7 +290,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. ] ---- +/--- ## In this specific scenario @@ -299,7 +300,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide. - What if we had changed other parameters in the Pod spec? ---- +/--- ## Listing versions @@ -320,7 +321,7 @@ We might see something like 1, 4, 5. (Depending on how many "undos" we did before.) ---- +/--- ## Explaining deployment revisions @@ -337,7 +338,7 @@ We might see something like 1, 4, 5. ] ---- +/--- class: extra-details @@ -353,7 +354,7 @@ class: extra-details (if we wanted to!) ---- +/--- ## Rolling back to an older version @@ -370,7 +371,7 @@ class: extra-details ] ---- +/--- class: extra-details @@ -401,7 +402,7 @@ spec: ``` ] ---- +/--- class: extra-details @@ -437,3 +438,5 @@ class: extra-details ] ] + +-->