github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-26 00:06:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

Advanced rollout and security

Browse Source
This commit is contained in:
Julien Girardin
2020-01-29 18:04:03 +01:00
parent 3a1549e3a4
commit e569388618
7 changed files with 200 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
slides/3.yml
View File

@@ -44,9 +44,11 @@ chapters:
- k8s/exercise-ci-build.md
- k8s/kaniko.md
- k8s/exercise-ci-kaniko.md
- k8s/rollout.md
- k8s/advanced-rollout.md
- k8s/devs-and-ops-joined-topics.md
-
- k8s/monitoring-intro.md
#- k8s/prometheus-intro.md
- k8s/prometheus-endpoint.md
- k8s/exercise-prometheus.md
@@ -54,6 +56,8 @@ chapters:
- k8s/exercise-opentelemetry.md
-
- k8s/kubernetes-security.md
#-
# - k8s/rollout.md
# - k8s/blue-green.md

76
slides/k8s/advanced-rollout.md Normal file
View File

@@ -0,0 +1,76 @@
# Advanced Rollout
- In some cases the built-in mechanism of kubernetes is not enough.
- You want more control on the rollout, include a feedback of the monitoring, deploying
on multiple clusters, etc
- Two "main" strategies exist here:
- canary deployment
- blue/green deployment
---
## Canary deployment
- focus on one component of the stack
- deploy a new version of the component close to the production
- redirect some portion of prod traffic to new version
- scale up new version, redirect more traffic, checking everything is ok
- scale down old version
- move component to component with the same procedure
- That's what kubernetes does by default, but does every components at the same time
- Could be paired with `kubectl wait --for` and applying component sequentially,
for hand made canary deployement
---
## Blue/Green deployment
- focus on all stack
- deploy en new stack
- check the new stack work as espected
- put traffic on new stack, rollback if any goes wrong
- garbage collect the previous infra structure
- there is nothing like that by default in kubernetes
- helm chart with multiple releases is the closest one
- could be paired with ingress feature like `nginx.ingress.kubernetes.io/canary-*`
---
## Not hand-made ?
There is a few additionnal controllers that help achieving those kind of rollout behaviours
They leverage kubernetes API at different levels to achieve this goal.
---
## Spinnaker
- https://www.spinnaker.io
- Help to deploy the same app on multiple cluster.
- Is able to analyse rollout status (canary analysis) and correlate it to monitoring
- Rollback if anything goes wrong
- also support Blue/Green
- Configuration done via UI
---
## Argo-rollout
- https://github.com/argoproj/argo-rollouts
- Replace your deployments with CRD (Custom Resource Definition) "deployment-like"
- Full control via CRDs
- BlueGreen and Canary deployment

10
slides/k8s/devs-and-ops-joined-topics.md Normal file
View File

@@ -0,0 +1,10 @@
## We are done, what else ?
We have seen what means developping an application on kubernetes.
There still few subjects to tackle that are not purely relevant for developers
They have *some involvement* for developers:
- Monitoring
- Security

78
slides/k8s/kubernetes-security.md Normal file
View File

@@ -0,0 +1,78 @@
# Security and kubernetes
There are many mechanisms in kubernetes to ensure the security.
Obviously the more you constrain your app, the better.
There is also mechanism to forbid "unsafe" application to be launched on
kubernetes, but that's more for ops-guys 😈 (more on that next days)
Let's focus on what can we do on the developer latop, to make app
compatible with secure system, enforced or not (it's always a good practice)
---
## No container in privileged mode
- risks:
- If one privileged container get compromised,
we basically get full access to the node from within a container
(not need to tamper auth logs, alter binary).
- Sniffing networks allow often to get access to the entire cluster.
- how to avoid:
```
[...]
spec:
containers:
- name: foo
securityContext:
privileged: false
```
Luckily that's the default !
---
## No container run as "root"
- risks:
- bind mounting a directory like /usr/bin allow to change node system core
</br>ex: copy a tampered version of "ping", wait for an admin to login
and to issue a ping command and bingo !
- how to avoid:
```
[...]
spec:
containers:
- name: foo
securityContext:
runAsUser: 1000
runAsGroup: 100
```
- The default is to use the image default
- If your writing your own Dockerfile, don't forget about the `USER` instruction
---
## Capabilities
- You can give capabilities one-by-one to a container
- It's useful if you need more capabilities (for some reason), but not grating 'root' privileged
- risks: no risks whatsoever, except by granting a big list of capabilities
- how to use:
```
[...]
spec:
containers:
- name: foo
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME"]
drop: []
```
The default use the container runtime defaults
- and we can also drop default capabilities granted by the container runtime !

6
slides/k8s/monitoring-intro.md
View File

@@ -1,6 +0,0 @@
# Monitoring
We have shipped our application to kubernetes, now we need to tackle
the last item of the development cycle:
- monitoring

14
slides/k8s/prometheus-endpoint.md
View File

@@ -1,13 +1,19 @@
# Prometheus
Prometheus is monitoring system with small storage io footprint. It's quite ubiquitous
Prometheus is a monitoring system with small storage I/O footprint
in the kubernetes world. This section is not a description
It's quite ubiquitous in the kubernetes world.
This section is not a description of prometheus.
*Note: More on prometheus next day*
---
## Prometheus endpoint
The goal here is to expose an HTTP endoint for prometheus. Sample response:
- Prometheus "pull" metrics from different HTTP endpoint
- The goal for the developper is to expose an HTTP endoint for prometheus. Sample response:
.small[
```
@@ -21,7 +27,6 @@ The goal here is to expose an HTTP endoint for prometheus. Sample response:
```
]
To achieve this multiple strategies could be used:
- developping in the application itself (especialy if it's already an httpserver)
@@ -46,7 +51,6 @@ Links (do you see a pattern ?):
- https://github.com/prometheus/client_ruby
- https://github.com/prometheus/client_golang
---
## Add sidecar Exporter

41
slides/k8s/rollout.md
View File

@@ -79,7 +79,8 @@
- Rolling updates can be monitored with the `kubectl rollout` subcommand
---
<!--
/---
## Rolling out the new `worker` service
@@ -108,7 +109,7 @@
That rollout should be pretty quick. What shows in the web UI?
---
/---
## Give it some time
@@ -130,7 +131,7 @@ That rollout should be pretty quick. What shows in the web UI?
(The grace period is 30 seconds, but [can be changed](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods) if needed)
---
/---
## Rolling out something invalid
@@ -148,10 +149,10 @@ That rollout should be pretty quick. What shows in the web UI?
kubectl rollout status deploy worker
```
<!--
/<!--
```wait Waiting for deployment```
```key ^C```
-->
/-->
]
@@ -161,7 +162,7 @@ Our rollout is stuck. However, the app is not dead.
(After a minute, it will stabilize to be 20-25% slower.)
---
/---
## What's going on with our rollout?
@@ -181,7 +182,7 @@ Our rollout is stuck. However, the app is not dead.
<br/>
but the total number of pods being rolled out is allowed to be 25+25=50%
---
/---
class: extra-details
@@ -201,7 +202,7 @@ class: extra-details
- Our rollout is stuck at this point!
---
/---
## Checking the dashboard during the bad rollout
@@ -217,7 +218,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
]
---
/---
## Recovering from a bad rollout
@@ -239,7 +240,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
]
---
/---
## Rolling back to an older version
@@ -249,7 +250,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
- How can we get back to the previous version?
---
/---
## Multiple "undos"
@@ -268,7 +269,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
🤔 That didn't work.
---
/---
## Multiple "undos" don't work
@@ -289,7 +290,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
]
---
/---
## In this specific scenario
@@ -299,7 +300,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
- What if we had changed other parameters in the Pod spec?
---
/---
## Listing versions
@@ -320,7 +321,7 @@ We might see something like 1, 4, 5.
(Depending on how many "undos" we did before.)
---
/---
## Explaining deployment revisions
@@ -337,7 +338,7 @@ We might see something like 1, 4, 5.
]
---
/---
class: extra-details
@@ -353,7 +354,7 @@ class: extra-details
(if we wanted to!)
---
/---
## Rolling back to an older version
@@ -370,7 +371,7 @@ class: extra-details
]
---
/---
class: extra-details
@@ -401,7 +402,7 @@ spec:
```
]
---
/---
class: extra-details
@@ -437,3 +438,5 @@ class: extra-details
]
]
-->