Add admission webhook content

slides/k8s/admission.md

@@ -0,0 +1,549 @@
+# Dynamic Admission Control
+
+- This is one of the many ways to extend the Kubernetes API
+
+- High level summary: dynamic admission control relies on webhooks that are ...
+
+  - dynamic (can be added/removed on the fly)
+
+  - running inside our outside the cluster
+
+  - *validating* (yay/nay) or *mutating* (can change objects that are created/updated)
+
+  - selective (can be configured to apply only to some kinds, some selectors...)
+
+  - mandatory or optional (should it block operations when webhook is down?)
+
+- Used for themselves (e.g. policy enforcement) or as part of operators
+
+---
+
+## Use cases
+
+Some examples ...
+
+- Stand-alone admission controllers
+
+  *validating:* policy enforcement (e.g. quotas, naming conventions ...)
+
+  *mutating:* inject or provide default values (e.g. pod presets)
+
+- Admission controllers part of a greater system
+
+  *validating:* advanced typing for operators
+
+  *mutating:* inject sidecars for service meshes
+
+---
+
+## You said *dynamic?*
+
+- Some admission controllers are built in the API server
+
+- They are enabled/disabled through Kubernetes API server configuration
+
+  (e.g. `--enable-admission-plugins`/`--disable-admission-plugins` flags)
+
+- Here, we're talking about *dynamic* admission controllers
+
+- They can be added/remove while the API server is running
+
+  (without touching the configuration files or even having access to them)
+
+- This is done through two kinds of cluster-scope resources:
+
+  ValidatingWebhookConfiguration and MutatingWebhookConfiguration
+
+---
+
+## You said *webhooks?*
+
+- A ValidatingWebhookConfiguration or MutatingWebhookConfiguration contains:
+
+  - a resource filter
+    <br/>
+    (e.g. "all pods", "deployments in namespace xyz", "everything"...)
+
+  - an operations filter
+    <br/>
+    (e.g. CREATE, UPDATE, DELETE)
+
+  - the address of the webhook server
+
+- Each time an operation matches the filters, it is sent to the webhook server
+
+---
+
+## What gets sent exactly?
+
+- The API server will `POST` a JSON object to the webhook
+
+- That object will be a Kubernetes API message with `kind` `AdmissionReview`
+
+- It will contain a `request` field, with, notably:
+
+  - `request.uid` (to be used when replying)
+
+  - `request.object` (the object created/deleted/changed)
+
+  - `request.oldObject` (when an object is modified)
+
+  - `request.userInfo` (who was making the request to the API in the first place)
+
+(See [the documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request) for a detailed example showing more fields.)
+
+---
+
+## How should the webhook respond?
+
+- By replying with another `AdmissionReview` in JSON
+
+- It should have a `response` field, with, notably:
+
+  - `response.uid` (matching the `request.uid`)
+
+  - `response.allowed` (`true`/`false`)
+
+  - `response.status.message` (optional string; useful when denying requests)
+
+  - `response.patchType` (when a mutating webhook changes the object; e.g. `json`)
+
+  - `response.patch` (the patch, encoded in base64)
+
+---
+
+## What if the webhook *does not* respond?
+
+- If "something bad" happens, the API server follows the `failurePolicy` option
+
+  - this is a per-webhook option (specified in the webhook configuration)
+
+  - it can be `Fail` (the default) or `Ignore` ("allow all, unmodified")
+
+- What's "something bad"?
+
+  - webhook responds with something invalid
+
+  - webhook takes more than 10 seconds to respond
+    <br/>
+    (this can be changed with `timeoutSeconds` field in the webhook config)
+
+  - webhook is down or has invalid certificates
+    <br/>
+    (TLS! It's not just a good idea; for admission control, it's the law!)
+
+---
+
+## What did you say about TLS?
+
+- The webhook configuration can indicate:
+
+  - either `url` of the webhook server (has to begin with `https://`)
+
+  - or `service.name` and `service.namespace` of a Service on the cluster
+
+- In the latter case, the Service has to accept TLS connections on port 443
+
+- It has to use a certificate with CN `<name>.<namespace>.svc`
+
+  (**and** a `subjectAltName` extension with `DNS:<name>.<namespace>.svc`)
+
+- The certificate needs to be valid (signed by a CA trusted by the API server)
+
+  ... alternatively, we can pass a `caBundle` in the webhook configuration
+
+---
+
+## Webhook server inside or outside
+
+- "Outside" webhook server is defined with `url` option
+
+  - convenient for external webooks (e.g. tamper-resistent audit trail)
+
+  - also great for initial development (e.g. with ngrok)
+
+  - requires outbound connectivity (duh) and can become a SPOF
+
+- "Inside" webhook server is defined with `service` option
+
+  - convenient when the webhook needs to be deployed and managed on the cluster
+
+  - also great for air gapped clusters
+
+  - development can be harder (but tools like [Tilt](https://tilt.dev) can help)
+
+---
+
+## Developing a simple admission webhook
+
+- We're going to register a custom webhook!
+
+- First, we'll just dump the `AdmissionRequest` object
+
+  (using a little Node app)
+
+- Then, we'll implement a strict policy on a specific label
+
+  (using a little Flask app)
+
+- Development will happen in local containers, plumbed with ngrok
+
+- The we will deploy to the cluster 🔥
+
+---
+
+## Running the webhook locally
+
+- We prepared a Docker Compose file to start the whole stack
+
+  (the Node "echo" app, the Flask app, and one ngrok tunnel for each of them)
+
+.exercise[
+
+- Go to the webhook directory:
+  ```bash
+  cd ~/container.training/webhooks/admission
+  ```
+
+- Start the webhook in Docker containers:
+  ```bash
+  docker-compose up
+  ```
+
+]
+
+*Note the URL in `ngrok-echo_1` looking like `url=https://xxxx.ngrok.io`.*
+
+---
+
+class: extra-details
+
+## What's ngrok?
+
+- Ngrok provides secure tunnels to access local services
+
+- Example: run `ngrok http 1234`
+
+- `ngrok` will display a publicly-available URL (e.g. https://xxxxyyyyzzzz.ngrok.io)
+
+- Connections to https://xxxxyyyyzzzz.ngrok.io will terminate at `localhost:1234`
+
+- Basic product is free; extra features (vanity domains, end-to-end TLS...) for $$$
+
+- Perfect to develop our webhook!
+
+- Probably not for production, though
+
+  (webhook requests and responses now pass through the ngrok platform)
+
+---
+
+## Update the webhook configuration
+
+- We have a webhook configuration in `k8s/webhook-configuration.yaml`
+
+- We need to update the configuration with the correct `url`
+
+.exercise[
+
+- Edit the webhook configuration manifest:
+  ```bash
+  vim k8s/webhook-configuration.yaml
+  ```
+
+- **Uncomment** the `url:` line
+
+- **Update** the `.ngrok.io` URL with the URL shown by Compose
+
+- Save and quit
+
+]
+
+---
+
+## Register the webhook configuration
+
+- Just after we register the webhook, it will be called for each matching request
+
+  (CREATE and UPDATE on Pods in all namespaces)
+
+- The `failurePolicy` is `Ignore`
+
+  (so if the webhook server is down, we can still create pods)
+
+.exercise[
+
+- Register the webhook:
+  ```bash
+  kubectl apply -f k8s/webhook-configuration.yaml
+  ```
+
+]
+
+It is strongly recommended to tail the logs of the API server while doing that.
+
+---
+
+## Create a pod
+
+- Let's create a pod and try to set a `color` label
+
+.exercise[
+
+- Create a pod named `chroma`:
+  ```bash
+  kubectl run --restart=Never chroma --image=nginx
+  ```
+
+- Add a label `color` set to `pink`:
+  ```bash
+  kubectl label pod chroma color=pink
+  ```
+
+]
+
+We should see the `AdmissionReview` objects in the Compose logs.
+
+Note: the webhook doesn't do anything (other than printing the request payload).
+
+---
+
+## Use the "real" admission webhook
+
+- We have a small Flask app implementing a particular policy on pod labels:
+
+  - if a pod sets a label `color`, it must be `blue`, `green`, `red`
+
+  - once that `color` label is set, it cannot be removed or changed
+
+- That Flask app was started when we did `docker-compose up` earlier
+
+- It is exposed through its own ngrok tunnel
+
+- We are going to use that webhook instead of the other one
+
+  (by changing only the `url` field in the ValidatingWebhookConfiguration)
+
+---
+
+## Update the webhook configuration
+
+.exercise[
+
+- First, check the ngrok URL of the tunnel for the Flask app:
+  ```bash
+  docker-compose logs ngrok-flask
+  ```
+
+- Then, edit the webhook configuration:
+  ```bash
+  kubectl edit validatingwebhookconfiguration admission.container.training
+  ```
+- Find the `url:` field with the `.ngrok.io` URL and update it
+
+- Save and quit; the new configuration is applied immediately
+
+]
+
+---
+
+## Verify the behavior of the webhook
+
+- Try to create a few pods and/or change labels on existing pods
+
+- What happens if we try to make changes to the earlier pod?
+
+  (the one that has `label=pink`)
+
+---
+
+## Deploying the webhook on the cluster
+
+- Let's see what's needed to self-host the webhook server!
+
+- The webhook needs to be reachable through a Service on our cluster
+
+- The Service needs to accept TLS connections on port 443
+
+- We need a proper TLS certificate:
+
+  - with the right `CN` and `subjectAltName` (`<servicename>.<namespace>.svc`)
+
+  - signed by a trusted CA
+
+- We can either use a "real" CA, or use the `caBundle` option to specify the CA cert
+
+  (the latter makes it easy to use self-signed certs)
+
+---
+
+## In practice
+
+- We're going to generate a key pair and a self-signed certificate
+
+- We will store them in a Secret
+
+- We will run the webhook in a Deployment, exposed with a Service
+
+- We will update the webhook configuration to use that Service
+
+- The Service will be named `admission`, in Namespace `webhooks`
+
+  (keep in mind that the ValidatingWebhookConfiguration itself is at cluster scope)
+
+---
+
+## Let's get to work!
+
+.exercise[
+
+- Make sure we're in the right directory:
+  ```bash
+  cd ~/container.training/webhooks/admission
+  ```
+
+- Create the namespace:
+  ```bash
+  kubectl create namespace webhooks
+  ```
+
+- Switch to the namespace:
+  ```bash
+  kubectl config set-context --current --namespace=webhooks
+  ```
+
+]
+
+---
+
+## Deploying the webhook
+
+- *Normally,* we would author an image for this
+
+- Since our webhook is just *one* Python source file ...
+
+  ... we'll store it in a ConfigMap, and install dependencies on the fly
+
+.exercise[
+
+- Load the webhook source in a ConfigMap:
+  ```bash
+  kubectl create configmap admission --from-file=flask/webhook.py
+  ```
+
+- Create the Deployment and Service:
+  ```bash
+  kubectl apply -f k8s/webhook-server.yaml
+  ```
+
+]
+
+---
+
+## Generating the key pair and certificate
+
+- Let's call OpenSSL to the rescue!
+
+  (of course, there are plenty others options; e.g. `cfssl`)
+
+.exercise[
+
+- Generate a self-signed certificate:
+  ```bash
+    NAMESPACE=webhooks
+    SERVICE=admission
+    CN=$SERVICE.$NAMESPACE.svc
+    openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem \
+        -days 30 -subj /CN=$CN -addext subjectAltName=DNS:$CN
+  ```
+
+- Load up the key and cert in a Secret:
+  ```bash
+  kubectl create secret tls admission --cert=cert.pem --key=key.pem
+  ```
+
+]
+
+---
+
+## Update the webhook configuration
+
+- Let's reconfigure the webhook to use our Service instead of ngrok
+
+.exercise[
+
+- Edit the webhook configuration manifest:
+  ```bash
+  vim k8s/webhook-configuration.yaml
+  ```
+
+- Comment out the `url:` line
+
+- Uncomment the `service:` section
+
+- Save, quit
+
+- Update the webhook configuration:
+  ```bash
+  kubectl apply -f k8s/webhook-configuration.yaml
+  ```
+
+]
+
+---
+
+## Add our self-signed cert to the `caBundle`
+
+- The API server won't accept our self-signed certificate
+
+- We need to add it to the `caBundle` field in the webhook configuration
+
+- The `caBundle` will be our `cert.pem` file, encoded in base64
+
+---
+
+Shell to the rescue!
+
+.exercise[
+
+- Load up our cert and encode it in base64:
+  ```bash
+  CA=$(base64 -w0 < cert.pem)
+  ```
+
+- Define a patch operation to update the `caBundle`:
+  ```bash
+    PATCH='[{
+        "op": "replace",
+        "path": "/webhooks/0/clientConfig/caBundle",
+        "value":"'$CA'"
+    }]'
+  ```
+
+- Patch the webhook configuration:
+  ```bash
+    kubectl patch validatingwebhookconfiguration \
+                  admission.webhook.container.training \
+                  --type='json' -p="$PATCH"
+  ```
+
+]
+
+---
+
+## Try it out!
+
+- Keep an eye on the API server logs
+
+- Tail the logs of the pod running the webhook server
+
+- Create a few pods; we should see requests in the webhook server logs
+
+- Check that the label `color` is enforced correctly
+
+  (it should only allow values of `red`, `green`, `blue`)
+
+???
+
+:EN:- Dynamic admission control with webhooks
+:FR:- Contrôle d'admission dynamique (webhooks)

slides/kube-fullday.yml

@@ -100,6 +100,7 @@ content:
   #- k8s/local-persistent-volumes.md
   #- k8s/portworx.md
   #- k8s/extending-api.md
+  #- k8s/admission.md
   #- k8s/operators.md
   #- k8s/operators-design.md
   #- k8s/staticpods.md

slides/kube-selfpaced.yml

@@ -111,6 +111,7 @@ content:
   - k8s/horizontal-pod-autoscaler.md
 -
   - k8s/extending-api.md
+  - k8s/admission.md
   - k8s/operators.md
   - k8s/operators-design.md
   - k8s/owners-and-dependents.md

slides/kube-twodays.yml

@@ -102,6 +102,7 @@ content:
   - k8s/local-persistent-volumes.md
   - k8s/portworx.md
   #- k8s/extending-api.md
+  #- k8s/admission.md
   #- k8s/operators.md
   #- k8s/operators-design.md
   #- k8s/staticpods.md

webhooks/admission/README.md

@@ -0,0 +1,38 @@
+# Validating Webhook Demo
+
+This webhook applies to pods. If a pod has a label `color`, then
+that label must be `blue`, `green`, or `red`. If it is anything
+else, the pod will be rejected. Furthermore, once the `color` label
+has been set, it cannot be changed or removed.
+
+
+## Cheatsheet
+
+Generating a key pair and a self-signed certificate:
+```bash
+NAMESPACE=webhooks
+SERVICE=admission
+CN=$SERVICE.$NAMESPACE.svc
+openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem \
+	-days 30 -subj /CN=$CN -addext subjectAltName=DNS:$CN
+```
+(The API server *requires* that the certificate uses a `subjectAltName`.)
+
+Loading up the key and certificate in a secret:
+```bash
+kubectl create secret tls $SERVICE \
+	--namespace=$NAMESPACE --cert=cert.pem --key=key.pem
+```
+
+After loading the webhook configuration, patch up the `caBundle`:
+```bash
+CA=$(base64 -w0 < cert.pem)
+PATCH='[{"op": "replace",
+         "path": "/webhooks/0/clientConfig/caBundle",
+         "value":"'$CA'"}]'
+kubectl patch validatingwebhookconfiguration \
+        admission.webhook.container.training \
+	--type='json' -p="$PATCH"
+```
+
+Remember to always look at the logs of the API server while troubleshooting this!

webhooks/admission/docker-compose.yml

@@ -0,0 +1,32 @@
+version: "3"
+
+services:
+
+  ngrok-echo:
+    build: ngrok
+    command: ngrok http --log=stdout localhost:3000
+    ports:
+    - 3000
+
+  echo:
+    network_mode: service:ngrok-echo
+    image: node
+    command: npx http-echo-server
+
+  ngrok-flask:
+    build: ngrok
+    command: ngrok http --log=stdout localhost:5000
+    ports:
+    - 5000
+
+  flask:
+    network_mode: service:ngrok-flask
+    build: flask
+    volumes:
+    - ./flask:/src
+    working_dir: /src
+    environment:
+      FLASK_APP: webhook.py
+      FLASK_ENV: development
+    command: flask run --host=0.0.0.0
+

webhooks/admission/flask/Dockerfile

@@ -0,0 +1,2 @@
+FROM python
+RUN pip install Flask IPython PyYAML

webhooks/admission/flask/webhook.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python
+
+ACCEPTED_COLORS = ["blue", "green", "red"]
+
+import json
+import pprint
+import yaml
+
+from flask import Flask, request
+
+app = Flask(__name__)
+
+
+# Since most or all the things that we might want to print are going to
+# be Kubernetes resource manifests (or fragments thereof), and these
+# manifests are usually represented as YAML, we might as well print them
+# as YAML when we need to view them.
+def debug(obj):
+    app.logger.debug(yaml.dump(obj))
+
+
+@app.route("/", methods=["POST"])
+def webhook():
+
+    payload = json.loads(request.data)
+    debug(payload)
+
+    # Let's check that we were called the right way.
+    assert payload["kind"] == "AdmissionReview"
+    uid = payload["request"]["uid"]
+    pod = payload["request"]["object"]
+    assert pod["kind"] == "Pod"
+    assert pod["apiVersion"] == "v1"
+
+    # If the pod has a "color" label, it has to be one of the accepted ones.
+    labels = pod["metadata"].get("labels", {})
+    if "color" in labels:
+        color = labels["color"]
+        if color not in ACCEPTED_COLORS:
+            return response(
+                uid,
+                False,
+                "color {!r} is not in the allowed set ({!r})".format(
+                    color, ACCEPTED_COLORS
+                ),
+            )
+
+    # If this is an UPDATE request, oldObject has the old version.
+    # (Otherwise, it's null aka None in Python.)
+    oldPod = payload["request"]["oldObject"]
+    if oldPod:
+	    oldLabels = oldPod["metadata"].get("labels", {})
+	    # If the pod *had* a "color" label, it cannot be removed.
+	    if "color" in oldLabels and "color" not in labels:
+	        return response(uid, False, "cannot remove color from a colored pod")
+	    # The "color" label also cannot be changed to a different value.
+	    if "color" in oldLabels and "color" in labels:
+	        if oldLabels["color"] != labels["color"]:
+	            return response(uid, False, "cannot change color of a pod")
+
+    # Otherwise, accept the request.
+    return response(uid, True)
+
+
+def response(uid, allowed, message=None):
+    payload = {
+        "apiVersion": "admission.k8s.io/v1",
+        "kind": "AdmissionReview",
+        "response": {"uid": uid, "allowed": allowed},
+    }
+    if message is not None:
+        payload["response"]["status"] = {"message": message}
+    return payload

webhooks/admission/k8s/webhook-configuration.yaml

@@ -0,0 +1,22 @@
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: admission.webhook.container.training
+webhooks:
+- name: admission.webhook.container.training
+  rules:
+  - apiGroups:   [""]
+    apiVersions: ["v1"]
+    operations:  ["CREATE", "UPDATE"]
+    resources:   ["pods"]
+    scope:       "Namespaced"
+  clientConfig:
+    #service:
+    #  namespace: webhooks
+    #  name: admission
+    #caBundle: ""
+    #url: https://xxxxyyyyzzzz.ngrok.io
+  admissionReviewVersions: ["v1"]
+  sideEffects: None
+  timeoutSeconds: 5
+  failurePolicy: Ignore # defaults to Fail

webhooks/admission/k8s/webhook-server.yaml

@@ -0,0 +1,51 @@
+---
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: admission
+spec:
+  selector:
+    matchLabels:
+      app: admission
+  template:
+    metadata:
+      labels:
+        app: admission
+    spec:
+      volumes:
+      - name: tls
+        secret:
+          secretName: admission
+      - name: app
+        configMap:
+          name: admission
+      containers:
+      - name: admission
+        image: python
+        command:
+        - sh
+        - -c
+        - |
+          pip install Flask gunicorn PyYAML
+          exec gunicorn \
+            --bind 0.0.0.0:443 --keyfile /tls/tls.key --certfile /tls/tls.crt \
+            --access-logfile - \
+            --chdir /app webhook:app
+        volumeMounts:
+        - name: tls
+          mountPath: /tls
+        - name: app
+          mountPath: /app
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: admission
+spec:
+  selector:
+    app: admission
+  ports:
+  - protocol: TCP
+    port: 443
+    targetPort: 443
+  type: ClusterIP

webhooks/admission/ngrok/Dockerfile

@@ -0,0 +1,7 @@
+FROM alpine
+RUN apk add curl unzip
+RUN curl -O https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
+RUN unzip ngrok-stable-linux-amd64
+FROM alpine
+COPY --from=0 /ngrok /usr/local/bin/ngrok
+