github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-02-14 09:39:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

🏭️ Rework Kyverno chapter

Browse Source
This commit is contained in:
Jérôme Petazzoni
2025-05-11 18:34:11 +02:00
parent cd346ecace
commit 785d704726
7 changed files with 226 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
k8s/kyverno-pod-color-1.yaml
View File

@@ -3,7 +3,6 @@ kind: ClusterPolicy
metadata:
name: pod-color-policy-1
spec:
validationFailureAction: Enforce
rules:
- name: ensure-pod-color-is-valid
match:
@@ -18,5 +17,6 @@ spec:
operator: NotIn
values: [ red, green, blue ]
validate:
failureAction: Enforce
message: "If it exists, the label color must be red, green, or blue."
deny: {}

2
k8s/kyverno-pod-color-2.yaml
View File

@@ -3,7 +3,6 @@ kind: ClusterPolicy
metadata:
name: pod-color-policy-2
spec:
validationFailureAction: Enforce
background: false
rules:
- name: prevent-color-change
@@ -22,6 +21,7 @@ spec:
operator: NotEquals
value: ""
validate:
failureAction: Enforce
message: "Once label color has been added, it cannot be changed."
deny:
conditions:

6
k8s/kyverno-pod-color-3.yaml
View File

@@ -3,7 +3,6 @@ kind: ClusterPolicy
metadata:
name: pod-color-policy-3
spec:
validationFailureAction: Enforce
background: false
rules:
- name: prevent-color-change
@@ -22,7 +21,6 @@ spec:
operator: Equals
value: ""
validate:
failureAction: Enforce
message: "Once label color has been added, it cannot be removed."
deny:
conditions:
deny: {}

5
slides/images/admission-control-phases.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

33
slides/k8s/admission.md
View File

@@ -32,7 +32,7 @@
- Problem mitigation
*block nodes with vulnerable kernels, inject log4j mitigations...*
*block nodes with vulnerable kernels, inject log4j mitigations, rewrite images...*
- Extended validation for operators
@@ -583,19 +583,38 @@ Shell to the rescue!
---
## Coming soon...
## Real world examples
- [kube-image-keeper][kuik] rewrites image references to use cached images
(e.g. `nginx` → `localhost:7439/nginx`)
- [Kyverno] implements very extensive policies
(validation, generation... it deserves a whole chapter on its own!)
[kuik]: https://github.com/enix/kube-image-keeper
[kyverno]: https://kyverno.io/
---
## Alternatives
- Kubernetes Validating Admission Policies
- Integrated with the Kubernetes API server
- Relatively recent (alpha: 1.26, beta: 1.28, GA: 1.30)
- Lets us define policies using [CEL (Common Expression Language)][cel-spec]
- Declare validation rules with Common Expression Language ([CEL][cel-spec])
- Available in beta in Kubernetes 1.28 <!-- ##VERSION## -->
- Validation is done entirely within the API server
- Check this [CNCF Blog Post][cncf-blog-vap] for more details
(no external webhook = no latency, no deployment complexity...)
[cncf-blog-vap]: https://www.cncf.io/blog/2023/09/14/policy-management-in-kubernetes-is-changing/
- Not as powerful as full-fledged webhook engines like Kyverno
(see e.g. [this page of the Kyverno doc][kyverno-vap] for a comparison)
[kyverno-vap]: https://kyverno.io/docs/policy-types/validating-policy/
[cel-spec]: https://github.com/google/cel-spec
???

94
slides/k8s/extending-api.md
View File

@@ -6,7 +6,7 @@ We are going to cover:
- Controllers
- Dynamic Admission Webhooks
- Admission Control
- Custom Resource Definitions (CRDs)
@@ -128,23 +128,36 @@ then make or request changes where needed.*
---
## Admission controllers
## Admission control
- Admission controllers can vet or transform API requests
- Validate (approve/deny) or mutate (modify) API requests
- The diagram on the next slide shows the path of an API request
- In modern Kubernetes, we have at least 3 ways to achieve that:
(courtesy of Banzai Cloud)
- [admission controllers][ac-controllers] (built in the API server)
- [dynamic admission control][ac-webhooks] (with webhooks)
- [validating admission policies][ac-vap] (using CEL, Common Expression Language)
- More is coming; e.g. [mutating admission policies][ac-map] (alpha in Kubernetes 1.32)
[ac-controllers]: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
[ac-webhooks]: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
[ac-vap]: https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
[ac-map]: https://kubernetes.io/docs/reference/access-authn-authz/mutating-admission-policy/
---
class: pic
![API request lifecycle](images/api-request-lifecycle.png)
![API request lifecycle; from Kubernetes documentation](images/admission-control-phases.svg)
---
## Types of admission controllers
## Admission controllers
- Built in the API server
- *Validating* admission controllers can accept/reject the API call
@@ -156,9 +169,13 @@ class: pic
- There are a number of built-in admission controllers
(see [documentation](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do) for a list)
([and a bunch of them are enabled by default][ac-default])
- We can also dynamically define and register our own
- They can be enabled/disabled with API server command-line flags
(this is not always possible when using *managed* Kubernetes!)
[ac-default]: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default
---
@@ -202,6 +219,8 @@ class: extra-details
---
class: extra-details
## Webhook Configuration
- A ValidatingWebhookConfiguration or MutatingWebhookConfiguration contains:
@@ -229,15 +248,39 @@ class: extra-details
- Sidecar injection
(Used by some service meshes)
(used by some service meshes)
- Type validation
(More on this later, in the CRD section)
(more on this later, in the CRD section)
- And many other creative + useful scenarios!
(for example in [kube-image-keeper][kuik], to rewrite image references)
[kuik]: https://github.com/enix/kube-image-keeper
---
## Kubernetes API types
## Validating Admission Policies
- Relatively recent (alpha: 1.26, beta: 1.28, GA: 1.30)
- Declare validation rules with Common Expression Language (CEL)
- Validation is done entirely within the API server
(no external webhook = no latency, no deployment complexity...)
- Not as powerful as full-fledged webhook engines like Kyverno
(see e.g. [this page of the Kyverno doc][kyverno-vap] for a comparison)
[kyverno-vap]: https://kyverno.io/docs/policy-types/validating-policy/
---
## Kubernetes API resource types
- Almost everything in Kubernetes is materialized by a resource
@@ -271,21 +314,21 @@ class: extra-details
## Examples
- Representing configuration for controllers and operators
(e.g. Prometheus scrape targets, gitops configuration, certificates...)
- Representing composite resources
(e.g. clusters like databases, messages queues ...)
(e.g. database cluster, message queue...)
- Representing external resources
(e.g. virtual machines, object store buckets, domain names ...)
- Representing configuration for controllers and operators
(e.g. custom Ingress resources, certificate issuers, backups ...)
(e.g. virtual machines, object store buckets, domain names...)
- Alternate representations of other objects; services and service instances
(e.g. encrypted secret, git endpoints ...)
(e.g. encrypted secret, git endpoints...)
---
@@ -339,17 +382,18 @@ class: extra-details
---
## Documentation
## And more...
- [Custom Resource Definitions: when to use them](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
- Some specifics areas of Kubernetes also have extension points
- [Custom Resources Definitions: how to use them](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/)
- Example: scheduler
- [Built-in Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
- it's possible to [customize the behavior of the scheduler][sched-config]
- [Dynamic Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
- or even run [multiple schedulers][sched-multiple]
- [Aggregation Layer](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)
[sched-config]: https://kubernetes.io/docs/reference/scheduling/config/
[sched-multiple]: https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/
???

219
slides/k8s/kyverno.md
View File

@@ -1,44 +1,78 @@
# Policy Management with Kyverno
- The Kubernetes permission management system is very flexible ...
- Kyverno is a policy engine for Kubernetes
- ... But it can't express *everything!*
- It has many use cases, including:
- Examples:
- validating resources when they are created/edited
<br/>(blocking or logging violations)
- forbid using `:latest` image tag
- preventing some modifications
<br/>(e.g. restricting modifications to some fields, labels...)
- enforce that each Deployment, Service, etc. has an `owner` label
<br/>(except in e.g. `kube-system`)
- modifying resources automatically
- enforce that each container has at least a `readinessProbe` healthcheck
- generating resources automatically
- How can we address that, and express these more complex *policies?*
- clean up resources automatically
---
## Admission control
## Examples (validation)
- The Kubernetes API server provides a generic mechanism called *admission control*
- [Disallow `:latest` tag](https://kyverno.io/policies/best-practices/disallow-latest-tag/disallow-latest-tag/)
- Admission controllers will examine each write request, and can:
- [Disallow secrets in environment variables](https://kyverno.io/policies/other/disallow-secrets-from-env-vars/disallow-secrets-from-env-vars/)
- approve/deny it (for *validating* admission controllers)
- [Require that containers drop all capabilities](https://kyverno.io/policies/best-practices/require-drop-all/require-drop-all/)
- additionally *update* the object (for *mutating* admission controllers)
- [Prevent creation of Deployment, ReplicaSet, etc. without an HPA](https://kyverno.io/policies/other/check-hpa-exists/check-hpa-exists/)
- These admission controllers can be:
- [Forbid CPU limits](https://kyverno.io/policies/other/forbid-cpu-limits/forbid-cpu-limits/)
- plug-ins built into the Kubernetes API server
<br/>(selectively enabled/disabled by e.g. command-line flags)
- [Check that memory requests are equal to limits](https://kyverno.io/policies/other/memory-requests-equal-limits/memory-requests-equal-limits/)
- webhooks registered dynamically with the Kubernetes API server
- [Require containers to have healthchecks](https://kyverno.io/policies/best-practices/require-probes/require-probes/)
---
## What's Kyverno?
## Examples (mutation)
- Policy management solution for Kubernetes
- [Automatically add environment variables from a ConfigMap](https://kyverno.io/policies/other/add-env-vars-from-cm/add-env-vars-from-cm/)
- [Add image as an environment variable](https://kyverno.io/policies/other/add-image-as-env-var/add-image-as-env-var/)
- [Add image `LABEL` as an environment variable](https://kyverno.io/policies/other/inject-env-var-from-image-label/inject-env-var-from-image-label/)
- [When creating a Deployment, copy some labels from its Namespace](https://kyverno.io/policies/other/copy-namespace-labels/copy-namespace-labels/)
- [Automatically restart a given Deployment when a given ConfigMap changes](https://kyverno.io/policies/other/restart-deployment-on-secret-change/restart-deployment-on-secret-change/)
---
## Examples (generation)
- [Automatically create a PDB when a Deployment is created](https://kyverno.io/policies/other/create-default-pdb/create-default-pdb/)
- [Create an event when an object is deleted (for auditing purposes)](https://kyverno.io/policies/other/audit-event-on-delete/audit-event-on-delete/)
- [Create an audit event when using `kubectl exec`](https://kyverno.io/policies/other/audit-event-on-exec/audit-event-on-exec/)
- [Automatically create a Secret (e.g. for registry auth) when a Namespace is created](https://kyverno.io/policies/other/sync-secrets/sync-secrets/)
---
## Examples (advanced validation)
- [Only allow root user in images coming from a trusted registry](https://kyverno.io/policies/other/only-trustworthy-registries-set-root/only-trustworthy-registries-set-root/)
- [Prevent images that haven't been checked by a vulnerability scanner](https://kyverno.io/policies/other/require-vulnerability-scan/require-vulnerability-scan/)
- [Prevent ingress with the same host and path](https://kyverno.io/policies/other/unique-ingress-host-and-path/unique-ingress-host-and-path/)
---
## More about Kyverno
- Open source (https://github.com/kyverno/kyverno/)
@@ -54,49 +88,21 @@
---
## What can Kyverno do?
- *Validate* resource manifests
(accept/deny depending on whether they conform to our policies)
- *Mutate* resources when they get created or updated
(to add/remove/change fields on the fly)
- *Generate* additional resources when a resource gets created
(e.g. when namespace is created, automatically add quotas and limits)
- *Audit* existing resources
(warn about resources that violate certain policies)
---
## How does it do it?
## How does it work?
- Kyverno is implemented as a *controller* or *operator*
- It typically runs as a Deployment on our cluster
- Policies are defined as *custom resource definitions*
- Policies are defined as *custom resources*
- They are implemented with a set of *dynamic admission control webhooks*
--
🤔
--
- Let's unpack that!
---
## Custom resource definitions
- When we install Kyverno, it will register new resource types:
- When we install Kyverno, it will register new resource types, including:
- Policy and ClusterPolicy (per-namespace and cluster-scope policies)
@@ -112,32 +118,6 @@
---
## Dynamic admission control webhooks
- When we install Kyverno, it will register a few webhooks for its use
(by creating ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources)
- All subsequent resource modifications are submitted to these webhooks
(creations, updates, deletions)
---
## Controller
- When we install Kyverno, it creates a Deployment (and therefore, a Pod)
- That Pod runs the server used by the webhooks
- It also runs a controller that will:
- run checks in the background (and generate PolicyReport objects)
- process GenerateRequest objects asynchronously
---
## Kyverno in action
- We're going to install Kyverno on our cluster
@@ -152,7 +132,6 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
(It's also possible to install with a single YAML manifest.)
.lab[
- Install Kyverno:
@@ -326,6 +305,8 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
(with an error similar to `JMESPAth query failed: Unknown key ... in path`)
- If a precondition fails, the policy will be skipped altogether (and ignored!)
- To work around that, [use an OR expression][non-existence-checks]:
`{{ requests.object.metadata.labels.color || '' }}`
@@ -334,7 +315,7 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
(e.g. in *preconditions*, a missing label would evalute to an empty string)
[non-existence-checks]: https://kyverno.io/docs/writing-policies/jmespath/#non-existence-checks
[non-existence-checks]: https://kyverno.io/docs/policy-types/cluster-policy/jmespath/#non-existence-checks
---
@@ -363,11 +344,39 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
---
## `background`
## `spec.rules.validate.failureAction`
- What is this `background: false` option, and why do we need it?
- By default, this is set to `Audit`
--
- This means that rule violations are not enforced
- They still generate a warning (at the API level) and a PolicyReport
(more on that later)
- We need to change the `failureAction` to `Enforce`
---
## `background`, `admission`, `emitWarning`
- Policies have three boolean flags to control what they do and when
- `admission` = run that policy at admission
(when an object gets created/updated and validation controllers get invoked)
- `background` = run that policy in the background
(periodically check if existing objects fit the policy)
- `emitWarning` = generate an `Event` of type `Warning` associated to the validated objct
(visible with e.g. `kubectl describe` on that object)
---
## Background checks
- Admission controllers are only invoked when we change an object
@@ -379,17 +388,15 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
(we'll see later how they are reported)
- `background: false` disables that
- `background: true/false` controls that
--
- Alright, but ... *why* do we need it?
- When would we want to disabled it? 🤔
---
## Accessing `AdmissionRequest` context
- In this specific policy, we want to prevent an *update*
- In some of our policies, we want to prevent an *update*
(as opposed to a mere *create* operation)
@@ -403,10 +410,6 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
- We access the `AdmissionRequest` object through `{{ request }}`
--
- Alright, but ... what's the link with `background: false`?
---
## `{{ request }}`
@@ -419,6 +422,16 @@ The recommended [installation method][install-kyverno] is to use Helm charts.
(it can only be used when an object is actually created/updated/deleted)
--
- *Well, actually...*
--
- Kyverno exposes `{{ request.object }}` and `{{ request.namespace }}`
(see [the documentation](https://kyverno.io/docs/policy-reports/background/) for details!)
---
## Immutable primary colors, take 3
@@ -608,7 +621,19 @@ class: extra-details
---
## Footprint
## Footprint (current versions)
- 14 CRDs
- 10 webhooks
- 6 services, 4 Deployments, 2 ConfigMaps
- Internal resources (GenerateRequest) "parked" in a Namespace
---
## Footprint (older versions)
- 8 CRDs
@@ -616,18 +641,14 @@ class: extra-details
- 2 Services, 1 Deployment, 2 ConfigMaps
- Internal resources (GenerateRequest) "parked" in a Namespace
- Kyverno packs a lot of features in a small footprint
*We can see the number of resources increased over time, as Kyverno added features.*
---
I
## Strengths
- Kyverno is very easy to install
(it's harder to get easier than one `kubectl apply -f`)
- The setup of the webhooks is fully automated
(including certificate generation)
@@ -638,6 +659,10 @@ class: extra-details
(e.g. `matchExpressions`)
- It has pretty good documentation, including many examples
- There is also a CLI tool (not discussed here)
---
## Caveats