github/container.training

Fork 0

mirror of https://github.com/jpetazzo/container.training.git synced 2026-02-14 17:49:59 +00:00

Files

Jérôme Petazzoni 078e799666 ⏫ Update Kustomize content

2025-10-28 16:22:54 +01:00

14 KiB

Raw Blame History

Kustomize

Kustomize lets us transform Kubernetes resources:

YAML + kustomize → new YAML
Starting point = valid resource files

(i.e. something that we could load with kubectl apply -f)
Recipe = a kustomization file

(describing how to transform the resources)
Result = new resource files

(that we can load with kubectl apply -f)

Pros and cons

Relatively easy to get started

(just get some existing YAML files)
Easy to leverage existing "upstream" YAML files

(or other kustomizations)
Integrated with kubectl
No central index like the Artifact Hub (but is there a need for it?)
Some modifications will be difficult to make
100% file-based (won't use command-line flags, environment variables...)

Kustomize in a nutshell

Get some valid YAML (our "resources")
Write a kustomization (technically, a file named kustomization.yaml)
- reference our resources
- reference other kustomizations
- add some patches and other transformations
- ...
Use that kustomization either with kustomize build or kubectl apply -k
Write new kustomizations referencing the first one to handle minor differences

A simple kustomization

This features a Deployment, Service, and Ingress (in separate files), and a couple of patches (to change the number of replicas and the hostname used in the Ingress).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
patches:
- path: scale-deployment.yaml
- path: ingress-hostname.yaml

On the next slide, let's see a more complex example ...

A more complex Kustomization

.small[

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonAnnotations:
  last-commit-message: "Bump libfoo to version 1.2.3"
labels:
- pairs:
    last-commit-hash: "39bc2d"
resources:
- github.com/example/front?ref=tag-or-branch
- github.com/example/api?ref=tag-or-branch
- db/
- workers/
- ingress.yaml
- rbac.yaml
configMapGenerator:
- name: appconfig
  files:
  - global.conf
  - local.conf=prod.conf
patches:
- path: healthchecks.yaml
- path: resources-requests-and-limits.yaml

]

Architecture

Internally, Kustomize has three phases:
- generators (=produce a bunch of YAML, e.g. by reading manifest files)
- transformers (=transform/patch that YAML in various ways)
- validators (=has the ability to stop the process is something's wrong)
In the previous examples:
- resources and configMapGenerator are generators
- commonAnnotations, labels, patches are transformers

Glossary

A base is a kustomization that is referred to by other kustomizations
An overlay is a kustomization that refers to other kustomizations
A kustomization can be both a base and an overlay at the same time

(a kustomization can refer to another, which can refer to a third)
A variant is the final outcome of applying bases + overlays

(See the kustomize glossary for more definitions!)

Kustomize workflows

The Kustomize documentation proposes two different workflows
Bespoke configuration
- base and overlays managed by the same team
Off-the-shelf configuration (OTS)
- base and overlays managed by different teams
- base is regularly updated by "upstream" (e.g. a vendor)
- our overlays and patches should (hopefully!) apply cleanly
- we may regularly update the base, or use a remote base
No technical difference; these are just different use-cases!

Managing `kustomization.yaml`

There are many ways to manage kustomization.yaml files, including:
- the kustomize CLI
- opening the file with our favorite text editor
- ~~web wizards like Replicated Ship~~ (deprecated)
Let's see these in action!

Working with the `kustomize` CLI

General workflow:

kustomize create to generate an empty kustomization.yaml file
kustomize edit add resource to add Kubernetes YAML files to it
kustomize edit add patch to add patches to said resources
kustomize edit add ... or kustomize edit set ... (many options!)
kustomize build | kubectl apply -f- or kubectl apply -k .
Repeat steps 4-5 as many times as necessary!

Why work with the CLI?

Editing manually can introduce errors and typos
With the CLI, we don't need to remember the name of all the options and parameters

(just add --help after any command to see possible options!)
Make sure to install the completion and try e.g. kustomize edit add [TAB][TAB]

`kustomize create`

.lab[

Change to a new directory:
```
mkdir ~/kustomcoins
cd ~/kustomcoins
```

Run kustomize create with the kustomcoins repository:

kustomize create --resources https://github.com/jpetazzo/kubercoins

Run kustomize build | kubectl apply -f-

]

`kubectl` integration

Kustomize has been integrated in kubectl (since Kubernetes 1.14)
- kubectl kustomize is an equivalent to kustomize build
- commands that use -f can also use -k (kubectl apply/delete/...)
The kustomize tool is still needed if we want to use create, edit, ...
Kubernetes 1.14 to 1.20 uses Kustomize 2.0.3
Kubernetes 1.21 jumps to Kustomize 4.1.2
Kustomize is now officially part of sig-cli
kubectl is usually in sync with recent versions of Kustomize

(but it can still lag behind a bit, so some features might not be available!)

Kustomize features

A good starting point for Kustomize features is the Kustomization File reference
Unfortunately, the Kustomize documentation is far from perfect
Some features are undocumented
Some features are deprecated / replaced by others

(but that's not well indicated in the docs; e.g. commonLabels)
Some features are documented but not released yet

(e.g. regex selectors in replacements as of October 2025)

Kustomization basics

bases (deprecated), resources

include YAML manifests
buildMetadata

automatically generate interesting labels / annotations
commonAnnotations, commonLabels, labels

add custom labels / annotations to all resources
configMapGenerator, secretGenerator, generatorOptions

generate ConfigMaps and Secrets; appending a hash suffix to their name

Transforming resources

patches patchesJson6902, patchesStrategicMerge
- perform (almost) arbitrary modifications to resources
- can be used to remove fields or even entire resources
- patches can be in separate files, or inlined within Kustomization file
- patches can apply to a specific resource, or to selected resources
images, namePrefix, namespace, nameSuffix, replicas
- helpers for basic modifications; concise and easy to use, but limited

More transformations

replacements
- update individual fields (a bit like patches)
- can also do substring updates (e.g. replace an image registry)
- can copy a field from a place to another (e.g. a whole container spec)
- can apply to individual resources or to selected resources
- resources can also be filtered out (to be excluded from replacement)

Teach Kustomize new tricks

crds = make Kustomize aware of e.g. ConfigMap fields in CRDs
openapi = give a schema to teach Kustomize about merge keys etc.
sortOptions = output resources in a specific order
helmCharts = evaluate a Helm chart template

(only available with --enable-helm flag; not standard / GA yet!)
vars = define variables and reuse them elsewhere

(limited to some specific fields; ...actually, it's being deprecated already!)
components = somewhat similar to an "include"

(points to a "component" and invoke all its generators and transformers)

Remote bases

Kustomize can also use bases that are remote git repositories
Example:

https://github.com/kubernetes-sigs/kustomize//examples/multibases/?ref=v3.3.1
See remoteBuild.md for details about remote targets URL format
Note that this only works for kustomizations, not individual resources

(the specified repository or directory must contain a kustomization.yaml file)

What Kustomize cannot do

By design, there are a number of things that Kustomize won't do
For instance:
- using command-line arguments or environment variables to generate a variant
- overlays can only add resources, not remove them¹
See the full list of eschewed features for more details

.footnote[¹That's actually not true; patches can remove resources.]

Changing image references

We're going to see a few different ways to change image references
Let's assume that our app uses multiple images:

redis, dockercoins/hasher, dockercoins/worker...
We want to update the dockercoins/* images to use a registry mirror

(e.g. ghcr.io/dockercoins/*)
We don't want to touch the other images

Changing images with the CLI

We can use the following CLI command:

kustomize edit set image name=[newName][:newTag][@digest]

[] denote optional parameters
: and @ are the delimiters used to indicate a field

Examples:

kustomize edit set image dockercoins/worker=ghcr.io/dockercoins/worker
kustomize edit set image dockercoins/worker=ghcr.io/dockercoins/worker:v0.2
kustomize edit set image dockercoins/worker=:v0.2

This will add entries in the images: section of the kustomization.

Changing images with `images:`

Here are a few examples of the images: Kustomization directive:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
images:
- name: postgres
  newName: harbor.enix.io/my-postgres
- name: dockercoins/worker
  newTag: v0.2
- name: dockercoins/hasher
  newName: registry.dockercoins.io/hasher
  newTag: v0.2
- name: alpine
  digest: sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

`images:` in practice

This is what we would need for our app:

- name: dockercoins/hasher
  newName: ghcr.io/dockercoins/hasher
- name: dockercoins/rng
  newName: ghcr.io/dockercoins/rng
- name: dockercoins/webui
  newName: ghcr.io/dockercoins/webui
- name: dockercoins/worker
  newName: ghcr.io/dockercoins/worker

It works, but requires two lines per image. Can we do better? 🤔

`images:`, pros and cons

Very convenient when the same image appears multiple times
Very convenient to define tags (or pin to hashes) outside of the main YAML
Doesn't support wildcard or generic substitutions:
- cannot "replace dockercoins/* with ghcr.io/dockercoins/*"
- cannot "tag all dockercoins/* with v0.2"
Only patches "well-known" image fields (won't work with CRDs referencing images)
If our app uses 4 images, we'll need 4 images: section in the Kustomization file

`PrefixSuffixTransformer`

Internally, the namePrefix directive relies on a PrefixSuffixTransformer
By default, that transformer acts on metadata.name
It can be invoked manually and configured to act on other fields

(for instance, spec.template.spec.containers.image in a Deployment manifest)

`PrefixSuffixTransformer` in action

@@INCLUDE[k8s/kustomize-examples/registry-with-prefix-transformer/microservices/kustomization.yaml]

However, this will transform all Deployments!
Or rather, all resources with a spec.template.spec.containers.image field

Limiting `PrefixSuffixTransformer`

PrefixSuffixTransformer applies to all resources

(as of Kustomize 5.7 (October 2025), there is no way to specify a filter)
One workaround:
- break down the app in multiple Kustomizations
- in one Kustomization, put the components that need to be transformed
- put the other components in another Kustomization
Not great if multiple transformations need to be applied on different resources

`replacements`

Allows to entirely replace a field
Can also replace part of a field (using some delimiter, e.g. / for images)
Replacements can apply to selected resources
Let's see an example!

`replacements` in action

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dockercoins.yaml
replacements:
- sourceValue: ghcr.io/dockercoins
  targets:
  - select:
      kind: Deployment
      labelSelector: "app in (hasher,rng,webui,worker)"
    fieldPaths:
    - spec.template.spec.containers.*.image
    options:
      delimiter: "/"
      index: 0

(Note the different fieldPath format, compared to the earlier transformer!)

Discussion

There are multiple ways to rewrite image references to use a registry mirror
They all have pros and cons
Main problem: they require to enumerate the resources that we want to transform

(or to split them in a separate Kustomization)
No (easy?) way to do something like:

"Replace all images starting by dockercoins/ with ghcr.io/dockercoins/"

Inconsistencies

Sometimes it's possible to filter / select resources, sometimes not
When it's possible, it's not always done the same way
Field paths are also different in different places:

/spec/template/spec/containers/0/image in JSON patches

spec.template.spec.containers.0.image in replacements' fieldPaths

spec/template/spec/containers/image in transformers' fieldSpecs

.spec.template.spec.containers[].image with tools like jq
fieldPaths also have interesting extensions, like:

spec.template.spec.containers.[name=hello].image

Conclusions

It's possible to do a lot of transformations with Kustomize
In complex scenarios, it can quickly becomes a maintenance nightmare
One possible strategy:
- keep each Kustomization as simple as possible
- compose multiple Kustomizations together
See this kustomization for a creative example with sidecars and more!

???

:EN:- Packaging and running apps with Kustomize :FR:- Packaging d'applications avec Kustomize

14 KiB Raw Blame History