container.training/slides/k8s/kustomize.md

Kustomize

• Kustomize lets us transform Kubernetes resources:

  YAML + kustomize → new YAML

• Starting point = valid resource files

  (i.e. something that we could load with kubectl apply -f)

• Recipe = a kustomization file

  (describing how to transform the resources)

• Result = new resource files

  (that we can load with kubectl apply -f)

---

Pros and cons

• Relatively easy to get started

  (just get some existing YAML files)

• Easy to leverage existing "upstream" YAML files

  (or other kustomizations)

• Somewhat integrated with kubectl

  (but only "somewhat" because of version discrepancies)

• Less complex than e.g. Helm, but also less powerful

• No central index like the Artifact Hub (but is there a need for it?)

---

Kustomize in a nutshell

• Get some valid YAML (our "resources")

• Write a kustomization (technically, a file named kustomization.yaml)

  • reference our resources

  • reference other kustomizations

  • add some patches

  • ...

• Use that kustomization either with kustomize build or kubectl apply -k

• Write new kustomizations referencing the first one to handle minor differences

---

A simple kustomization

This features a Deployment, Service, and Ingress (in separate files), and a couple of patches (to change the number of replicas and the hostname used in the Ingress).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesStrategicMerge:
- scale-deployment.yaml
- ingress-hostname.yaml
resources:
- deployment.yaml
- service.yaml
- ingress.yaml

On the next slide, let's see a more complex example ...

---

A more complex Kustomization

.small[

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonAnnotations:
  mood: 😎
commonLabels:
  add-this-to-all-my-resources: please
namePrefix: prod-
patchesStrategicMerge:
- prod-scaling.yaml
- prod-healthchecks.yaml
bases:
- api/
- frontend/
- db/
- github.com/example/app?ref=tag-or-branch
resources:
- ingress.yaml
- permissions.yaml
configMapGenerator:
- name: appconfig
  files:
  - global.conf
  - local.conf=prod.conf

]

---

Glossary

• A base is a kustomization that is referred to by other kustomizations

• An overlay is a kustomization that refers to other kustomizations

• A kustomization can be both a base and an overlay at the same time

  (a kustomization can refer to another, which can refer to a third)

• A patch describes how to alter an existing resource

  (e.g. to change the image in a Deployment; or scaling parameters; etc.)

• A variant is the final outcome of applying bases + overlays

(See the kustomize glossary for more definitions!)

---

What Kustomize cannot do

• By design, there are a number of things that Kustomize won't do

• For instance:

  • using command-line arguments or environment variables to generate a variant

  • overlays can only add resources, not remove them

• See the full list of eschewed features for more details

---

Kustomize workflows

• The Kustomize documentation proposes two different workflows

• Bespoke configuration

  • base and overlays managed by the same team

• Off-the-shelf configuration (OTS)

  • base and overlays managed by different teams

  • base is regularly updated by "upstream" (e.g. a vendor)

  • our overlays and patches should (hopefully!) apply cleanly

  • we may regularly update the base, or use a remote base

---

Remote bases

• Kustomize can also use bases that are remote git repositories

• Examples:

  github.com/jpetazzo/kubercoins (remote git repository)

  github.com/jpetazzo/kubercoins?ref=kustomize (specific tag or branch)

• Note that this only works for kustomizations, not individual resources

  (the specified repository or directory must contain a kustomization.yaml file)

---

class: extra-details

Hashicorp go-getter

• Some versions of Kustomize support additional forms for remote resources

• Examples:

  https://releases.hello.io/k/1.0.zip (remote archive)

  https://releases.hello.io/k/1.0.zip//some-subdir (subdirectory in archive)

• This relies on hashicorp/go-getter

• ... But it prevents Kustomize inclusion in kubectl

• Avoid them!

• See kustomize#3578 for details

---

Managing kustomization.yaml

• There are many ways to manage kustomization.yaml files, including:

  • the kustomize CLI

  • opening the file with our favorite text editor

  • web wizards like Replicated Ship (deprecated)

• Let's see these in action!

---

Working with the kustomize CLI

General workflow:

1. kustomize create to generate an empty kustomization.yaml file

2. kustomize edit add resource to add Kubernetes YAML files to it

3. kustomize edit add patch to add patches to said resources

4. kustomize edit add ... or kustomize edit set ... (many options!)

5. kustomize build | kubectl apply -f- or kubectl apply -k .

6. Repeat steps 4-5 as many times as necessary!

---

Why work with the CLI?

• Editing manually can introduce errors and typos

• With the CLI, we don't need to remember the name of all the options and parameters

  (just add --help after any command to see possible options!)

• Make sure to install the completion and try e.g. kustomize edit add [TAB][TAB]

---

kustomize create

.lab[

• Change to a new directory:

  mkdir ~/kustomcoins
  cd ~/kustomcoins
  

• Run kustomize create with the kustomcoins repository:

  kustomize create --resources https://github.com/jpetazzo/kubercoins
  

• Run kustomize build | kubectl apply -f-

]

---

kubectl integration

• Kustomize has been integrated in kubectl (since Kubernetes 1.14)

  • kubectl kustomize is an equivalent to kustomize build

  • commands that use -f can also use -k (kubectl apply/delete/...)

• The kustomize tool is still needed if we want to use create, edit, ...

• Kubernetes 1.14 to 1.20 uses Kustomize 2.0.3

• Kubernetes 1.21 jumps to Kustomize 4.1.2

• Future versions should track Kustomize updates more closely

---

class: extra-details

Differences between 2.0.3 and later

• Kustomize 2.1 / 3.0 deprecates bases (they should be listed in resources)

  (this means that "modern" kustomize edit add resource won't work with "old" kubectl apply -k)

• Kustomize 2.1 introduces replicas and envs

• Kustomize 3.1 introduces multipatches

• Kustomize 3.2 introduce inline patches in kustomization.yaml

• Kustomize 3.3 to 3.10 is mostly internal refactoring

• Kustomize 4.0 drops go-getter again

• Kustomize 4.1 allows patching kind and name

---

Adding labels

Labels can be added to all resources liks this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
commonLabels:
  app.kubernetes.io/name: dockercoins

Or with the equivalent CLI command:

kustomize edit add label app.kubernetes.io/name:dockercoins

---

Use cases for labels

• Example: clean up components that have been removed from the kustomization

• Assuming that commonLabels have been set as shown on the previous slide:

    kubectl apply -k . --prune --selector app.kubernetes.io/name=dockercoins
  

• ... This command removes resources that have been removed from the kustomization

• Technically, resources with:

  • a kubectl.kubernetes.io/last-applied-configuration annotation

  • labels matching the given selector

---

Scaling

Instead of using a patch, scaling can be done like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
replicas:
- name: worker
  count: 5

or the CLI equivalent:

kustomize edit set replicas worker=5

It will automatically work with Deployments, ReplicaSets, StatefulSets.

(For other resource types, fall back to a patch.)

---

Updating images

Instead of using patches, images can be changed like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
images:
- name: postgres
  newName: harbor.enix.io/my-postgres
- name: dockercoins/worker
  newTag: v0.2
- name: dockercoins/hasher
  newName: registry.dockercoins.io/hasher
  newTag: v0.2
- name: alpine
  digest: sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

---

Updating images with the CLI

To add an entry in the images: section of the kustomization:

kustomize edit set image name=[newName][:newTag][@digest]

• [] denote optional parameters

• : and @ are the delimiters used to indicate a field

Examples:

kustomize edit set image dockercoins/worker=ghcr.io/dockercoins/worker
kustomize edit set image dockercoins/worker=ghcr.io/dockercoins/worker:v0.2
kustomize edit set image dockercoins/worker=:v0.2

---

Updating images, pros and cons

• Very convenient when the same image appears multiple times

• Very convenient to define tags (or pin to hashes) outside of the main YAML

• Doesn't support wildcard or generic substitutions:

  • cannot "replace dockercoins/* with ghcr.io/dockercoins/*"

  • cannot "tag all dockercoins/* with v0.2"

• Only patches "well-known" image fields (won't work with CRDs referencing images)

• Helm can deal with these scenarios, for instance:

  image: {{ .Values.registry }}/worker:{{ .Values.version }}
  

---

Advanced resource patching

The example below shows how to:

• patch multiple resources with a selector (new in Kustomize 3.1)
• use an inline patch instead of a separate patch file (new in Kustomize 3.2)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
patches:
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: alpine
  target:
    kind: Deployment
    labelSelector: "app"

(This replaces all images of Deployments matching the app selector with alpine.)

---

Advanced resource patching, pros and cons

• Very convenient to patch an arbitrary number of resources

• Very convenient to patch any kind of resource, including CRDs

• Doesn't support "fine-grained" patching (e.g. image registry or tag)

• Once again, Helm can do it:

  image: {{ .Values.registry }}/worker:{{ .Values.version }}
  

---

Differences with Helm

• Helm charts generally require more upfront work

  (while kustomize "bases" are standard Kubernetes YAML)

• ... But Helm charts are also more powerful; their templating language can:

  • conditionally include/exclude resources or blocks within resources

  • generate values by concatenating, hashing, transforming parameters

  • generate values or resources by iteration ({{ range ... }})

  • access the Kubernetes API during template evaluation

  • and much more

???

:EN:- Packaging and running apps with Kustomize :FR:- Packaging d'applications avec Kustomize