From f9660ba9dc438d74e29bcefdba93f19a98d9c974 Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Fri, 13 Nov 2020 18:46:16 +0100 Subject: [PATCH] Add kubebuilder tutorial --- slides/k8s/kubebuilder.md | 653 ++++++++++++++++++++++++++++++++++++++ slides/kube-selfpaced.yml | 1 + 2 files changed, 654 insertions(+) create mode 100644 slides/k8s/kubebuilder.md diff --git a/slides/k8s/kubebuilder.md b/slides/k8s/kubebuilder.md new file mode 100644 index 00000000..8f336332 --- /dev/null +++ b/slides/k8s/kubebuilder.md @@ -0,0 +1,653 @@ +# Kubebuilder + +- Writing a quick and dirty operator is (relatively) easy + +- Doing it right, however ... + +-- + +- We need: + + - proper CRD with schema validation + + - controller performing a reconcilation loop + + - manage errors, retries, dependencies between resources + + - maybe webhooks for admission and/or conversion + +😱 + +--- + +## Frameworks + +- There are a few frameworks available out there: + + - [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder) + ([book](https://book.kubebuilder.io/)): + go-centric, very close to Kubernetes' core types + + - [operator-framework](https://operatorframework.io/): + higher level; also supports Ansible and Helm + + - [KUDO](https://kudo.dev/): + declarative operators written in YAML + + - [KOPF](https://kopf.readthedocs.io/en/latest/): + operators in Python + + - ... + +--- + +## Kubebuilder workflow + +- Kubebuilder will create scaffolding for us + + (Go stubs for types and controllers) + +- Then we edit these types and controllers files + +- Kubebuilder generates CRD manifests from our type definitions + + (and regenerates the manifests whenver we update the types) + +- It also gives us tools to quickly run the controller against a cluster + + (not necessarily *on* the cluster) + +--- + +## Our objective + +- We're going to implement a *useless machine* + + [basic example](https://www.youtube.com/watch?v=aqAUmgE3WyM) + | + [playful example](https://www.youtube.com/watch?v=kproPsch7i0) + | + [advanced example](https://www.youtube.com/watch?v=Nqk_nWAjBus) + | + [another advanced example](https://www.youtube.com/watch?v=eLtUB8ncEnA) + +- A machine manifest will look like this: + ```yaml + kind: Machine + apiVersion: useless.container.training/v1alpha1 + metadata: + name: machine-1 + spec: + # Our useless operator will change that to "down" + switchPosition: up + ``` + +- Each time we change the `switchPosition`, the operator will move it back to `down` + +(This is inspired by the +[uselessoperator](https://github.com/tilt-dev/uselessoperator) +written by +[L Körbes](https://twitter.com/ellenkorbes). +Highly recommend!💯) + +--- + +## Preparation + +- Install kubebuilder + +- Initialize our workspace: + ```bash + mkdir useless + cd useless + go mod init container.training/useless + kubebuilder init --domain container.training + ``` + +--- + +## Create scaffolding + +- Create a type and corresponding controller: + ```bash + kubebuilder create api --group useless --version v1alpha1 --kind Machine + ``` + +- Answer `y` to both questions + +- Then we need to edit the type that just got created! + +--- + +## Edit type + +Edit `api/v1alpha1/machine_types.go`. + +Add the `switchPosition` field in the `spec` structure: + +```go +// MachineSpec defines the desired state of Machine +type MachineSpec struct { + // Position of the switch on the machine, for instance up or down. + SwitchPosition string ``json:"switchPosition,omitempty"`` +} +``` + +⚠️ The backticks above should be simple backticks, not double-backticks. Sorry. + +--- + +## Go markers + +We can use Go *marker comments* to give `controller-gen` extra details about how to handle our type, for instance: + +``` +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string +``` + +(See +[marker syntax](https://book.kubebuilder.io/reference/markers.html), +[CRD generation](https://book.kubebuilder.io/reference/markers/crd.html), +[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html) +) + +--- + +class: extra-details + +## Using CRD v1 + +- By default, kubebuilder generates v1alpha1 CRDs + +- If we want to generate v1 CRDs: + + - edit `Makefile` + + - update `crd:crdVersions=v1` + +--- + +## Installing the CRD + +After making these changes, we can run `make install`. + +This will build the Go code, but also: + +- generate the CRD manifest + +- and apply the manifest to the cluster + +--- + +## Creating a machine + +Edit `config/samples/useless_v1alpha1_machine.yaml`: + +```yaml +kind: Machine +apiVersion: useless.container.training/v1alpha1 +metadata: + name: machine-1 +spec: + # Our useless operator will change that to "down" + switchPosition: up +``` + +... and apply it to the cluster. + +--- + +## Designing the controller + +- Our controller needs to: + + - notice when a `switchPosition` is not `down` + + - move it to `down` when that happens + +- Later, we can add fancy improvements (wait a bit before moving it, etc.) + +--- + +## Reconciler logic + +- Kubebuilder will call our *reconciler* when necessary + +- When necessary = when changes happen ... + + - on our resource + + - or resources that it *watches* (related resources) + +- After "doing stuff", the reconciler can return ... + + - `ctrl.Result{},nil` = all is good + + - `ctrl.Result{Requeue...},nil` = all is good, but call us back in a bit + + - `ctrl.Result{},err` = something's wrong, try again later + +--- + +## Loading an object + +Open `controllers/machine_controller.go` and add that code in the `Reconcile` method: + +```go +var machine uselessv1alpha1.Machine + +if err := r.Get(ctx, req.NamespacedName, &machine); err != nil { + log.Info("error getting object") + return ctrl.Result{}, err +} + +r.Log.Info( + "reconciling", + "machine", req.NamespaceName, + "switchPosition", machine.Spec.SwitchPosition, +) +``` + +--- + +## Running the controller + +Our controller is not done yet, but let's try what we have right now! + +This will compile the controller and run it: +``` +make run +``` + +Then: + +- create a machine +- change the `switchPosition` +- delete the machine + +-- + +🤔 + +--- + +## `IgnoreNotFound` + +When we are called for object deletion, the object has *already* been deleted. + +(Unless we're using finalizers, but that's another story.) + +When we return `err`, the controller will try to access the object ... + +... We need to tell it to *not* do that. + +Don't just return `err`, but instead, wrap it around `client.IgnoreNotFound`: + +```go +return ctrl.Result{}, client.IgnoreNotFound(err) +``` + +Update the code, `make run` again, create/change/delete again. + +-- + +🎉 + +--- + +## Updating the machine + +Let's try to update the machine like this: + +```go +if machine.Spec.SwitchPosition != "down" { + machine.Spec.SwitchPosition = "down" + if err := r.Update(ctx, &machine); err != nil { + log.Info("error updating switch position") + return ctrl.Result{}, client.IgnoreNotFound(err) + } +} +``` + +Again - update, `make run`, test. + +--- + +## Spec vs Status + +- Spec = desired state + +- Status = observed state + +- If Status is lost, the controller should be able to reconstruct it + + (maybe with degraded behavior in the meantime) + +- Status will almost always be a sub-resource + + (so that it can be updated separately "cheaply") + +--- + +## "Improving" our controller + +- We want to wait a few seconds before flipping the switch + +- Let's add the following line of code to the controller: + ```go + time.Sleep(5 * time.Second) + ``` + +- `make run`, create a few machines, observe what happens + +-- + +💡 Concurrency! + +--- + +## Controller logic + +- Our controller shouldn't block (think "event loop") + +- There is a queue of objects that need to be reconciled + +- We can ask to be put back on the queue for later processing + +- When we need to block (wait for something to happen), two options: + + - ask for a *requeue* ("call me back later") + + - yield because we know we will be notified by another resource + +--- + +## To requeue ... + +`return ctrl.Result{RequeueAfter: 1 * time.Second}` + +- That means: "try again in 1 second, and I will check if progress was made" + +- This *does not* guarantee that we will be called exactly 1 second later: + + - we might be called before (if other changes happen) + + - we might be called after (if the controller is busy with other objects) + +- If we are waiting for another resource to change, there is an even better way! + +--- + +## ... or not to requeue + +`return ctrl.Result{}, nil` + +- That means: "no need to set an alarm; we'll be notified some other way" + +- Use this if we are waiting for another resource to update + + (e.g. a LoadBalancer to be provisioned, a Pod to be ready...) + +- For this to work, we need to set a *watch* (more on that later) + +--- + +## "Improving" our controller, take 2 + +- Let's store in the machine status the moment when we saw it + +```go +// +kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date + +type MachineStatus struct { + // Time at which the machine was noticed by our controller. + SeenAt *metav1.Time ``json:"seenAt,omitempty"`` +} +``` + +⚠️ The backticks above should be simple backticks, not double-backticks. Sorry. + +Note: `date` fields don't display timestamps in the future. + +(That's why for this example it's simpler to use `seenAt` rather than `changeAt`.) + +--- + +## Set `seenAt` + +Let's add the following block in our reconciler: + +```go +if machine.Status.SeenAt == nil { + now := metav1.Now() + machine.Status.SeenAt = &now + if err := r.Status().Update(ctx, &machine); err != nil { + log.Info("error updating status.seenAt") + return ctrl.Result{}, client.IgnoreNotFound(err) + } + return ctrl.Result{RequeueAfter: 5 * time.Second}, nil +} +``` + +(If needed, add `metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"` to our imports.) + +--- + +## Use `seenAt` + +Our switch-position-changing code can now become: + +```go +if machine.Spec.SwitchPosition != "down" { + now := metav1.Now() + changeAt := machine.Status.SeenAt.Time.Add(5 * time.Second) + if now.Time.After(changeAt) { + machine.Spec.SwitchPosition = "down" + if err := r.Update(ctx, &machine); err != nil { + log.Info("error updating switch position") + return ctrl.Result{}, client.IgnoreNotFound(err) + } + } +} +``` + +`make run`, create a few machines, tweak their switches. + +--- + +## Owner and dependents + +- Next, let's see how to have relationships between objects! + +- We will now have two kinds of objects: machines, and switches + +- Machines should have *at least* one switch, possibly *multiple ones* + +- The position will now be stored in the switch, not the machine + +- The machine will also expose the combined state of the switches + +- The switches will be tied to their machine through a label + +(See next slide for an example) + +--- + +## Switches and machines + +``` +[jp@hex ~]$ kubectl get machines +NAME SWITCHES POSITIONS +machine-cz2vl 3 ddd +machine-vf4xk 1 d + +[jp@hex ~]$ kubectl get switches --show-labels +NAME POSITION SEEN LABELS +switch-6wmjw down machine=machine-cz2vl +switch-b8csg down machine=machine-cz2vl +switch-fl8dq down machine=machine-cz2vl +switch-rc59l down machine=machine-vf4xk +``` + +(The field `status.positions` shows the first letter of the `position` of each switch.) + +--- + +## Tasks + +Create the new resource type (but don't create a controller): + +```bash +kubebuilder create api --group useless --version v1alpha1 --kind Switch +``` + +Update `machine_types.go` and `switch_types.go`. + +Implement the logic so that the controller flips all switches down immediately. + +Then change it so that a given machine doesn't flip more than one switch every 5 seconds. + +See next slides for hints! + +--- + +## Listing objects + +We can use the `List` method with filters: + +```go +var switches uselessv1alpha1.SwitchList + +if err := r.List(ctx, &switches, + client.InNamespace(req.Namespace), + client.MatchingLabels{"machine": req.Name}, + ); err != nil { + log.Error(err, "unable to list switches of the machine") + return ctrl.Result{}, client.IgnoreNotFound(err) +} + +log.Info("Found switches", "switches", switches) +``` + +--- + +## Creating objects + +We can use the `Create` method to create a new object: + +```go +sw := uselessv1alpha1.Switch{ + TypeMeta: metav1.TypeMeta{ + APIVersion: uselessv1alpha1.GroupVersion.String(), + Kind: "Switch", + }, + ObjectMeta: metav1.ObjectMeta{ + GenerateName: "switch-", + Namespace: machine.Namespace, + Labels: map[string]string{"machine": machine.Name}, + }, + Spec: uselessv1alpha1.SwitchSpec{ + Position: "down", + }, +} +if err := r.Create(ctx, &sw); err != nil { ... +``` + +--- + +## Watches + +- Our controller will correctly flip switches when it starts + +- It will also react to machine updates + +- But it won't react if we directly touch the switches! + +- By default, it only monitors machines, not switches + +- We need to tell it to watch switches + +- We also need to tell it how to map a switch to its machine + +--- + +## Mapping a switch to its machine + +Define the following helper function: + +```go +func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Request { + r.Log.Debug("mos", "obj", obj) + return []ctrl.Request{ + ctrl.Request{ + NamespacedName: types.NamespacedName{ + Name: obj.Meta.GetLabels()["machine"], + Namespace: obj.Meta.GetNamespace(), + }, + }, + } +} +``` + +--- + +## Telling the controller to watch switches + +Update the `SetupWithManager` method in the controller: + +```go +func (r *MachineReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&uselessv1alpha1.Machine{}). + Owns(&uselessv1alpha1.Switch{}). + Watches( + &source.Kind{Type: &uselessv1alpha1.Switch{}}, + &handler.EnqueueRequestsFromMapFunc{ + ToRequests: handler.ToRequestsFunc(r.machineOfSwitch), + }). + Complete(r) +} +``` + +After this, our controller should now react to switch changes. + +--- + +## Bonus points + +- Handle "scale down" of a machine (by deleting extraneous switches) + +- Automatically delete switches when a machine is deleted + + (ideally, using ownership information) + +- Test corner cases (e.g. changing a switch label) + +--- + +## Acknowledgements + +- Useless Operator, by [L Körbes](https://twitter.com/ellenkorbes) + + [code](https://github.com/tilt-dev/uselessoperator) + | + [video (EN)](https://www.youtube.com/watch?v=85dKpsFFju4) + | + [video (PT)](https://www.youtube.com/watch?v=Vt7Eg4wWNDw) + +- Zero To Operator, by [Solly Ross](https://twitter.com/directxman12) + + [code](https://pres.metamagical.dev/kubecon-us-2019/code) + | + [video](https://www.youtube.com/watch?v=KBTXBUVNF2I) + | + [slides](https://pres.metamagical.dev/kubecon-us-2019/) + +- The [kubebuilder book](https://book.kubebuilder.io/) + +??? + +:EN:- Implementing an operator with kubebuilder +:FR:- Implémenter un opérateur avec kubebuilder diff --git a/slides/kube-selfpaced.yml b/slides/kube-selfpaced.yml index 18898be3..8d29d5f3 100644 --- a/slides/kube-selfpaced.yml +++ b/slides/kube-selfpaced.yml @@ -118,6 +118,7 @@ content: - k8s/admission.md - k8s/operators.md - k8s/operators-design.md + - k8s/kubebuilder.md - k8s/kyverno.md - k8s/eck.md - k8s/finalizers.md