github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-08-18 11:36:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
livecycle
container.training/slides/k8s/kubebuilder.md
T

687 lines
15 KiB
Markdown
Raw Permalink Blame History

# Kubebuilder
- Writing a quick and dirty operator is (relatively) easy
- Doing it right, however ...
--
- We need:
- proper CRD with schema validation
- controller performing a reconcilation loop
- manage errors, retries, dependencies between resources
- maybe webhooks for admission and/or conversion
😱
---
## Frameworks
- There are a few frameworks available out there:
- [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder)
([book](https://book.kubebuilder.io/)):
go-centric, very close to Kubernetes' core types
- [operator-framework](https://operatorframework.io/):
higher level; also supports Ansible and Helm
- [KUDO](https://kudo.dev/):
declarative operators written in YAML
- [KOPF](https://kopf.readthedocs.io/en/latest/):
operators in Python
- ...
---
## Kubebuilder workflow
- Kubebuilder will create scaffolding for us
(Go stubs for types and controllers)
- Then we edit these types and controllers files
- Kubebuilder generates CRD manifests from our type definitions
(and regenerates the manifests whenver we update the types)
- It also gives us tools to quickly run the controller against a cluster
(not necessarily *on* the cluster)
---
## Our objective
- We're going to implement a *useless machine*
[basic example](https://www.youtube.com/watch?v=aqAUmgE3WyM)
|
[playful example](https://www.youtube.com/watch?v=kproPsch7i0)
|
[advanced example](https://www.youtube.com/watch?v=Nqk_nWAjBus)
|
[another advanced example](https://www.youtube.com/watch?v=eLtUB8ncEnA)
- A machine manifest will look like this:
```yaml
kind: Machine
apiVersion: useless.container.training/v1alpha1
metadata:
name: machine-1
spec:
# Our useless operator will change that to "down"
switchPosition: up
```
- Each time we change the `switchPosition`, the operator will move it back to `down`
(This is inspired by the
[uselessoperator](https://github.com/tilt-dev/uselessoperator)
written by
[L Körbes](https://twitter.com/ellenkorbes).
Highly recommend!💯)
---
class: extra-details
## Local vs remote
- Building Go code can be a little bit slow on our modest lab VMs
- It will typically be *much* faster on a local machine
- All the demos and labs in this section will run fine either way!
---
## Preparation
- Install Go
(on our VMs: `sudo snap install go --classic`)
- Install kubebuilder
([get a release](https://github.com/kubernetes-sigs/kubebuilder/releases/), untar, move the `kubebuilder` binary to the `$PATH`)
- Initialize our workspace:
```bash
mkdir useless
cd useless
go mod init container.training/useless
kubebuilder init --domain container.training
```
---
## Create scaffolding
- Create a type and corresponding controller:
```bash
kubebuilder create api --group useless --version v1alpha1 --kind Machine
```
- Answer `y` to both questions
- Then we need to edit the type that just got created!
---
## Edit type
Edit `api/v1alpha1/machine_types.go`.
Add the `switchPosition` field in the `spec` structure:
```go
// MachineSpec defines the desired state of Machine
type MachineSpec struct {
// Position of the switch on the machine, for instance up or down.
SwitchPosition string ``json:"switchPosition,omitempty"``
}
```
⚠️ The backticks above should be simple backticks, not double-backticks. Sorry.
---
## Go markers
We can use Go *marker comments* to give `controller-gen` extra details about how to handle our type, for instance:
```
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string
```
(See
[marker syntax](https://book.kubebuilder.io/reference/markers.html),
[CRD generation](https://book.kubebuilder.io/reference/markers/crd.html),
[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html)
)
---
class: extra-details
## Using CRD v1
- By default, kubebuilder generates v1alpha1 CRDs
- If we want to generate v1 CRDs:
- edit `Makefile`
- update `crd:crdVersions=v1`
---
## Installing the CRD
After making these changes, we can run `make install`.
This will build the Go code, but also:
- generate the CRD manifest
- and apply the manifest to the cluster
---
## Creating a machine
Edit `config/samples/useless_v1alpha1_machine.yaml`:
```yaml
kind: Machine
apiVersion: useless.container.training/v1alpha1
metadata:
name: machine-1
spec:
# Our useless operator will change that to "down"
switchPosition: up
```
... and apply it to the cluster.
---
## Designing the controller
- Our controller needs to:
- notice when a `switchPosition` is not `down`
- move it to `down` when that happens
- Later, we can add fancy improvements (wait a bit before moving it, etc.)
---
## Reconciler logic
- Kubebuilder will call our *reconciler* when necessary
- When necessary = when changes happen ...
- on our resource
- or resources that it *watches* (related resources)
- After "doing stuff", the reconciler can return ...
- `ctrl.Result{},nil` = all is good
- `ctrl.Result{Requeue...},nil` = all is good, but call us back in a bit
- `ctrl.Result{},err` = something's wrong, try again later
---
## Loading an object
Open `controllers/machine_controller.go` and add that code in the `Reconcile` method:
```go
var machine uselessv1alpha1.Machine
if err := r.Get(ctx, req.NamespacedName, &machine); err != nil {
log.Info("error getting object")
return ctrl.Result{}, err
}
r.Log.Info(
"reconciling",
"machine", req.NamespaceName,
"switchPosition", machine.Spec.SwitchPosition,
)
```
---
## Running the controller
Our controller is not done yet, but let's try what we have right now!
This will compile the controller and run it:
```
make run
```
Then:
- create a machine
- change the `switchPosition`
- delete the machine
--
🤔
---
## `IgnoreNotFound`
When we are called for object deletion, the object has *already* been deleted.
(Unless we're using finalizers, but that's another story.)
When we return `err`, the controller will try to access the object ...
... We need to tell it to *not* do that.
Don't just return `err`, but instead, wrap it around `client.IgnoreNotFound`:
```go
return ctrl.Result{}, client.IgnoreNotFound(err)
```
Update the code, `make run` again, create/change/delete again.
--
🎉
---
## Updating the machine
Let's try to update the machine like this:
```go
if machine.Spec.SwitchPosition != "down" {
machine.Spec.SwitchPosition = "down"
if err := r.Update(ctx, &machine); err != nil {
log.Info("error updating switch position")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
}
```
Again - update, `make run`, test.
---
## Spec vs Status
- Spec = desired state
- Status = observed state
- If Status is lost, the controller should be able to reconstruct it
(maybe with degraded behavior in the meantime)
- Status will almost always be a sub-resource
(so that it can be updated separately "cheaply")
---
class: extra-details
## Spec vs Status (in depth)
- The `/status` subresource is handled differently by the API server
- Updates to `/status` don't alter the rest of the object
- Conversely, updates to the object ignore changes in the status
(See [the docs](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource) for the fine print.)
---
## "Improving" our controller
- We want to wait a few seconds before flipping the switch
- Let's add the following line of code to the controller:
```go
time.Sleep(5 * time.Second)
```
- `make run`, create a few machines, observe what happens
--
💡 Concurrency!
---
## Controller logic
- Our controller shouldn't block (think "event loop")
- There is a queue of objects that need to be reconciled
- We can ask to be put back on the queue for later processing
- When we need to block (wait for something to happen), two options:
- ask for a *requeue* ("call me back later")
- yield because we know we will be notified by another resource
---
## To requeue ...
`return ctrl.Result{RequeueAfter: 1 * time.Second}`
- That means: "try again in 1 second, and I will check if progress was made"
- This *does not* guarantee that we will be called exactly 1 second later:
- we might be called before (if other changes happen)
- we might be called after (if the controller is busy with other objects)
- If we are waiting for another resource to change, there is an even better way!
---
## ... or not to requeue
`return ctrl.Result{}, nil`
- That means: "no need to set an alarm; we'll be notified some other way"
- Use this if we are waiting for another resource to update
(e.g. a LoadBalancer to be provisioned, a Pod to be ready...)
- For this to work, we need to set a *watch* (more on that later)
---
## "Improving" our controller, take 2
- Let's store in the machine status the moment when we saw it
```go
// +kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date
type MachineStatus struct {
// Time at which the machine was noticed by our controller.
SeenAt *metav1.Time ``json:"seenAt,omitempty"``
}
```
⚠️ The backticks above should be simple backticks, not double-backticks. Sorry.
Note: `date` fields don't display timestamps in the future.
(That's why for this example it's simpler to use `seenAt` rather than `changeAt`.)
---
## Set `seenAt`
Let's add the following block in our reconciler:
```go
if machine.Status.SeenAt == nil {
now := metav1.Now()
machine.Status.SeenAt = &now
if err := r.Status().Update(ctx, &machine); err != nil {
log.Info("error updating status.seenAt")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
}
```
(If needed, add `metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"` to our imports.)
---
## Use `seenAt`
Our switch-position-changing code can now become:
```go
if machine.Spec.SwitchPosition != "down" {
now := metav1.Now()
changeAt := machine.Status.SeenAt.Time.Add(5 * time.Second)
if now.Time.After(changeAt) {
machine.Spec.SwitchPosition = "down"
if err := r.Update(ctx, &machine); err != nil {
log.Info("error updating switch position")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
}
}
```
`make run`, create a few machines, tweak their switches.
---
## Owner and dependents
- Next, let's see how to have relationships between objects!
- We will now have two kinds of objects: machines, and switches
- Machines should have *at least* one switch, possibly *multiple ones*
- The position will now be stored in the switch, not the machine
- The machine will also expose the combined state of the switches
- The switches will be tied to their machine through a label
(See next slide for an example)
---
## Switches and machines
```
[jp@hex ~]$ kubectl get machines
NAME SWITCHES POSITIONS
machine-cz2vl 3 ddd
machine-vf4xk 1 d
[jp@hex ~]$ kubectl get switches --show-labels
NAME POSITION SEEN LABELS
switch-6wmjw down machine=machine-cz2vl
switch-b8csg down machine=machine-cz2vl
switch-fl8dq down machine=machine-cz2vl
switch-rc59l down machine=machine-vf4xk
```
(The field `status.positions` shows the first letter of the `position` of each switch.)
---
## Tasks
Create the new resource type (but don't create a controller):
```bash
kubebuilder create api --group useless --version v1alpha1 --kind Switch
```
Update `machine_types.go` and `switch_types.go`.
Implement the logic so that the controller flips all switches down immediately.
Then change it so that a given machine doesn't flip more than one switch every 5 seconds.
See next slides for hints!
---
## Listing objects
We can use the `List` method with filters:
```go
var switches uselessv1alpha1.SwitchList
if err := r.List(ctx, &switches,
client.InNamespace(req.Namespace),
client.MatchingLabels{"machine": req.Name},
); err != nil {
log.Error(err, "unable to list switches of the machine")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
log.Info("Found switches", "switches", switches)
```
---
## Creating objects
We can use the `Create` method to create a new object:
```go
sw := uselessv1alpha1.Switch{
TypeMeta: metav1.TypeMeta{
APIVersion: uselessv1alpha1.GroupVersion.String(),
Kind: "Switch",
},
ObjectMeta: metav1.ObjectMeta{
GenerateName: "switch-",
Namespace: machine.Namespace,
Labels: map[string]string{"machine": machine.Name},
},
Spec: uselessv1alpha1.SwitchSpec{
Position: "down",
},
}
if err := r.Create(ctx, &sw); err != nil { ...
```
---
## Watches
- Our controller will correctly flip switches when it starts
- It will also react to machine updates
- But it won't react if we directly touch the switches!
- By default, it only monitors machines, not switches
- We need to tell it to watch switches
- We also need to tell it how to map a switch to its machine
---
## Mapping a switch to its machine
Define the following helper function:
```go
func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Request {
r.Log.Debug("mos", "obj", obj)
return []ctrl.Request{
ctrl.Request{
NamespacedName: types.NamespacedName{
Name: obj.Meta.GetLabels()["machine"],
Namespace: obj.Meta.GetNamespace(),
},
},
}
}
```
---
## Telling the controller to watch switches
Update the `SetupWithManager` method in the controller:
```go
func (r *MachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&uselessv1alpha1.Machine{}).
Owns(&uselessv1alpha1.Switch{}).
Watches(
&source.Kind{Type: &uselessv1alpha1.Switch{}},
&handler.EnqueueRequestsFromMapFunc{
ToRequests: handler.ToRequestsFunc(r.machineOfSwitch),
}).
Complete(r)
}
```
After this, our controller should now react to switch changes.
---
## Bonus points
- Handle "scale down" of a machine (by deleting extraneous switches)
- Automatically delete switches when a machine is deleted
(ideally, using ownership information)
- Test corner cases (e.g. changing a switch label)
---
## Acknowledgements
- Useless Operator, by [L Körbes](https://twitter.com/ellenkorbes)
[code](https://github.com/tilt-dev/uselessoperator)
|
[video (EN)](https://www.youtube.com/watch?v=85dKpsFFju4)
|
[video (PT)](https://www.youtube.com/watch?v=Vt7Eg4wWNDw)
- Zero To Operator, by [Solly Ross](https://twitter.com/directxman12)
[code](https://pres.metamagical.dev/kubecon-us-2019/code)
|
[video](https://www.youtube.com/watch?v=KBTXBUVNF2I)
|
[slides](https://pres.metamagical.dev/kubecon-us-2019/)
- The [kubebuilder book](https://book.kubebuilder.io/)
???
:EN:- Implementing an operator with kubebuilder
:FR:- Implémenter un opérateur avec kubebuilder
Reference in New Issue View Git Blame Copy Permalink