github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-28 09:11:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

♻️ Update kubebuilder workshop

Browse Source
This commit is contained in:
Jérôme Petazzoni
2022-10-28 12:32:05 +02:00
parent 2943ef4e26
commit c86474a539
1 changed files with 243 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files

339
slides/k8s/kubebuilder.md
View File

@@ -86,8 +86,8 @@
(This is inspired by the
[uselessoperator](https://github.com/tilt-dev/uselessoperator)
written by
[L Körbes](https://twitter.com/ellenkorbes).
written by
[V Körbes](https://twitter.com/veekorbes).
Highly recommend!💯)
---
@@ -160,34 +160,31 @@ type MachineSpec struct {
We can use Go *marker comments* to give `controller-gen` extra details about how to handle our type, for instance:
```go
//+kubebuilder:object:root=true
```
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string
→ top-level type exposed through API (as opposed to "member field of another type")
```go
//+kubebuilder:subresource:status
```
→ automatically generate a `status` subresource (very common with many types)
```go
//+kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string
```
(See
[marker syntax](https://book.kubebuilder.io/reference/markers.html),
[CRD generation](https://book.kubebuilder.io/reference/markers/crd.html),
[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html)
[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html),
[Object/DeepCopy](https://master.book.kubebuilder.io/reference/markers/object.html)
)
---
class: extra-details
## Using CRD v1
- By default, kubebuilder generates v1alpha1 CRDs
- If we want to generate v1 CRDs:
- edit `Makefile`
- update `crd:crdVersions=v1`
---
## Installing the CRD
After making these changes, we can run `make install`.
@@ -208,6 +205,7 @@ Edit `config/samples/useless_v1alpha1_machine.yaml`:
kind: Machine
apiVersion: useless.container.training/v1alpha1
metadata:
labels: # ...
name: machine-1
spec:
# Our useless operator will change that to "down"
@@ -252,20 +250,23 @@ spec:
## Loading an object
Open `controllers/machine_controller.go` and add that code in the `Reconcile` method:
Open `controllers/machine_controller.go`.
Add that code in the `Reconcile` method, at the `TODO(user)` location:
```go
var machine uselessv1alpha1.Machine
logger := log.FromContext(ctx)
if err := r.Get(ctx, req.NamespacedName, &machine); err != nil {
log.Info("error getting object")
return ctrl.Result{}, err
logger.Info("error getting object")
return ctrl.Result{}, err
}
r.Log.Info(
"reconciling",
"machine", req.NamespaceName,
"switchPosition", machine.Spec.SwitchPosition,
logger.Info(
"reconciling",
"machine", req.NamespacedName,
"switchPosition", machine.Spec.SwitchPosition,
)
```
@@ -288,7 +289,7 @@ Then:
--
🤔
We get a bunch of errors and go stack traces! 🤔
---
@@ -324,7 +325,7 @@ Let's try to update the machine like this:
if machine.Spec.SwitchPosition != "down" {
machine.Spec.SwitchPosition = "down"
if err := r.Update(ctx, &machine); err != nil {
log.Info("error updating switch position")
logger.Info("error updating switch position")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
}
@@ -344,9 +345,9 @@ Again - update, `make run`, test.
(maybe with degraded behavior in the meantime)
- Status will almost always be a sub-resource
- Status will almost always be a sub-resource, so that it can be updated separately
(so that it can be updated separately "cheaply")
(and potentially with different permissions)
---
@@ -399,8 +400,8 @@ class: extra-details
## To requeue ...
`return ctrl.Result{RequeueAfter: 1 * time.Second}`
`return ctrl.Result{RequeueAfter: 1 * time.Second}, nil`
- That means: "try again in 1 second, and I will check if progress was made"
- This *does not* guarantee that we will be called exactly 1 second later:
@@ -409,7 +410,9 @@ class: extra-details
- we might be called after (if the controller is busy with other objects)
- If we are waiting for another resource to change, there is an even better way!
- If we are waiting for another Kubernetes resource to change, there is a better way
(explained on next slide)
---
@@ -417,23 +420,41 @@ class: extra-details
`return ctrl.Result{}, nil`
- That means: "no need to set an alarm; we'll be notified some other way"
- That means: "we're done here!"
- Use this if we are waiting for another resource to update
- This is also what we should use if we are waiting for another resource
(e.g. a LoadBalancer to be provisioned, a Pod to be ready...)
- For this to work, we need to set a *watch* (more on that later)
- In that case, we will need to set a *watch* (more on that later)
---
## Keeping track of state
- If we simply requeue the object to examine it 1 second later...
- ...We'll keep examining/requeuing it forever!
- We need to "remember" that we saw it (and when)
- Option 1: keep state in controller
(e.g. an internal `map`)
- Option 2: keep state in the object
(typically in its status field)
- Tradeoffs: concurrency / failover / control plane overhead...
---
## "Improving" our controller, take 2
- Let's store in the machine status the moment when we saw it
Let's store in the machine status the moment when we saw it:
```go
// +kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date
type MachineStatus struct {
// Time at which the machine was noticed by our controller.
SeenAt *metav1.Time ``json:"seenAt,omitempty"``
@@ -446,6 +467,12 @@ Note: `date` fields don't display timestamps in the future.
(That's why for this example it's simpler to use `seenAt` rather than `changeAt`.)
And for better visibility, add this along with the other `printcolumn` comments:
```go
//+kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date
```
---
## Set `seenAt`
@@ -457,7 +484,7 @@ if machine.Status.SeenAt == nil {
now := metav1.Now()
machine.Status.SeenAt = &now
if err := r.Status().Update(ctx, &machine); err != nil {
log.Info("error updating status.seenAt")
logger.Info("error updating status.seenAt")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
@@ -478,8 +505,9 @@ if machine.Spec.SwitchPosition != "down" {
changeAt := machine.Status.SeenAt.Time.Add(5 * time.Second)
if now.Time.After(changeAt) {
machine.Spec.SwitchPosition = "down"
machine.Status.SeenAt = nil
if err := r.Update(ctx, &machine); err != nil {
log.Info("error updating switch position")
logger.Info("error updating switch position")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
}
@@ -496,15 +524,33 @@ if machine.Spec.SwitchPosition != "down" {
- We will now have two kinds of objects: machines, and switches
- Machines will store the number of switches in their spec
- Machines should have *at least* one switch, possibly *multiple ones*
- The position will now be stored in the switch, not the machine
- Our controller will automatically create switches if needed
- The machine will also expose the combined state of the switches
(a bit like the ReplicaSet controller automatically creates Pods)
- The switches will be tied to their machine through a label
(See next slide for an example)
(let's pick `machine=name-of-the-machine`)
---
## Switch state
- The position of a switch will now be stored in the switch
(not in the machine like in the first scenario)
- The machine will also expose the combined state of the switches
(through its status)
- The machine's status will be automatically updated by the controller
(each time a switch is added/changed/removed)
---
@@ -516,7 +562,7 @@ NAME SWITCHES POSITIONS
machine-cz2vl 3 ddd
machine-vf4xk 1 d
[jp@hex ~]$ kubectl get switches --show-labels
[jp@hex ~]$ kubectl get switches --show-labels
NAME POSITION SEEN LABELS
switch-6wmjw down machine=machine-cz2vl
switch-b8csg down machine=machine-cz2vl
@@ -530,39 +576,95 @@ switch-rc59l down machine=machine-vf4xk
## Tasks
Create the new resource type (but don't create a controller):
1. Create the new resource type (but don't create a controller)
2. Update `machine_types.go` and `switch_types.go`
3. Implement logic to display machine status (status of its switches)
4. Implement logic to automatically create switches
5. Implement logic to flip all switches down immediately
6. Then tweak it so that a given machine doesn't flip more than one switch every 5 seconds
*See next slides for detailed steps!*
---
## Creating the new type
```bash
kubebuilder create api --group useless --version v1alpha1 --kind Switch
```
Update `machine_types.go` and `switch_types.go`.
Implement the logic so that the controller flips all switches down immediately.
Then change it so that a given machine doesn't flip more than one switch every 5 seconds.
See next slides for hints!
Note: this time, only create a new custom resource; not a new controller.
---
## Listing objects
## Updating our types
We can use the `List` method with filters:
- Move the "switch position" and "seen at" to the new `Switch` type
```go
var switches uselessv1alpha1.SwitchList
- Update the `Machine` type to have:
if err := r.List(ctx, &switches,
client.InNamespace(req.Namespace),
client.MatchingLabels{"machine": req.Name},
); err != nil {
log.Error(err, "unable to list switches of the machine")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
- `spec.switches` (Go type: `int`, JSON type: `integer`)
log.Info("Found switches", "switches", switches)
```
- `status.positions` of type `string`
- Bonus points for adding [CRD Validation](https://book.kubebuilder.io/reference/markers/crd-validation.html) to the numbers of switches!
- Then install the new CRDs with `make install`
- Create a Machine, and a Switch linked to the Machine (by setting the `machine` label)
---
## Listing switches
- Switches are associated to Machines with a label
(`kubectl label switch switch-xyz machine=machine-xyz`)
- We can retrieve associated switches like this:
```go
var switches uselessv1alpha1.SwitchList
if err := r.List(ctx, &switches,
client.InNamespace(req.Namespace),
client.MatchingLabels{"machine": req.Name},
); err != nil {
logger.Error(err, "unable to list switches of the machine")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
logger.Info("Found switches", "switches", switches)
```
---
## Updating status
- Each time we reconcile a Machine, let's update its status:
```go
status := ""
for _, sw := range switches.Items {
status += string(sw.Spec.Position[0])
}
machine.Status.Positions = status
if err := r.Status().Update(ctx, &machine); err != nil {
...
```
- Run the controller and check that POSITIONS gets updated
- Add more switches linked to the same machine
- ...The POSITIONS don't get updated, unless we restart the controller
- We'll see later how to fix that!
---
@@ -590,20 +692,28 @@ if err := r.Create(ctx, &sw); err != nil { ...
---
## Create missing switches
- In our reconciler, if a machine doesn't have enough switches, create them!
- Option 1: directly create the number of missing switches
- Option 2: create only one switch (and rely on later requeuing)
- Note: option 2 won't quite work yet, since we haven't set up *watches* yet
---
## Watches
- Our controller will correctly flip switches when it starts
- It will also react to machine updates
- But it won't react if we directly touch the switches!
- By default, it only monitors machines, not switches
- Our controller doesn't react when switches are created/updated/deleted
- We need to tell it to watch switches
- We also need to tell it how to map a switch to its machine
(so that the correct machine gets queued and reconciled when a switch is updated)
---
## Mapping a switch to its machine
@@ -611,16 +721,15 @@ if err := r.Create(ctx, &sw); err != nil { ...
Define the following helper function:
```go
func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Request {
r.Log.Debug("mos", "obj", obj)
return []ctrl.Request{
ctrl.Request{
NamespacedName: types.NamespacedName{
Name: obj.Meta.GetLabels()["machine"],
Namespace: obj.Meta.GetNamespace(),
},
},
}
func (r *MachineReconciler) machineOfSwitch(obj client.Object) []ctrl.Request {
return []ctrl.Request{
ctrl.Request{
NamespacedName: types.NamespacedName{
Name: obj.GetLabels()["machine"],
Namespace: obj.GetNamespace(),
},
},
}
}
```
@@ -631,24 +740,46 @@ func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Reques
Update the `SetupWithManager` method in the controller:
```go
// SetupWithManager sets up the controller with the Manager.
func (r *MachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&uselessv1alpha1.Machine{}).
Owns(&uselessv1alpha1.Switch{}).
Watches(
&source.Kind{Type: &uselessv1alpha1.Switch{}},
&handler.EnqueueRequestsFromMapFunc{
ToRequests: handler.ToRequestsFunc(r.machineOfSwitch),
}).
Complete(r)
return ctrl.NewControllerManagedBy(mgr).
For(&uselessv1alpha1.Machine{}).
Owns(&uselessv1alpha1.Switch{}).
Watches(
&source.Kind{Type: &uselessv1alpha1.Switch{}},
handler.EnqueueRequestsFromMapFunc(r.machineOfSwitch),
).
Complete(r)
}
```
After this, our controller should now react to switch changes.
---
## ...And a few extra imports
Import the following packages referenced by the previous code:
```go
"sigs.k8s.io/controller-runtime/pkg/handler"
"sigs.k8s.io/controller-runtime/pkg/source"
"k8s.io/apimachinery/pkg/types"
```
After this, when we update a switch, it should reflect on the machine.
(Try to change switch positions and see the machine status update!)
---
## Bonus points
## Flipping switches
- Now re-add logic to flip switches that are not in "down" position
- Re-add logic to wait a few seconds before flipping a switch
- Change the logic to toggle one switch per machine every few seconds
(i.e. don't change all the switches for a machine; move them one at a time)
- Handle "scale down" of a machine (by deleting extraneous switches)
@@ -660,9 +791,25 @@ After this, our controller should now react to switch changes.
---
## Other possible improvements
- Formalize resource ownership
(by setting `ownerReferences` in the switches)
- This can simplify the watch mechanism a bit
- Allow to define a selector
(instead of using the hard-coded `machine` label)
- And much more!
---
## Acknowledgements
- Useless Operator, by [L Körbes](https://twitter.com/ellenkorbes)
- Useless Operator, by [V Körbes](https://twitter.com/veekorbes)
[code](https://github.com/tilt-dev/uselessoperator)
|