diff --git a/slides/k8s/kubebuilder.md b/slides/k8s/kubebuilder.md index 531aa86d..a92c6e3b 100644 --- a/slides/k8s/kubebuilder.md +++ b/slides/k8s/kubebuilder.md @@ -86,8 +86,8 @@ (This is inspired by the [uselessoperator](https://github.com/tilt-dev/uselessoperator) -written by -[L Körbes](https://twitter.com/ellenkorbes). +written by +[V Körbes](https://twitter.com/veekorbes). Highly recommend!💯) --- @@ -160,34 +160,31 @@ type MachineSpec struct { We can use Go *marker comments* to give `controller-gen` extra details about how to handle our type, for instance: +```go +//+kubebuilder:object:root=true ``` -// +kubebuilder:object:root=true -// +kubebuilder:subresource:status -// +kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string + +→ top-level type exposed through API (as opposed to "member field of another type") + +```go +//+kubebuilder:subresource:status +``` + +→ automatically generate a `status` subresource (very common with many types) + +```go +//+kubebuilder:printcolumn:JSONPath=".spec.switchPosition",name=Position,type=string ``` (See [marker syntax](https://book.kubebuilder.io/reference/markers.html), [CRD generation](https://book.kubebuilder.io/reference/markers/crd.html), -[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html) +[CRD validation](https://book.kubebuilder.io/reference/markers/crd-validation.html), +[Object/DeepCopy](https://master.book.kubebuilder.io/reference/markers/object.html) ) --- -class: extra-details - -## Using CRD v1 - -- By default, kubebuilder generates v1alpha1 CRDs - -- If we want to generate v1 CRDs: - - - edit `Makefile` - - - update `crd:crdVersions=v1` - ---- - ## Installing the CRD After making these changes, we can run `make install`. @@ -208,6 +205,7 @@ Edit `config/samples/useless_v1alpha1_machine.yaml`: kind: Machine apiVersion: useless.container.training/v1alpha1 metadata: + labels: # ... name: machine-1 spec: # Our useless operator will change that to "down" @@ -252,20 +250,23 @@ spec: ## Loading an object -Open `controllers/machine_controller.go` and add that code in the `Reconcile` method: +Open `controllers/machine_controller.go`. + +Add that code in the `Reconcile` method, at the `TODO(user)` location: ```go var machine uselessv1alpha1.Machine +logger := log.FromContext(ctx) if err := r.Get(ctx, req.NamespacedName, &machine); err != nil { - log.Info("error getting object") - return ctrl.Result{}, err + logger.Info("error getting object") + return ctrl.Result{}, err } -r.Log.Info( - "reconciling", - "machine", req.NamespaceName, - "switchPosition", machine.Spec.SwitchPosition, +logger.Info( + "reconciling", + "machine", req.NamespacedName, + "switchPosition", machine.Spec.SwitchPosition, ) ``` @@ -288,7 +289,7 @@ Then: -- -🤔 +We get a bunch of errors and go stack traces! 🤔 --- @@ -324,7 +325,7 @@ Let's try to update the machine like this: if machine.Spec.SwitchPosition != "down" { machine.Spec.SwitchPosition = "down" if err := r.Update(ctx, &machine); err != nil { - log.Info("error updating switch position") + logger.Info("error updating switch position") return ctrl.Result{}, client.IgnoreNotFound(err) } } @@ -344,9 +345,9 @@ Again - update, `make run`, test. (maybe with degraded behavior in the meantime) -- Status will almost always be a sub-resource +- Status will almost always be a sub-resource, so that it can be updated separately - (so that it can be updated separately "cheaply") + (and potentially with different permissions) --- @@ -399,8 +400,8 @@ class: extra-details ## To requeue ... -`return ctrl.Result{RequeueAfter: 1 * time.Second}` - +`return ctrl.Result{RequeueAfter: 1 * time.Second}, nil` + - That means: "try again in 1 second, and I will check if progress was made" - This *does not* guarantee that we will be called exactly 1 second later: @@ -409,7 +410,9 @@ class: extra-details - we might be called after (if the controller is busy with other objects) -- If we are waiting for another resource to change, there is an even better way! +- If we are waiting for another Kubernetes resource to change, there is a better way + + (explained on next slide) --- @@ -417,23 +420,41 @@ class: extra-details `return ctrl.Result{}, nil` -- That means: "no need to set an alarm; we'll be notified some other way" +- That means: "we're done here!" -- Use this if we are waiting for another resource to update +- This is also what we should use if we are waiting for another resource (e.g. a LoadBalancer to be provisioned, a Pod to be ready...) -- For this to work, we need to set a *watch* (more on that later) +- In that case, we will need to set a *watch* (more on that later) + +--- + +## Keeping track of state + +- If we simply requeue the object to examine it 1 second later... + +- ...We'll keep examining/requeuing it forever! + +- We need to "remember" that we saw it (and when) + +- Option 1: keep state in controller + + (e.g. an internal `map`) + +- Option 2: keep state in the object + + (typically in its status field) + +- Tradeoffs: concurrency / failover / control plane overhead... --- ## "Improving" our controller, take 2 -- Let's store in the machine status the moment when we saw it +Let's store in the machine status the moment when we saw it: ```go -// +kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date - type MachineStatus struct { // Time at which the machine was noticed by our controller. SeenAt *metav1.Time ``json:"seenAt,omitempty"`` @@ -446,6 +467,12 @@ Note: `date` fields don't display timestamps in the future. (That's why for this example it's simpler to use `seenAt` rather than `changeAt`.) +And for better visibility, add this along with the other `printcolumn` comments: + +```go +//+kubebuilder:printcolumn:JSONPath=".status.seenAt",name=Seen,type=date +``` + --- ## Set `seenAt` @@ -457,7 +484,7 @@ if machine.Status.SeenAt == nil { now := metav1.Now() machine.Status.SeenAt = &now if err := r.Status().Update(ctx, &machine); err != nil { - log.Info("error updating status.seenAt") + logger.Info("error updating status.seenAt") return ctrl.Result{}, client.IgnoreNotFound(err) } return ctrl.Result{RequeueAfter: 5 * time.Second}, nil @@ -478,8 +505,9 @@ if machine.Spec.SwitchPosition != "down" { changeAt := machine.Status.SeenAt.Time.Add(5 * time.Second) if now.Time.After(changeAt) { machine.Spec.SwitchPosition = "down" + machine.Status.SeenAt = nil if err := r.Update(ctx, &machine); err != nil { - log.Info("error updating switch position") + logger.Info("error updating switch position") return ctrl.Result{}, client.IgnoreNotFound(err) } } @@ -496,15 +524,33 @@ if machine.Spec.SwitchPosition != "down" { - We will now have two kinds of objects: machines, and switches +- Machines will store the number of switches in their spec + - Machines should have *at least* one switch, possibly *multiple ones* -- The position will now be stored in the switch, not the machine +- Our controller will automatically create switches if needed -- The machine will also expose the combined state of the switches + (a bit like the ReplicaSet controller automatically creates Pods) - The switches will be tied to their machine through a label -(See next slide for an example) + (let's pick `machine=name-of-the-machine`) + +--- + +## Switch state + +- The position of a switch will now be stored in the switch + + (not in the machine like in the first scenario) + +- The machine will also expose the combined state of the switches + + (through its status) + +- The machine's status will be automatically updated by the controller + + (each time a switch is added/changed/removed) --- @@ -516,7 +562,7 @@ NAME SWITCHES POSITIONS machine-cz2vl 3 ddd machine-vf4xk 1 d -[jp@hex ~]$ kubectl get switches --show-labels +[jp@hex ~]$ kubectl get switches --show-labels NAME POSITION SEEN LABELS switch-6wmjw down machine=machine-cz2vl switch-b8csg down machine=machine-cz2vl @@ -530,39 +576,95 @@ switch-rc59l down machine=machine-vf4xk ## Tasks -Create the new resource type (but don't create a controller): +1. Create the new resource type (but don't create a controller) + +2. Update `machine_types.go` and `switch_types.go` + +3. Implement logic to display machine status (status of its switches) + +4. Implement logic to automatically create switches + +5. Implement logic to flip all switches down immediately + +6. Then tweak it so that a given machine doesn't flip more than one switch every 5 seconds + +*See next slides for detailed steps!* + +--- + +## Creating the new type ```bash kubebuilder create api --group useless --version v1alpha1 --kind Switch ``` -Update `machine_types.go` and `switch_types.go`. - -Implement the logic so that the controller flips all switches down immediately. - -Then change it so that a given machine doesn't flip more than one switch every 5 seconds. - -See next slides for hints! +Note: this time, only create a new custom resource; not a new controller. --- -## Listing objects +## Updating our types -We can use the `List` method with filters: +- Move the "switch position" and "seen at" to the new `Switch` type -```go -var switches uselessv1alpha1.SwitchList +- Update the `Machine` type to have: -if err := r.List(ctx, &switches, - client.InNamespace(req.Namespace), - client.MatchingLabels{"machine": req.Name}, - ); err != nil { - log.Error(err, "unable to list switches of the machine") - return ctrl.Result{}, client.IgnoreNotFound(err) -} + - `spec.switches` (Go type: `int`, JSON type: `integer`) -log.Info("Found switches", "switches", switches) -``` + - `status.positions` of type `string` + +- Bonus points for adding [CRD Validation](https://book.kubebuilder.io/reference/markers/crd-validation.html) to the numbers of switches! + +- Then install the new CRDs with `make install` + +- Create a Machine, and a Switch linked to the Machine (by setting the `machine` label) + +--- + +## Listing switches + +- Switches are associated to Machines with a label + + (`kubectl label switch switch-xyz machine=machine-xyz`) + +- We can retrieve associated switches like this: + + ```go + var switches uselessv1alpha1.SwitchList + + if err := r.List(ctx, &switches, + client.InNamespace(req.Namespace), + client.MatchingLabels{"machine": req.Name}, + ); err != nil { + logger.Error(err, "unable to list switches of the machine") + return ctrl.Result{}, client.IgnoreNotFound(err) + } + + logger.Info("Found switches", "switches", switches) + ``` + +--- + +## Updating status + +- Each time we reconcile a Machine, let's update its status: + + ```go + status := "" + for _, sw := range switches.Items { + status += string(sw.Spec.Position[0]) + } + machine.Status.Positions = status + if err := r.Status().Update(ctx, &machine); err != nil { + ... + ``` + +- Run the controller and check that POSITIONS gets updated + +- Add more switches linked to the same machine + +- ...The POSITIONS don't get updated, unless we restart the controller + +- We'll see later how to fix that! --- @@ -590,20 +692,28 @@ if err := r.Create(ctx, &sw); err != nil { ... --- +## Create missing switches + +- In our reconciler, if a machine doesn't have enough switches, create them! + +- Option 1: directly create the number of missing switches + +- Option 2: create only one switch (and rely on later requeuing) + +- Note: option 2 won't quite work yet, since we haven't set up *watches* yet + +--- + ## Watches -- Our controller will correctly flip switches when it starts - -- It will also react to machine updates - -- But it won't react if we directly touch the switches! - -- By default, it only monitors machines, not switches +- Our controller doesn't react when switches are created/updated/deleted - We need to tell it to watch switches - We also need to tell it how to map a switch to its machine + (so that the correct machine gets queued and reconciled when a switch is updated) + --- ## Mapping a switch to its machine @@ -611,16 +721,15 @@ if err := r.Create(ctx, &sw); err != nil { ... Define the following helper function: ```go -func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Request { - r.Log.Debug("mos", "obj", obj) - return []ctrl.Request{ - ctrl.Request{ - NamespacedName: types.NamespacedName{ - Name: obj.Meta.GetLabels()["machine"], - Namespace: obj.Meta.GetNamespace(), - }, - }, - } +func (r *MachineReconciler) machineOfSwitch(obj client.Object) []ctrl.Request { + return []ctrl.Request{ + ctrl.Request{ + NamespacedName: types.NamespacedName{ + Name: obj.GetLabels()["machine"], + Namespace: obj.GetNamespace(), + }, + }, + } } ``` @@ -631,24 +740,46 @@ func (r *MachineReconciler) machineOfSwitch(obj handler.MapObject) []ctrl.Reques Update the `SetupWithManager` method in the controller: ```go +// SetupWithManager sets up the controller with the Manager. func (r *MachineReconciler) SetupWithManager(mgr ctrl.Manager) error { - return ctrl.NewControllerManagedBy(mgr). - For(&uselessv1alpha1.Machine{}). - Owns(&uselessv1alpha1.Switch{}). - Watches( - &source.Kind{Type: &uselessv1alpha1.Switch{}}, - &handler.EnqueueRequestsFromMapFunc{ - ToRequests: handler.ToRequestsFunc(r.machineOfSwitch), - }). - Complete(r) + return ctrl.NewControllerManagedBy(mgr). + For(&uselessv1alpha1.Machine{}). + Owns(&uselessv1alpha1.Switch{}). + Watches( + &source.Kind{Type: &uselessv1alpha1.Switch{}}, + handler.EnqueueRequestsFromMapFunc(r.machineOfSwitch), + ). + Complete(r) } ``` -After this, our controller should now react to switch changes. +--- + +## ...And a few extra imports + +Import the following packages referenced by the previous code: + +```go +"sigs.k8s.io/controller-runtime/pkg/handler" +"sigs.k8s.io/controller-runtime/pkg/source" +"k8s.io/apimachinery/pkg/types" +``` + +After this, when we update a switch, it should reflect on the machine. + +(Try to change switch positions and see the machine status update!) --- -## Bonus points +## Flipping switches + +- Now re-add logic to flip switches that are not in "down" position + +- Re-add logic to wait a few seconds before flipping a switch + +- Change the logic to toggle one switch per machine every few seconds + + (i.e. don't change all the switches for a machine; move them one at a time) - Handle "scale down" of a machine (by deleting extraneous switches) @@ -660,9 +791,25 @@ After this, our controller should now react to switch changes. --- +## Other possible improvements + +- Formalize resource ownership + + (by setting `ownerReferences` in the switches) + +- This can simplify the watch mechanism a bit + +- Allow to define a selector + + (instead of using the hard-coded `machine` label) + +- And much more! + +--- + ## Acknowledgements -- Useless Operator, by [L Körbes](https://twitter.com/ellenkorbes) +- Useless Operator, by [V Körbes](https://twitter.com/veekorbes) [code](https://github.com/tilt-dev/uselessoperator) |