container.training/slides/k8s/extending-api.md

Extending the Kubernetes API

There are multiple ways to extend the Kubernetes API.

We are going to cover:

• Custom Resource Definitions (CRDs)

• Admission Webhooks

• The Aggregation Layer

---

Revisiting the API server

• The Kubernetes API server is a central point of the control plane

  (everything connects to it: controller manager, scheduler, kubelets)

• Almost everything in Kubernetes is materialized by a resource

• Resources have a type (or "kind")

  (similar to strongly typed languages)

• We can see existing types with kubectl api-resources

• We can list resources of a given type with kubectl get <type>

---

Creating new types

• We can create new types with Custom Resource Definitions (CRDs)

• CRDs are created dynamically

  (without recompiling or restarting the API server)

• CRDs themselves are resources:

  • we can create a new type with kubectl create and some YAML

  • we can see all our custom types with kubectl get crds

• After we create a CRD, the new type works just like built-in types

---

A very simple CRD

The YAML below describes a very simple CRD representing different kinds of coffee:

apiVersion: apiextensions.k8s.io/v1alpha1
kind: CustomResourceDefinition
metadata:
  name: coffees.container.training
spec:
  group: container.training
  version: v1alpha1
  scope: Namespaced
  names:
    plural: coffees
    singular: coffee
    kind: Coffee
    shortNames:
    - cof

---

Creating a CRD

• Let's create the Custom Resource Definition for our Coffee resource

.exercise[

• Load the CRD:

  kubectl apply -f ~/container.training/k8s/coffee-1.yaml
  

• Confirm that it shows up:

  kubectl get crds
  

]

---

Creating custom resources

The YAML below defines a resource using the CRD that we just created:

kind: Coffee
apiVersion: container.training/v1alpha1
metadata:
  name: arabica
spec:
  taste: strong

.exercise[

• Create a few types of coffee beans:

  kubectl apply -f ~/container.training/k8s/coffees.yaml
  

]

---

Viewing custom resources

• By default, kubectl get only shows name and age of custom resources

.exercise[

• View the coffee beans that we just created:

  kubectl get coffees
  

]

• We can improve that, but it's outside the scope of this section!

---

What can we do with CRDs?

There are many possibilities!

• Operators encapsulate complex sets of resources

  (e.g.: a PostgreSQL replicated cluster; an etcd cluster...
  see awesome operators and OperatorHub to find more)

• Custom use-cases like gitkube

  • creates a new custom type, Remote, exposing a git+ssh server

  • deploy by pushing YAML or Helm charts to that remote

• Replacing built-in types with CRDs

  (see this lightning talk by Tim Hockin)

---

Little details

• By default, CRDs are not validated

  (we can put anything we want in the spec)

• When creating a CRD, we can pass an OpenAPI v3 schema (BETA!)

  (which will then be used to validate resources)

• Generally, when creating a CRD, we also want to run a controller

  (otherwise nothing will happen when we create resources of that type)

• The controller will typically watch our custom resources

  (and take action when they are created/updated)

Examples: YAML to install the gitkube CRD, YAML to install a redis operator CRD *

---

(Ab)using the API server

• If we need to store something "safely" (as in: in etcd), we can use CRDs

• This gives us primitives to read/write/list objects (and optionally validate them)

• The Kubernetes API server can run on its own

  (without the scheduler, controller manager, and kubelets)

• By loading CRDs, we can have it manage totally different objects

  (unrelated to containers, clusters, etc.)

---

Service catalog

• Service catalog is another extension mechanism

• It's not extending the Kubernetes API strictly speaking

  (but it still provides new features!)

• It doesn't create new types; it uses:

  • ClusterServiceBroker
  • ClusterServiceClass
  • ClusterServicePlan
  • ServiceInstance
  • ServiceBinding

• It uses the Open service broker API

---

Admission controllers

• Admission controllers are another way to extend the Kubernetes API

• Instead of creating new types, admission controllers can transform or vet API requests

• The diagram on the next slide shows the path of an API request

  (courtesy of Banzai Cloud)

---

class: pic

---

Types of admission controllers

• Validating admission controllers can accept/reject the API call

• Mutating admission controllers can modify the API request payload

• Both types can also trigger additional actions

  (e.g. automatically create a Namespace if it doesn't exist)

• There are a number of built-in admission controllers

  (see documentation for a list)

• We can also dynamically define and register our own

---

class: extra-details

Some built-in admission controllers

• ServiceAccount:

  automatically adds a ServiceAccount to Pods that don't explicitly specify one

• LimitRanger:

  applies resource constraints specified by LimitRange objects when Pods are created

• NamespaceAutoProvision:

  automatically creates namespaces when an object is created in a non-existent namespace

Note: #1 and #2 are enabled by default; #3 is not.

---

Admission Webhooks

• We can setup admission webhooks to extend the behavior of the API server

• The API server will submit incoming API requests to these webhooks

• These webhooks can be validating or mutating

• Webhooks can be set up dynamically (without restarting the API server)

• To setup a dynamic admission webhook, we create a special resource:

  a ValidatingWebhookConfiguration or a MutatingWebhookConfiguration

• These resources are created and managed like other resources

  (i.e. kubectl create, kubectl get...)

---

Webhook Configuration

• A ValidatingWebhookConfiguration or MutatingWebhookConfiguration contains:

  • the address of the webhook

  • the authentication information to use with the webhook

  • a list of rules

• The rules indicate for which objects and actions the webhook is triggered

  (to avoid e.g. triggering webhooks when setting up webhooks)

---

The aggregation layer

• We can delegate entire parts of the Kubernetes API to external servers

• This is done by creating APIService resources

  (check them with kubectl get apiservices!)

• The APIService resource maps a type (kind) and version to an external service

• All requests concerning that type are sent (proxied) to the external service

• This allows to have resources like CRDs, but that aren't stored in etcd

• Example: metrics-server

  (storing live metrics in etcd would be extremely inefficient)

• Requires significantly more work than CRDs!

---

Documentation

• Custom Resource Definitions: when to use them

• Custom Resources Definitions: how to use them

• Service Catalog

• Built-in Admission Controllers

• Dynamic Admission Controllers

• Aggregation Layer

???

:EN:- Extending the Kubernetes API :EN:- Custom Resource Definitions (CRDs) :EN:- The aggregation layer :EN:- Admission control and webhooks

:FR:- Comment étendre l'API Kubernetes :FR:- Les CRDs (Custom Resource Definitions) :FR:- Extension via aggregation layer, admission control, webhooks