mirror of
https://github.com/jpetazzo/container.training.git
synced 2026-07-21 22:07:13 +00:00
384 lines
8.7 KiB
Markdown
384 lines
8.7 KiB
Markdown
# Kubernetes architecture
|
|
|
|
We can arbitrarily split Kubernetes in two parts:
|
|
|
|
- the *nodes*, a set of machines that run our containerized workloads;
|
|
|
|
- the *control plane*, a set of processes implementing the Kubernetes APIs.
|
|
|
|
Kubernetes also relies on underlying infrastructure:
|
|
|
|
- servers, network connectivity (obviously!),
|
|
|
|
- optional components like storage systems, load balancers ...
|
|
|
|
---
|
|
|
|
## Control plane location
|
|
|
|
The control plane can run:
|
|
|
|
- in containers, on the same nodes that run other application workloads
|
|
|
|
(example: [Minikube](https://github.com/kubernetes/minikube); 1 node runs everything, [kind](https://kind.sigs.k8s.io/))
|
|
|
|
- on a dedicated node
|
|
|
|
(example: a cluster installed with kubeadm)
|
|
|
|
- on a dedicated set of nodes
|
|
|
|
(example: [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way); [kops](https://github.com/kubernetes/kops))
|
|
|
|
- outside of the cluster
|
|
|
|
(example: most managed clusters like AKS, EKS, GKE)
|
|
|
|
---
|
|
|
|
class: pic
|
|
|
|
|
|
|
|
---
|
|
|
|
## What runs on a node
|
|
|
|
- Our containerized workloads
|
|
|
|
- A container engine like Docker, CRI-O, containerd...
|
|
|
|
(in theory, the choice doesn't matter, as the engine is abstracted by Kubernetes)
|
|
|
|
- kubelet: an agent connecting the node to the cluster
|
|
|
|
(it connects to the API server, registers the node, receives instructions)
|
|
|
|
- kube-proxy: a component used for internal cluster communication
|
|
|
|
(note that this is *not* an overlay network or a CNI plugin!)
|
|
|
|
---
|
|
|
|
## What's in the control plane
|
|
|
|
- Everything is stored in etcd
|
|
|
|
(it's the only stateful component)
|
|
|
|
- Everyone communicates exclusively through the API server:
|
|
|
|
- we (users) interact with the cluster through the API server
|
|
|
|
- the nodes register and get their instructions through the API server
|
|
|
|
- the other control plane components also register with the API server
|
|
|
|
- API server is the only component that reads/writes from/to etcd
|
|
|
|
---
|
|
|
|
## Communication protocols: API server
|
|
|
|
- The API server exposes a REST API
|
|
|
|
(except for some calls, e.g. to attach interactively to a container)
|
|
|
|
- Almost all requests and responses are JSON following a strict format
|
|
|
|
- For performance, the requests and responses can also be done over protobuf
|
|
|
|
(see this [design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md) for details)
|
|
|
|
- In practice, protobuf is used for all internal communication
|
|
|
|
(between control plane components, and with kubelet)
|
|
|
|
---
|
|
|
|
## Communication protocols: on the nodes
|
|
|
|
The kubelet agent uses a number of special-purpose protocols and interfaces, including:
|
|
|
|
- CRI (Container Runtime Interface)
|
|
|
|
- used for communication with the container engine
|
|
- abstracts the differences between container engines
|
|
- based on gRPC+protobuf
|
|
|
|
- [CNI (Container Network Interface)](https://github.com/containernetworking/cni/blob/master/SPEC.md)
|
|
|
|
- used for communication with network plugins
|
|
- network plugins are implemented as executable programs invoked by kubelet
|
|
- network plugins provide IPAM
|
|
- network plugins set up network interfaces in pods
|
|
|
|
---
|
|
|
|
class: pic
|
|
|
|
|
|
|
|
---
|
|
|
|
# The Kubernetes API
|
|
|
|
[
|
|
*The Kubernetes API server is a "dumb server" which offers storage, versioning, validation, update, and watch semantics on API resources.*
|
|
](
|
|
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md#proposal-and-motivation
|
|
)
|
|
|
|
([Clayton Coleman](https://twitter.com/smarterclayton), Kubernetes Architect and Maintainer)
|
|
|
|
What does that mean?
|
|
|
|
---
|
|
|
|
## The Kubernetes API is declarative
|
|
|
|
- We cannot tell the API, "run a pod"
|
|
|
|
- We can tell the API, "here is the definition for pod X"
|
|
|
|
- The API server will store that definition (in etcd)
|
|
|
|
- *Controllers* will then wake up and create a pod matching the definition
|
|
|
|
---
|
|
|
|
## The core features of the Kubernetes API
|
|
|
|
- We can create, read, update, and delete objects
|
|
|
|
- We can also *watch* objects
|
|
|
|
(be notified when an object changes, or when an object of a given type is created)
|
|
|
|
- Objects are strongly typed
|
|
|
|
- Types are *validated* and *versioned*
|
|
|
|
- Storage and watch operations are provided by etcd
|
|
|
|
(note: the [k3s](https://k3s.io/) project allows us to use sqlite instead of etcd)
|
|
|
|
---
|
|
|
|
## Let's experiment a bit!
|
|
|
|
- For the exercises in this section, connect to the first node of the `test` cluster
|
|
|
|
.exercise[
|
|
|
|
- SSH to the first node of the test cluster
|
|
|
|
- Check that the cluster is operational:
|
|
```bash
|
|
kubectl get nodes
|
|
```
|
|
|
|
- All nodes should be `Ready`
|
|
|
|
]
|
|
|
|
---
|
|
|
|
## Create
|
|
|
|
- Let's create a simple object
|
|
|
|
.exercise[
|
|
|
|
- Create a namespace with the following command:
|
|
```bash
|
|
kubectl create -f- <<EOF
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: hello
|
|
EOF
|
|
```
|
|
|
|
]
|
|
|
|
This is equivalent to `kubectl create namespace hello`.
|
|
|
|
---
|
|
|
|
## Read
|
|
|
|
- Let's retrieve the object we just created
|
|
|
|
.exercise[
|
|
|
|
- Read back our object:
|
|
```bash
|
|
kubectl get namespace hello -o yaml
|
|
```
|
|
|
|
]
|
|
|
|
We see a lot of data that wasn't here when we created the object.
|
|
|
|
Some data was automatically added to the object (like `spec.finalizers`).
|
|
|
|
Some data is dynamic (typically, the content of `status`.)
|
|
|
|
---
|
|
|
|
## API requests and responses
|
|
|
|
- Almost every Kubernetes API payload (requests and responses) has the same format:
|
|
```yaml
|
|
apiVersion: xxx
|
|
kind: yyy
|
|
metadata:
|
|
name: zzz
|
|
(more metadata fields here)
|
|
(more fields here)
|
|
```
|
|
|
|
- The fields shown above are mandatory, except for some special cases
|
|
|
|
(e.g.: in lists of resources, the list itself doesn't have a `metadata.name`)
|
|
|
|
- We show YAML for convenience, but the API uses JSON
|
|
|
|
(with optional protobuf encoding)
|
|
|
|
---
|
|
|
|
class: extra-details
|
|
|
|
## API versions
|
|
|
|
- The `apiVersion` field corresponds to an *API group*
|
|
|
|
- It can be either `v1` (aka "core" group or "legacy group"), or `group/versions`; e.g.:
|
|
|
|
- `apps/v1`
|
|
- `rbac.authorization.k8s.io/v1`
|
|
- `extensions/v1beta1`
|
|
|
|
- It does not indicate which version of Kubernetes we're talking about
|
|
|
|
- It *indirectly* indicates the version of the `kind`
|
|
|
|
(which fields exist, their format, which ones are mandatory...)
|
|
|
|
- A single resource type (`kind`) is rarely versioned alone
|
|
|
|
(e.g.: the `batch` API group contains `jobs` and `cronjobs`)
|
|
|
|
---
|
|
|
|
## Update
|
|
|
|
- Let's update our namespace object
|
|
|
|
- There are many ways to do that, including:
|
|
|
|
- `kubectl apply` (and provide an updated YAML file)
|
|
- `kubectl edit`
|
|
- `kubectl patch`
|
|
- many helpers, like `kubectl label`, or `kubectl set`
|
|
|
|
- In each case, `kubectl` will:
|
|
|
|
- get the current definition of the object
|
|
- compute changes
|
|
- submit the changes (with `PATCH` requests)
|
|
|
|
---
|
|
|
|
## Adding a label
|
|
|
|
- For demonstration purposes, let's add a label to the namespace
|
|
|
|
- The easiest way is to use `kubectl label`
|
|
|
|
.exercise[
|
|
|
|
- In one terminal, watch namespaces:
|
|
```bash
|
|
kubectl get namespaces --show-labels -w
|
|
```
|
|
|
|
- In the other, update our namespace:
|
|
```bash
|
|
kubectl label namespaces hello color=purple
|
|
```
|
|
|
|
]
|
|
|
|
We demonstrated *update* and *watch* semantics.
|
|
|
|
---
|
|
|
|
## What's special about *watch*?
|
|
|
|
- The API server itself doesn't do anything: it's just a fancy object store
|
|
|
|
- All the actual logic in Kubernetes is implemented with *controllers*
|
|
|
|
- A *controller* watches a set of resources, and takes action when they change
|
|
|
|
- Examples:
|
|
|
|
- when a Pod object is created, it gets scheduled and started
|
|
|
|
- when a Pod belonging to a ReplicaSet terminates, it gets replaced
|
|
|
|
- when a Deployment object is updated, it can trigger a rolling update
|
|
|
|
---
|
|
|
|
# Other control plane components
|
|
|
|
- API server ✔️
|
|
|
|
- etcd ✔️
|
|
|
|
- Controller manager
|
|
|
|
- Scheduler
|
|
|
|
---
|
|
|
|
## Controller manager
|
|
|
|
- This is a collection of loops watching all kinds of objects
|
|
|
|
- That's where the actual logic of Kubernetes lives
|
|
|
|
- When we create a Deployment (e.g. with `kubectl create deployment web --image=nginx`),
|
|
|
|
- we create a Deployment object
|
|
|
|
- the Deployment controller notices it, and creates a ReplicaSet
|
|
|
|
- the ReplicaSet controller notices the ReplicaSet, and creates a Pod
|
|
|
|
---
|
|
|
|
## Scheduler
|
|
|
|
- When a pod is created, it is in `Pending` state
|
|
|
|
- The scheduler (or rather: *a scheduler*) must bind it to a node
|
|
|
|
- Kubernetes comes with an efficient scheduler with many features
|
|
|
|
- if we have special requirements, we can add another scheduler
|
|
<br/>
|
|
(example: this [demo scheduler](https://github.com/kelseyhightower/scheduler) uses the cost of nodes, stored in node annotations)
|
|
|
|
- A pod might stay in `Pending` state for a long time:
|
|
|
|
- if the cluster is full
|
|
|
|
- if the pod has special constraints that can't be met
|
|
|
|
- if the scheduler is not running (!)
|