github/container.training

Fork 0

mirror of https://github.com/jpetazzo/container.training.git synced 2026-05-09 10:26:39 +00:00

Files

Jerome Petazzoni ce8dc2cdff 🔧 Minor tweaks and improvements

2021-04-07 19:23:55 +02:00

8.2 KiB

Raw Blame History

The Aggregation Layer

The aggregation layer is a way to extend the Kubernetes API
It is similar to CRDs
- it lets us define new resource types
- these resources can then be used with kubectl and other clients
The implementation is very different
- CRDs are handled within the API server
- the aggregation layer offloads requests to another process
They are designed for very different use-cases

CRDs vs aggregation layer

The Kubernetes API is a REST-ish API with a hierarchical structure
It can be extended with Custom Resource Definifions (CRDs)
Custom resources are managed by the Kubernetes API server
- we don't need to write code
- the API server does all the heavy lifting
- these resources are persisted in Kubernetes' "standard" database
  (for most installations, that's etcd)
We can also define resources that are not managed by the API server

(the API server merely proxies the requests to another server)

Which one is best?

For things that "map" well to objects stored in a traditional database:

probably CRDs
For things that "exist" only in Kubernetes and don't represent external resources:

probably CRDs
For things that are read-only, at least from Kubernetes' perspective:

probably aggregation layer
For things that can't be stored in etcd because of size or access patterns:

probably aggregation layer

How are resources organized?

Let's have a look at the Kubernetes API hierarchical structure
We'll ask kubectl to show us the exacts requests that it's making

Check the URI for a cluster-scope, "core" resource, e.g. a Node:
```
kubectl -v6 get node node1
```
Check the URI for a cluster-scope, "non-core" resource, e.g. a ClusterRole:
```
kubectl -v6 get clusterrole view
```

]

Core vs non-core

This is the structure of the URIs that we just checked:

  /api/v1/nodes/node1
       ↑    ↑     ↑
   `version` `kind` `name`

  /apis/rbac.authorization.k8s.io/v1/clusterroles/view
                      ↑           ↑        ↑       ↑
                    `group`      `version`   `kind`    `name`

There is no group for "core" resources
Or, we could say that the group, core, is implied

Group-Version-Kind

In the API server, the Group-Version-Kind triple maps to a Go type

(look for all the "GVK" occurrences in the source code!)
In the API server URI router, the GVK is parsed "relatively early"

(so that the server can know which resource we're talking about)
"Well, actually ..." Things are a bit more complicated, see next slides!

Namespaced resources

What about namespaced resources?

Check the URI for a namespaced, "core" resource, e.g. a Service:
```
kubectl -v6 get service kubernetes --namespace default
```

]

Here are what namespaced resources URIs look like:

  /api/v1/namespaces/default/services/kubernetes
       ↑               ↑        ↑         ↑
    `version`        `namespace`  `kind`      `name`

  /apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy
         ↑   ↑                 ↑           ↑          ↑
     `group`  `version`        `namespace`     `kind`       `name`

Subresources

Many resources have subresources, for instance:
- /status (decouples status updates from other updates)
- /scale (exposes a consistent interface for autoscalers)
- /proxy (allows access to HTTP resources)
- /portforward (used by kubectl port-forward)
- /logs (access pod logs)
These are added at the end of the URI

Accessing a subresource

List kube-proxy pods:

  kubectl get pods --namespace=kube-system --selector=k8s-app=kube-proxy
  PODNAME=$(
    kubectl get pods --namespace=kube-system --selector=k8s-app=kube-proxy \
            -o json | jq -r .items[0].metadata.name)

Execute a command in a pod, showing the API requests:

kubectl -v6 exec --namespace=kube-system $PODNAME -- echo hello world

]

The full request looks like:

POST https://.../api/v1/namespaces/kube-system/pods/kube-proxy-c7rlw/exec?
command=echo&command=hello&command=world&container=kube-proxy&stderr=true&stdout=true

Listing what's supported on the server

There are at least three useful commands to introspect the API server

List resources types, their group, kind, short names, and scope:
```
kubectl api-resources
```
List API groups + versions:
```
kubectl api-versions
```
List APIServices:
```
kubectl get apiservices
```

]

🤔 What's the difference between the last two?

API registration

kubectl api-versions shows all API groups, including apiregistration.k8s.io
kubectl get apiservices shows the "routing table" for API requests
The latter doesn't show apiregistration.k8s.io

(APIServices belong to apiregistration.k8s.io)
Most API groups are Local (handled internally by the API server)
If we're running the metrics-server, it should handle metrics.k8s.io
This is an API group handled outside of the API server
This is the aggregation layer!

Finding resources

The following assumes that metrics-server is deployed on your cluster.

Check that the metrics.k8s.io is registered with metrics-server:
```
kubectl get apiservices | grep metrics.k8s.io
```
Check the resource kinds registered in the metrics.k8s.io group:
```
kubectl api-resources --api-group=metrics.k8s.io
```

]

(If the output of either command is empty, install metrics-server first.)

`nodes` vs `nodes`

We can have multiple resources with the same name

Look for resources named node:
```
kubectl api-resources | grep -w nodes
```

Compare the output of both commands:

kubectl get nodes
kubectl get nodes.metrics.k8s.io

]

🤔 What are the second kind of nodes? How can we see what's really in them?

Node vs NodeMetrics

nodes.metrics.k8s.io (aka NodeMetrics) don't have fancy printer columns
But we can look at the raw data (with -o json or -o yaml)

Look at NodeMetrics objects with one of these commands:

kubectl get -o yaml nodes.metrics.k8s.io
kubectl get -o yaml NodeMetrics

]

💡 Alright, these are the live metrics (CPU, RAM) for our nodes.

An easier way to consume metrics

We might have seen these metrics before ... With an easier command!

Display node metrics:
```
kubectl top nodes
```
Check which API requests happen behind the scenes:
```
kubectl top nodes -v6
```

]

Aggregation layer in practice

We can write an API server to handle a subset of the Kubernetes API
Then we can register that server by creating an APIService resource

Check the definition used for the metrics-server:

kubectl describe apiservices v1beta1.metrics.k8s.io

]

Group priority is used when multiple API groups provide similar kinds

(e.g. nodes and nodes.metrics.k8s.io as seen earlier)

Authentication flow

We have two Kubernetes API servers:
- "aggregator" (the main one; clients connect to it)
- "aggregated" (the one providing the extra API; aggregator connects to it)
Aggregator deals with client authentication
Aggregator authenticates with aggregated using mutual TLS
Aggregator passes (/forwards/proxies/...) requests to aggregated
Aggregated performs authorization by calling back aggregator

("can subject X perform action Y on resource Z?")

This doc page has very nice swim lanes showing that flow.

Discussion

Aggregation layer is great for metrics

(fast-changing, ephemeral data, that would be outrageously bad for etcd)
It could be a good fit to expose other REST APIs as a pass-thru

(but it's more common to see CRDs instead)

???

:EN:- The aggregation layer :FR:- Étendre l'API avec le aggregation layer

8.2 KiB Raw Blame History