container.training/slides/k8s/control-plane-auth.md

# Securing the control plane

- Many components accept connections (and requests) from others:

  - API server

  - etcd

  - kubelet

- We must secure these connections:

  - to deny unauthorized requests

  - to prevent eavesdropping secrets, tokens, and other sensitive information

- Disabling authentication and/or authorization is **strongly discouraged**

  (but it's possible to do it, e.g. for learning / troubleshooting purposes)

---

## Authentication and authorization

- Authentication (checking "who you are") is done with mutual TLS

 (both the client and the server need to hold a valid certificate)

- Authorization (checking "what you can do") is done in different ways

  - the API server implements a sophisticated permission logic (with RBAC)
  
  - some services will defer authorization to the API server (through webhooks)

  - some services require a certificate signed by a particular CA / sub-CA

---

## In practice

- We will review the various communication channels in the control plane

- We will describe how they are secured

- When TLS certificates are used, we will indicate:

  - which CA signs them

  - what their subject (CN) should be, when applicable

- We will indicate how to configure security (client- and server-side)

---

## etcd peers

- Replication and coordination of etcd happens on a dedicated port

  (typically port 2380; the default port for normal client connections is 2379)

- Authentication uses TLS certificates with a separate sub-CA

  (otherwise, anyone with a Kubernetes client certificate could access etcd!)

- The etcd command line flags involved are:

   `--peer-client-cert-auth=true` to activate it

   `--peer-cert-file`, `--peer-key-file`, `--peer-trusted-ca-file`

---

## etcd clients

- The only¹ thing that connects to etcd is the API server

- Authentication uses TLS certificates with a separate sub-CA

  (for the same reasons as for etcd inter-peer authentication)

- The etcd command line flags involved are:

  `--client-cert-auth=true` to activate it

  `--trusted-ca-file`, `--cert-file`, `--key-file`

- The API server command line flags involved are:

  `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`

.footnote[¹Technically, there is also the etcd healthcheck. Let's ignore it for now.]

---

## etcd authorization

- etcd supports RBAC, but Kubernetes doesn't use it by default

  (note: etcd RBAC is completely different from Kubernetes RBAC!)

- By default, etcd access is "all or nothing"

  (if you have a valid certificate, you get in)

- Be very careful if you use the same root CA for etcd and other things

  (if etcd trusts the root CA, then anyone with a valid cert gets full etcd access)

- For more details, check the following resources:

  - [etcd documentation on authentication](https://etcd.io/docs/current/op-guide/authentication/)

  - [PKI The Wrong Way](https://www.youtube.com/watch?v=gcOLDEzsVHI) at KubeCon NA 2020

---


## API server clients

- The API server has a sophisticated authentication and authorization system

- For connections coming from other components of the control plane:

  - authentication uses certificates (trusting the certificates' subject or CN)

  - authorization uses whatever mechanism is enabled (most oftentimes, RBAC)

- The relevant API server flags are:

  `--client-ca-file`, `--tls-cert-file`, `--tls-private-key-file`

- Each component connecting to the API server takes a `--kubeconfig` flag

  (to specify a kubeconfig file containing the CA cert, client key, and client cert)

- Yes, that kubeconfig file follows the same format as our `~/.kube/config` file!

---

## Kubelet and API server

- Communication between kubelet and API server can be established both ways

- Kubelet → API server:

  - kubelet registers itself ("hi, I'm node42, do you have work for me?")

  - connection is kept open and re-established if it breaks

  - that's how the kubelet knows which pods to start/stop

- API server → kubelet:

  - used to retrieve logs, exec, attach to containers

---

## Kubelet → API server

- Kubelet is started with `--kubeconfig` with API server information

- The client certificate of the kubelet will typically have:

  `CN=system:node:<nodename>` and groups `O=system:nodes`

- Nothing special on the API server side

  (it will authenticate like any other client)

---

## API server → kubelet

- Kubelet is started with the flag `--client-ca-file`

  (typically using the same CA as the API server)

- API server will use a dedicated key pair when contacting kubelet

  (specified with `--kubelet-client-certificate` and `--kubelet-client-key`)

- Authorization uses webhooks

  (enabled with `--authorization-mode=Webhook` on kubelet)

- The webhook server is the API server itself

  (the kubelet sends back a request to the API server to ask, "can this person do that?")

---

## Scheduler

- The scheduler connects to the API server like an ordinary client

- The certificate of the scheduler will have `CN=system:kube-scheduler`

---

## Controller manager

- The controller manager is also a normal client to the API server

- Its certificate will have `CN=system:kube-controller-manager`

- If we use the CSR API, the controller manager needs the CA cert and key

  (passed with flags `--cluster-signing-cert-file` and `--cluster-signing-key-file`)

- We usually want the controller manager to generate tokens for service accounts

- These tokens deserve some details (on the next slide!)

---

class: extra-details

## How are these permissions set up?

- A bunch of roles and bindings are defined as constants in the API server code:

  [auth/authorizer/rbac/bootstrappolicy/policy.go](https://github.com/kubernetes/kubernetes/blob/release-1.19/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go#L188)

- They are created automatically when the API server starts:

  [registry/rbac/rest/storage_rbac.go](https://github.com/kubernetes/kubernetes/blob/release-1.19/pkg/registry/rbac/rest/storage_rbac.go#L140)

- We must use the correct Common Names (`CN`) for the control plane certificates

  (since the bindings defined above refer to these common names)

---

## Service account tokens

- Each time we create a service account, the controller manager generates a token

- These tokens are JWT tokens, signed with a particular key

- These tokens are used for authentication with the API server

  (and therefore, the API server needs to be able to verify their integrity)

- This uses another keypair:

  - the private key (used for signature) is passed to the controller manager
    <br/>(using flags `--service-account-private-key-file` and `--root-ca-file`)

  - the public key (used for verification) is passed to the API server
    <br/>(using flag `--service-account-key-file`)

---

## kube-proxy

- kube-proxy is "yet another API server client"

- In many clusters, it runs as a Daemon Set

- In that case, it will have its own Service Account and associated permissions

- It will authenticate using the token of that Service Account

---

## Webhooks

- We mentioned webhooks earlier; how does that really work?

- The Kubernetes API has special resource types to check permissions

- One of them is SubjectAccessReview

- To check if a particular user can do a particular action on a particular resource:

  - we prepare a SubjectAccessReview object

  - we send that object to the API server

  - the API server responds with allow/deny (and optional explanations)

- Using webhooks for authorization = sending SAR to authorize each request

---

## Subject Access Review

Here is an example showing how to check if `jean.doe` can `get` some `pods` in `kube-system`:

```bash
kubectl -v9 create -f- <<EOF
apiVersion: authorization.k8s.io/v1
kind: SubjectAccessReview
spec:
  user: jean.doe
  groups:
  - foo
  - bar
  resourceAttributes:
    #group: blah.k8s.io
    namespace: kube-system
    resource: pods
    verb: get
    #name: web-xyz1234567-pqr89
EOF
```

???

:EN:- Control plane authentication
:FR:- Sécurisation du plan de contrôle