# Securing the control plane - Many components accept connections (and requests) from others: - API server - etcd - kubelet - We must secure these connections: - to deny unauthorized requests - to prevent eavesdropping secrets, tokens, and other sensitive information - Disabling authentication and/or authorization is **strongly discouraged** (but it's possible to do it, e.g. for learning / troubleshooting purposes) --- ## Authentication and authorization - Authentication (checking "who you are") can be done in different ways: - with mutual TLS (both client and server need to hold a valid certificate) - with service account tokens (issued by the Kubernetes API server) - Authorization (checking "what you can do") can also be done in multiple ways: - the API server implements a sophisticated permission logic (with RBAC) - some services will defer authorization to the API server (through webhooks) - some services require a certificate signed by a particular CA / sub-CA - there is also a special "Node Authorizer" (for kubelet API access) --- ## Mutual TLS vs tokens - Service account tokens: - automatically generated by API server - can be exposed to pods through e.g. volume mounts - require the control plane to be up and running - can't be used by kubelets or by static pods - Mutual TLS: - requires manual generation (and renewal!) - doesn't require the control plane to be up and running - particularly relevant for kubelets and static pods --- ## In practice - We will review the various communication channels in the control plane - We will describe how they are secured - When TLS certificates are used, we will indicate: - which CA signs them - what their subject (CN) should be, when applicable - We will indicate how to configure security (client- and server-side) --- ## etcd peers - Replication and coordination of etcd happens on a dedicated port (typically port 2380; the default port for normal client connections is 2379) - Authentication uses TLS certificates with a separate sub-CA (otherwise, anyone with a Kubernetes client certificate could access etcd!) - The etcd command line flags involved are: `--peer-client-cert-auth=true` to activate it `--peer-cert-file`, `--peer-key-file`, `--peer-trusted-ca-file` --- ## etcd clients - The only¹ thing that connects to etcd is the API server - Authentication uses TLS certificates with a separate sub-CA (for the same reasons as for etcd inter-peer authentication) - The etcd command line flags involved are: `--client-cert-auth=true` to activate it `--trusted-ca-file`, `--cert-file`, `--key-file` - The API server command line flags involved are: `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile` .footnote[¹Technically, there is also the etcd healthcheck. Let's ignore it for now.] --- ## etcd authorization - etcd supports RBAC, but Kubernetes doesn't use it by default (note: etcd RBAC is completely different from Kubernetes RBAC!) - By default, etcd access is "all or nothing" (if you have a valid certificate, you get in) - Be very careful if you use the same root CA for etcd and other things (if etcd trusts the root CA, then anyone with a valid cert gets full etcd access) - For more details, check the following resources: - [etcd documentation on authentication](https://etcd.io/docs/current/op-guide/authentication/) - [PKI The Wrong Way](https://www.youtube.com/watch?v=gcOLDEzsVHI) at KubeCon NA 2020 --- ## API server authentication with TLS certificates - Some control plane components will authenticate with TLS certificates (typically: scheduler, controller manager; also: kubelets!) - The relevant API server flags are: `--client-ca-file`, `--tls-cert-file`, `--tls-private-key-file` - These clients will typically accept a `--kubeconfig` flag (to specify a kubeconfig file containing the CA cert, client key, and client cert) - Yes, that kubeconfig file follows the same format as our `~/.kube/config` file! --- ## API server authentication with tokens - Some control plane components may authenticate with Service Account tokens (typically: controllers like CNI, CSI, Ingress...) - The relevant API server flags are: `--service-account-signing-key-file`, `--service-account-issuer`, `--service-account-key-file` - These clients will automatically detect that they should use "in cluster config" - That detection relies on the following things to exist: - environment variables `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` - token in file `/var/run/secrets/kubernetes.io/serviceaccount/token` --- ## API server clients authorization - Most clients will rely on the `RBAC` authorizer - enabled with API server flag `--authorization-mode=RBAC` - that flag will automatically create a bunch of roles and bindings - clients should use standard names (e.g. `system:kube-scheduler`) - Kubelets will rely on the `Node` authorizer - enabled with API server flag `--authorization-mode=Node` - this authorizer makes sure that kubelets work on a "need-to-know" basis - kubelets should use standard names (`system:node:`) - Note: to enable both authorizers, use `--authorization-mode=RBAC,Node` --- class: extra-details ## How are these permissions set up? - A bunch of roles and bindings are defined as constants in the API server code: [auth/authorizer/rbac/bootstrappolicy/policy.go](https://github.com/kubernetes/kubernetes/blob/release-1.19/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go#L188) - They are created automatically when the API server starts: [registry/rbac/rest/storage_rbac.go](https://github.com/kubernetes/kubernetes/blob/release-1.19/pkg/registry/rbac/rest/storage_rbac.go#L140) - We must use the correct Common Names (`CN`) for the control plane certificates (since the bindings defined above refer to these common names) --- class: extra-details ## The Node Authorizer - Question: when should node `X` be able to access secret `Y`? -- - Answer: if, and only if, node `X` runs a pod that uses secret `Y` - The Node Authorizer implements that kind of logic - It also allows kubelets to set labels and taints for themselves (but not for other nodes) --- ## Kubelet and API server - Communication between kubelet and API server can be established both ways - Kubelet → API server: - kubelet registers itself ("hi, I'm node42, do you have work for me?") - connection is kept open and re-established if it breaks - that's how the kubelet knows which pods to start/stop - API server → kubelet: - used to retrieve logs, exec, attach to containers --- ## Kubelet → API server - Kubelet is started with `--kubeconfig` with API server information - The client certificate of the kubelet will typically have: `CN=system:node:` and groups `O=system:nodes` - Nothing special on the API server side (it will authenticate like any other client) - Authorization will typically require the Node Authorizer mentioned earlier ⚠️ Kubelet certificates need to be renewed regularly! - This is typically done through the CSR API --- ## API server → kubelet - Kubelet is started with the flag `--client-ca-file` (typically using the same CA as the API server) - API server will use a dedicated key pair when contacting kubelet (specified with `--kubelet-client-certificate` and `--kubelet-client-key`) - Authorization uses webhooks (enabled with `--authorization-mode=Webhook` on kubelet) - The webhook server is the API server itself (the kubelet sends back a request to the API server to ask, "can this person do that?") --- ## Scheduler - The scheduler connects to the API server like an ordinary client - The certificate of the scheduler will have `CN=system:kube-scheduler` --- ## Controller manager - The controller manager is also a normal client to the API server - Its certificate will have `CN=system:kube-controller-manager` - To improve security posture, each controller can use an individual Service Account - This is enabled with flag `--use-service-account-credentials=true` --- ## Controller manager keys - The controller can create Secrets holding Service Account tokens - this is enabled with flag `--service-account-private-key-file` - this was used in older versions of Kubernetes (before *bound tokens*) - in modern clusters, kubelet uses the `TokenRequest` API instead - If we use the CSR API, the controller manager needs the CA cert and key - the CSR API is used in many clusters to renew kubelet certificates - it's enabled with `--cluster-signing-cert-file` and `--cluster-signing-key-file` --- ## Service account tokens recap - These tokens are JWT tokens, signed with a particular key - These tokens are used for authentication with the API server (and therefore, the API server needs to be able to verify their integrity) - That key is passed to the API server using a couple of flags: - `--service-account-private-key-file` (used to issue tokens) - `--service-account-key-file` (used to verify tokens) - The private key is also passed to the controller manager
(using flag `--service-account-private-key-file`) --- ## kube-proxy - kube-proxy is "yet another API server client" - In many clusters, it runs as a Daemon Set - In that case, it will have its own Service Account and associated permissions - It will authenticate using the token of that Service Account - It's also possible (but rare) to run it with e.g. static pods (it will then require TLS keys; possibly the same as kubelet's!) --- class: extra-details ## Webhooks - We mentioned webhooks earlier; how does that really work? - The Kubernetes API has special resource types to check permissions - One of them is SubjectAccessReview - To check if a particular user can do a particular action on a particular resource: - we prepare a SubjectAccessReview object - we send that object to the API server - the API server responds with allow/deny (and optional explanations) - Using webhooks for authorization = sending SAR to authorize each request --- class: extra-details ## Subject Access Review Here is an example showing how to check if `jean.doe` can `get` some `pods` in `kube-system`: ```bash kubectl -v9 create -f- <