container.training/slides/k8s/control-plane-auth.md

Securing the control plane

• Many components accept connections (and requests) from others:

  • API server

  • etcd

  • kubelet

• We must secure these connections:

  • to deny unauthorized requests

  • to prevent eavesdropping secrets, tokens, and other sensitive information

• Disabling authentication and/or authorization is strongly discouraged

  (but it's possible to do it, e.g. for learning / troubleshooting purposes)

---

Authentication and authorization

• Authentication (checking "who you are") can be done in different ways:

  • with mutual TLS (both client and server need to hold a valid certificate)

  • with service account tokens (issued by the Kubernetes API server)

• Authorization (checking "what you can do") can also be done in multiple ways:

  • the API server implements a sophisticated permission logic (with RBAC)

  • some services will defer authorization to the API server (through webhooks)

  • some services require a certificate signed by a particular CA / sub-CA

  • there is also a special "Node Authorizer" (for kubelet API access)

---

Mutual TLS vs tokens

• Service account tokens:

  • automatically generated by API server

  • can be exposed to pods through e.g. volume mounts

  • require the control plane to be up and running

  • can't be used by kubelets or by static pods

• Mutual TLS:

  • requires manual generation (and renewal!)

  • doesn't require the control plane to be up and running

  • particularly relevant for kubelets and static pods

---

In practice

• We will review the various communication channels in the control plane

• We will describe how they are secured

• When TLS certificates are used, we will indicate:

  • which CA signs them

  • what their subject (CN) should be, when applicable

• We will indicate how to configure security (client- and server-side)

---

etcd peers

• Replication and coordination of etcd happens on a dedicated port

  (typically port 2380; the default port for normal client connections is 2379)

• Authentication uses TLS certificates with a separate sub-CA

  (otherwise, anyone with a Kubernetes client certificate could access etcd!)

• The etcd command line flags involved are:

  --peer-client-cert-auth=true to activate it

  --peer-cert-file, --peer-key-file, --peer-trusted-ca-file

---

etcd clients

• The only¹ thing that connects to etcd is the API server

• Authentication uses TLS certificates with a separate sub-CA

  (for the same reasons as for etcd inter-peer authentication)

• The etcd command line flags involved are:

  --client-cert-auth=true to activate it

  --trusted-ca-file, --cert-file, --key-file

• The API server command line flags involved are:

  --etcd-cafile, --etcd-certfile, --etcd-keyfile

.footnote[¹Technically, there is also the etcd healthcheck. Let's ignore it for now.]

---

etcd authorization

• etcd supports RBAC, but Kubernetes doesn't use it by default

  (note: etcd RBAC is completely different from Kubernetes RBAC!)

• By default, etcd access is "all or nothing"

  (if you have a valid certificate, you get in)

• Be very careful if you use the same root CA for etcd and other things

  (if etcd trusts the root CA, then anyone with a valid cert gets full etcd access)

• For more details, check the following resources:

  • etcd documentation on authentication

  • PKI The Wrong Way at KubeCon NA 2020

---

API server authentication with TLS certificates

• Some control plane components will authenticate with TLS certificates

  (typically: scheduler, controller manager; also: kubelets!)

• The relevant API server flags are:

  --client-ca-file, --tls-cert-file, --tls-private-key-file

• These clients will typically accept a --kubeconfig flag

  (to specify a kubeconfig file containing the CA cert, client key, and client cert)

• Yes, that kubeconfig file follows the same format as our ~/.kube/config file!

---

API server authentication with tokens

• Some control plane components may authenticate with Service Account tokens

  (typically: controllers like CNI, CSI, Ingress...)

• The relevant API server flags are:

  --service-account-signing-key-file, --service-account-issuer, --service-account-key-file

• These clients will automatically detect that they should use "in cluster config"

• That detection relies on the following things to exist:

  • environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT

  • token in file /var/run/secrets/kubernetes.io/serviceaccount/token

---

API server clients authorization

• Most clients will rely on the RBAC authorizer

  • enabled with API server flag --authorization-mode=RBAC

  • that flag will automatically create a bunch of roles and bindings

  • clients should use standard names (e.g. system:kube-scheduler)

• Kubelets will rely on the Node authorizer

  • enabled with API server flag --authorization-mode=Node

  • this authorizer makes sure that kubelets work on a "need-to-know" basis

  • kubelets should use standard names (system:node:<name-of-the-node>)

• Note: to enable both authorizers, use --authorization-mode=RBAC,Node

---

class: extra-details

How are these permissions set up?

• A bunch of roles and bindings are defined as constants in the API server code:

  auth/authorizer/rbac/bootstrappolicy/policy.go

• They are created automatically when the API server starts:

  registry/rbac/rest/storage_rbac.go

• We must use the correct Common Names (CN) for the control plane certificates

  (since the bindings defined above refer to these common names)

---

class: extra-details

The Node Authorizer

• Question: when should node X be able to access secret Y?

--

• Answer: if, and only if, node X runs a pod that uses secret Y

• The Node Authorizer implements that kind of logic

• It also allows kubelets to set labels and taints for themselves

  (but not for other nodes)

---

Kubelet and API server

• Communication between kubelet and API server can be established both ways

• Kubelet → API server:

  • kubelet registers itself ("hi, I'm node42, do you have work for me?")

  • connection is kept open and re-established if it breaks

  • that's how the kubelet knows which pods to start/stop

• API server → kubelet:

  • used to retrieve logs, exec, attach to containers

---

Kubelet → API server

• Kubelet is started with --kubeconfig with API server information

• The client certificate of the kubelet will typically have:

  CN=system:node:<nodename> and groups O=system:nodes

• Nothing special on the API server side

  (it will authenticate like any other client)

• Authorization will typically require the Node Authorizer mentioned earlier

⚠️ Kubelet certificates need to be renewed regularly!

• This is typically done through the CSR API

---

API server → kubelet

• Kubelet is started with the flag --client-ca-file

  (typically using the same CA as the API server)

• API server will use a dedicated key pair when contacting kubelet

  (specified with --kubelet-client-certificate and --kubelet-client-key)

• Authorization uses webhooks

  (enabled with --authorization-mode=Webhook on kubelet)

• The webhook server is the API server itself

  (the kubelet sends back a request to the API server to ask, "can this person do that?")

---

Scheduler

• The scheduler connects to the API server like an ordinary client

• The certificate of the scheduler will have CN=system:kube-scheduler

---

Controller manager

• The controller manager is also a normal client to the API server

• Its certificate will have CN=system:kube-controller-manager

• To improve security posture, each controller can use an individual Service Account

• This is enabled with flag --use-service-account-credentials=true

---

Controller manager keys

• The controller can create Secrets holding Service Account tokens

  • this is enabled with flag --service-account-private-key-file

  • this was used in older versions of Kubernetes (before bound tokens)

  • in modern clusters, kubelet uses the TokenRequest API instead

• If we use the CSR API, the controller manager needs the CA cert and key

  • the CSR API is used in many clusters to renew kubelet certificates

  • it's enabled with --cluster-signing-cert-file and --cluster-signing-key-file

---

Service account tokens recap

• These tokens are JWT tokens, signed with a particular key

• These tokens are used for authentication with the API server

  (and therefore, the API server needs to be able to verify their integrity)

• That key is passed to the API server using a couple of flags:

  • --service-account-private-key-file (used to issue tokens)

  • --service-account-key-file (used to verify tokens)

• The private key is also passed to the controller manager
  (using flag --service-account-private-key-file)

---

kube-proxy

• kube-proxy is "yet another API server client"

• In many clusters, it runs as a Daemon Set

• In that case, it will have its own Service Account and associated permissions

• It will authenticate using the token of that Service Account

• It's also possible (but rare) to run it with e.g. static pods

  (it will then require TLS keys; possibly the same as kubelet's!)

---

class: extra-details

Webhooks

• We mentioned webhooks earlier; how does that really work?

• The Kubernetes API has special resource types to check permissions

• One of them is SubjectAccessReview

• To check if a particular user can do a particular action on a particular resource:

  • we prepare a SubjectAccessReview object

  • we send that object to the API server

  • the API server responds with allow/deny (and optional explanations)

• Using webhooks for authorization = sending SAR to authorize each request

---

class: extra-details

Subject Access Review

Here is an example showing how to check if jean.doe can get some pods in kube-system:

kubectl -v9 create -f- <<EOF
apiVersion: authorization.k8s.io/v1
kind: SubjectAccessReview
spec:
  user: jean.doe
  groups:
  - foo
  - bar
  resourceAttributes:
    #group: blah.k8s.io
    namespace: kube-system
    resource: pods
    verb: get
    #name: web-xyz1234567-pqr89
EOF

???

:EN:- Control plane authentication :FR:- Sécurisation du plan de contrôle