container.training/slides/k8s/bootstrap.md

TLS bootstrap

• kubelet needs TLS keys and certificates to communicate with the control plane

• How do we generate this information?

• How do we make it available to kubelet?

---

Option 1: push

• When we want to provision a node:

  • generate its keys, certificate, and sign centrally

  • push the files to the node

• OK for "traditional", on-premises deployments

• Not OK for cloud deployments with auto-scaling

---

Option 2: poll + push

• Discover nodes when they are created

  (e.g. with cloud API)

• When we detect a new node, push TLS material to the node

  (like in option 1)

• It works, but:

  • discovery code is specific to each provider

  • relies heavily on the cloud provider API

  • doesn't work on-premises

  • doesn't scale

---

Option 3: bootstrap tokens + CSR API

• Since Kubernetes 1.4, the Kubernetes API supports CSR

  (Certificate Signing Requests)

• This is similar to the protocol used to obtain e.g. HTTPS certificates:

  • subject (here, kubelet) generates TLS keys and CSR

  • subject submits CSR to CA

  • CA validates (or not) the CSR

  • CA sends back signed certificate to subject

• This is combined with bootstrap tokens

---

Bootstrap tokens

• A bootstrap token is an API access token

  • it is a Secret with type bootstrap.kubernetes.io/token

  • it is 6 public characters (ID) + 16 secret characters
    (example: whd3pq.d1ushuf6ccisjacu)

  • it gives access to groups system:bootstrap:<ID> and system:bootstrappers

  • additional groups can be specified in the Secret

---

Bootstrap tokens with kubeadm

• kubeadm automatically creates a bootstrap token

  (it is shown at the end of kubeadm init)

• That token adds the group system:bootstrappers:kubeadm:default-node-token

• kubeadm also creates a ClusterRoleBinding kubeadm:kubelet-bootstrap
  binding ...:default-node-token to ClusterRole system:node-bootstrapper

• That ClusterRole gives create/get/list/watch permissions on the CSR API

---

Bootstrap tokens in practice

• Let's list our bootstrap tokens on a cluster created with kubeadm

.exercise[

• Log into node test1

• View bootstrap tokens:

  sudo kubeadm token list
  

]

• Tokens are short-lived

• We can create new tokens with kubeadm if necessary

---

class: extra-details

Retrieving bootstrap tokens with kubectl

• Bootstrap tokens are Secrets with type bootstrap.kubernetes.io/token

• Token ID and secret are in data fields token-id and token-secret

• In Secrets, data fields are encoded with Base64

• This "very simple" command will show us the tokens:

kubectl -n kube-system get secrets -o json | 
        jq -r '.items[] 
        | select(.type=="bootstrap.kubernetes.io/token")
        | ( .data["token-id"] + "Lg==" + .data["token-secret"] + "Cg==")
        ' | base64 -d

(On recent versions of jq, you can simplify by using filter @base64d.)

---

class: extra-details

Using a bootstrap token

• The token we need to use has the form abcdef.1234567890abcdef

.exercise[

• Check that it is accepted by the API server:

  curl -k -H "Authorization: Bearer abcdef.1234567890abcdef"
  

• We should see that we are authenticated but not authorized:

  User \"system:bootstrap:abcdef\" cannot get path \"/\""
  

• Check that we can access the CSR API:

  curl -k -H "Authorization: Bearer abcdef.1234567890abcdef" \
       https://10.96.0.1/apis/certificates.k8s.io/v1beta1/certificatesigningrequests
  

]

---

The cluster-info ConfigMap

• Before we can talk to the API, we need:

  • the API server address (obviously!)

  • the cluster CA certificate

• That information is stored in a public ConfigMap

.exercise[

• Retrieve that ConfigMap:

  curl -k https://10.96.0.1/api/v1/namespaces/kube-public/configmaps/cluster-info
  

]

Extracting the kubeconfig file is left as an exercise for the reader.

---

class: extra-details

Signature of the config-map

• You might have noticed a few jws-kubeconfig-... fields

• These are config-map signatures

  (so that the client can protect against MITM attacks)

• These are JWS signatures using HMAC-SHA256

  (see here for more details)

---

Putting it all together

This is the TLS bootstrap mechanism, step by step.

• The node uses the cluster-info ConfigMap to get the cluster CA certificate

• The node generates its keys and CSR

• Using the bootstrap token, the node creates a CertificateSigningRequest object

• The node watches the CSR object

• The CSR object is accepted (automatically or by an admin)

• The node gets notified, and retrieves the certificate

• The node can now join the cluster

---

Bottom line

• If you paid attention, we still need a way to:

  • either safely get the bootstrap token to the nodes

  • or disable auto-approval and manually approve the nodes when they join

• The goal of the TLS bootstrap mechanism is not to solve this

  (in terms of information knowledge, it's fundamentally impossible!)

• But it reduces the differences between environments, infrastructures, providers ...

• It gives a mechanism that is easier to use, and flexible enough, for most scenarios

---

More information

• As always, the Kubernetes documentation has extra details:

  • TLS management

  • Authenticating with bootstrap tokens

  • TLS bootstrapping

  • kubeadm token command

  • kubeadm join command (has details about the join workflow)