mirror of
https://github.com/jpetazzo/container.training.git
synced 2026-08-18 03:26:23 +00:00
Jérôme Petazzoni
b56e54eaec
Now that we have a good number of longer exercises, it makes sense to rename the shorter demos/labs into 'labs' to avoid confusion between the two.
339 lines
7.2 KiB
Markdown
339 lines
7.2 KiB
Markdown
# Sealed Secrets
|
|
|
|
- Kubernetes provides the "Secret" resource to store credentials, keys, passwords ...
|
|
|
|
- Secrets can be protected with RBAC
|
|
|
|
(e.g. "you can write secrets, but only the app's service account can read them")
|
|
|
|
- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) is an operator that lets us store secrets in code repositories
|
|
|
|
- It uses asymetric cryptography:
|
|
|
|
- anyone can *encrypt* a secret
|
|
|
|
- only the cluster can *decrypt* a secret
|
|
|
|
---
|
|
|
|
## Principle
|
|
|
|
- The Sealed Secrets operator uses a *public* and a *private* key
|
|
|
|
- The public key is available publicly (duh!)
|
|
|
|
- We use the public key to encrypt secrets into a SealedSecret resource
|
|
|
|
- the SealedSecret resource can be stored in a code repo (even a public one)
|
|
|
|
- The SealedSecret resource is `kubectl apply`'d to the cluster
|
|
|
|
- The Sealed Secrets controller decrypts the SealedSecret with the private key
|
|
|
|
(this creates a classic Secret resource)
|
|
|
|
- Nobody else can decrypt secrets, since only the controller has the private key
|
|
|
|
---
|
|
|
|
## In action
|
|
|
|
- We will install the Sealed Secrets operator
|
|
|
|
- We will generate a Secret
|
|
|
|
- We will "seal" that Secret (generate a SealedSecret)
|
|
|
|
- We will load that SealedSecret on the cluster
|
|
|
|
- We will check that we now have a Secret
|
|
|
|
---
|
|
|
|
## Installing the operator
|
|
|
|
- The official installation is done through a single YAML file
|
|
|
|
- There is also a Helm chart if you prefer that
|
|
|
|
(if you're using Kubernetes 1.22+, see next slide!)
|
|
|
|
<!-- #VERSION# -->
|
|
|
|
.lab[
|
|
|
|
- Install the operator:
|
|
.small[
|
|
```bash
|
|
kubectl apply -f \
|
|
https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
|
|
```
|
|
]
|
|
|
|
]
|
|
|
|
Note: it installs into `kube-system` by default.
|
|
|
|
If you change that, you will also need to inform `kubeseal` later on.
|
|
|
|
---
|
|
|
|
class: extra-details
|
|
|
|
## Sealed Secrets on Kubernetes 1.22
|
|
|
|
- As of version 0.16, Sealed Secrets manifests uses RBAC v1beta1
|
|
|
|
- RBAC v1beta1 isn't supported anymore in Kubernetes 1.22
|
|
|
|
- Sealed Secerets Helm chart provides manifests using RBAC v1
|
|
|
|
- Conclusion: to install Sealed Secrets on Kubernetes 1.22, use the Helm chart:
|
|
|
|
```bash
|
|
helm install --repo https://bitnami-labs.github.io/sealed-secrets/ \
|
|
sealed-secrets-controller sealed-secrets --namespace kube-system
|
|
```
|
|
|
|
- Make sure to install in the `kube-system` Namespace
|
|
|
|
- Make sure that the release is named `sealed-secrets-controller`
|
|
|
|
(or pass a `--controller-name` option to `kubeseal` later)
|
|
|
|
---
|
|
|
|
## Creating a Secret
|
|
|
|
- Let's create a normal (unencrypted) secret
|
|
|
|
.lab[
|
|
|
|
- Create a Secret with a couple of API tokens:
|
|
```bash
|
|
kubectl create secret generic awskey \
|
|
--from-literal=AWS_ACCESS_KEY_ID=AKI... \
|
|
--from-literal=AWS_SECRET_ACCESS_KEY=abc123xyz... \
|
|
--dry-run=client -o yaml > secret-aws.yaml
|
|
```
|
|
|
|
]
|
|
|
|
- Note the `--dry-run` and `-o yaml`
|
|
|
|
(we're just generating YAML, not sending the secrets to our Kubernetes cluster)
|
|
|
|
- We could also write the YAML from scratch or generate it with other tools
|
|
|
|
---
|
|
|
|
## Creating a Sealed Secret
|
|
|
|
- This is done with the `kubeseal` tool
|
|
|
|
- It will obtain the public key from the cluster
|
|
|
|
.lab[
|
|
|
|
- Create the Sealed Secret:
|
|
```bash
|
|
kubeseal < secret-aws.yaml > sealed-secret-aws.json
|
|
```
|
|
|
|
]
|
|
|
|
- The file `sealed-secret-aws.json` can be committed to your public repo
|
|
|
|
(if you prefer YAML output, you can add `-o yaml`)
|
|
|
|
---
|
|
|
|
## Using a Sealed Secret
|
|
|
|
- Now let's `kubectl apply` that Sealed Secret to the cluster
|
|
|
|
- The Sealed Secret controller will "unseal" it for us
|
|
|
|
.lab[
|
|
|
|
- Check that our Secret doesn't exist (yet):
|
|
```bash
|
|
kubectl get secrets
|
|
```
|
|
|
|
- Load the Sealed Secret into the cluster:
|
|
```bash
|
|
kubectl create -f sealed-secret-aws.json
|
|
```
|
|
|
|
- Check that the secret is now available:
|
|
```bash
|
|
kubectl get secrets
|
|
```
|
|
|
|
]
|
|
|
|
---
|
|
|
|
## Tweaking secrets
|
|
|
|
- Let's see what happens if we try to rename the Secret
|
|
|
|
(or use it in a different namespace)
|
|
|
|
.lab[
|
|
|
|
- Delete both the Secret and the SealedSecret
|
|
|
|
- Edit `sealed-secret-aws.json`
|
|
|
|
- Change the name of the secret, or its namespace
|
|
|
|
(both in the SealedSecret metadata and in the Secret template)
|
|
|
|
- `kubectl apply -f` the new JSON file and observe the results 🤔
|
|
|
|
]
|
|
|
|
---
|
|
|
|
## Sealed Secrets are *scoped*
|
|
|
|
- A SealedSecret cannot be renamed or moved to another namespace
|
|
|
|
(at least, not by default!)
|
|
|
|
- Otherwise, it would allow to evade RBAC rules:
|
|
|
|
- if I can view Secrets in namespace `myapp` but not in namespace `yourapp`
|
|
|
|
- I could take a SealedSecret belonging to namespace `yourapp`
|
|
|
|
- ... and deploy it in `myapp`
|
|
|
|
- ... and view the resulting decrypted Secret!
|
|
|
|
- This can be changed with `--scope namespace-wide` or `--scope cluster-wide`
|
|
|
|
---
|
|
|
|
## Working offline
|
|
|
|
- We can obtain the public key from the server
|
|
|
|
(technically, as a PEM certificate)
|
|
|
|
- Then we can use that public key offline
|
|
|
|
(without contacting the server)
|
|
|
|
- Relevant commands:
|
|
|
|
`kubeseal --fetch-cert > seal.pem`
|
|
|
|
`kubeseal --cert seal.pem < secret.yaml > sealedsecret.json`
|
|
|
|
---
|
|
|
|
## Key rotation
|
|
|
|
- The controller generate new keys every month by default
|
|
|
|
- The keys are kept as TLS Secrets in the `kube-system` namespace
|
|
|
|
(named `sealed-secrets-keyXXXXX`)
|
|
|
|
- When keys are "rotated", old decryption keys are kept
|
|
|
|
(otherwise we can't decrypt previously-generated SealedSecrets)
|
|
|
|
---
|
|
|
|
## Key compromise
|
|
|
|
- If the *sealing* key (obtained with `--fetch-cert` is compromised):
|
|
|
|
*we don't need to do anything (it's a public key!)*
|
|
|
|
- However, if the *unsealing* key (the TLS secret in `kube-system`) is compromised ...
|
|
|
|
*we need to:*
|
|
|
|
- rotate the key
|
|
|
|
- rotate the SealedSecrets that were encrypted with that key
|
|
<br/>
|
|
(as they are compromised)
|
|
|
|
---
|
|
|
|
## Rotating the key
|
|
|
|
- By default, new keys are generated every 30 days
|
|
|
|
- To force the generation of a new key "right now":
|
|
|
|
- obtain an RFC1123 timestamp with `date -R`
|
|
|
|
- edit Deployment `sealed-secrets-controller` (in `kube-system`)
|
|
|
|
- add `--key-cutoff-time=TIMESTAMP` to the command-line
|
|
|
|
- *Then*, rotate the SealedSecrets that were encrypted with it
|
|
|
|
(generate new Secrets, then encrypt them with the new key)
|
|
|
|
---
|
|
|
|
## Discussion (the good)
|
|
|
|
- The footprint of the operator is rather small:
|
|
|
|
- only one CRD
|
|
|
|
- one Deployment, one Service
|
|
|
|
- a few RBAC-related objects
|
|
|
|
---
|
|
|
|
## Discussion (the less good)
|
|
|
|
- Events could be improved
|
|
|
|
- `no key to decrypt secret` when there is a name/namespace mismatch
|
|
|
|
- no event indicating that a SealedSecret was successfully unsealed
|
|
|
|
- Key rotation could be improved (how to find secrets corresponding to a key?)
|
|
|
|
- If the sealing keys are lost, it's impossible to unseal the SealedSecrets
|
|
|
|
(e.g. cluster reinstall)
|
|
|
|
- ... Which means that we need to back up the sealing keys
|
|
|
|
- ... Which means that we need to be super careful with these backups!
|
|
|
|
---
|
|
|
|
## Other approaches
|
|
|
|
- [Kamus](https://kamus.soluto.io/) ([git](https://github.com/Soluto/kamus)) offers "zero-trust" secrets
|
|
|
|
(the cluster cannot decrypt secrets; only the application can decrypt them)
|
|
|
|
- [Vault](https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar?in=vault/kubernetes) can do ... a lot
|
|
|
|
- dynamic secrets (generated on the fly for a consumer)
|
|
|
|
- certificate management
|
|
|
|
- integration outside of Kubernetes
|
|
|
|
- and much more!
|
|
|
|
???
|
|
|
|
:EN:- The Sealed Secrets Operator
|
|
:FR:- L'opérateur *Sealed Secrets* |