github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-24 07:16:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updates

Browse Source
This commit is contained in:
Bridget Kromhout
2019-07-11 13:41:25 -05:00
parent e3c41d9422
commit 7988e86aa2
7 changed files with 1087 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
slides/k8s-201.yml
View File

@@ -21,13 +21,11 @@ chapters:
- shared/about-slides.md
- shared/toc.md
- - k8s/prereqs-k8s201.md
- k8s/localkubeconfig.md
- k8s/localkubeconfig-k8s201.md
- k8s/architecture-k8s201.md
- k8s/setup-managed.md
- - k8s/healthchecks.md
# kubercoins?
- k8s/authn-authz.md
- k8s/podsecuritypolicy.md
- k8s/kubercoins-k8s201.md
- k8s/authn-authz-k8s201.md
- - k8s/resource-limits.md
- k8s/metrics-server.md
- - k8s/cluster-sizing.md

7
slides/k8s/architecture-k8s201.md
View File

@@ -192,7 +192,12 @@ What does that mean?
.exercise[
- Create a namespace with the following command:
- List existing namespaces:
```bash
kubectl get ns
```
- Create a new namespace with the following command:
```bash
kubectl create -f- <<EOF
apiVersion: v1

672
slides/k8s/authn-authz-k8s201.md Normal file
View File

@@ -0,0 +1,672 @@
# Authentication and authorization
*And first, a little refresher!*
- Authentication = verifying the identity of a person
On a UNIX system, we can authenticate with login+password, SSH keys ...
- Authorization = listing what they are allowed to do
On a UNIX system, this can include file permissions, sudoer entries ...
- Sometimes abbreviated as "authn" and "authz"
- In good modular systems, these things are decoupled
(so we can e.g. change a password or SSH key without having to reset access rights)
---
## Authentication in Kubernetes
- When the API server receives a request, it tries to authenticate it
(it examines headers, certificates... anything available)
- Many authentication methods are available and can be used simultaneously
(we will see them on the next slide)
- It's the job of the authentication method to produce:
- the user name
- the user ID
- a list of groups
- The API server doesn't interpret these; that'll be the job of *authorizers*
---
## Authentication methods
- TLS client certificates
(that's what we've been doing with `kubectl` so far)
- Bearer tokens
(a secret token in the HTTP headers of the request)
- [HTTP basic auth](https://en.wikipedia.org/wiki/Basic_access_authentication)
(carrying user and password in an HTTP header)
- Authentication proxy
(sitting in front of the API and setting trusted headers)
---
## Anonymous requests
- If any authentication method *rejects* a request, it's denied
(`401 Unauthorized` HTTP code)
- If a request is neither rejected nor accepted by anyone, it's anonymous
- the user name is `system:anonymous`
- the list of groups is `[system:unauthenticated]`
- By default, the anonymous user can't do anything
(that's what you get if you just `curl` the Kubernetes API)
---
## Authentication with TLS certificates
- This is enabled in most Kubernetes deployments
- The user name is derived from the `CN` in the client certificates
- The groups are derived from the `O` fields in the client certificate
- From the point of view of the Kubernetes API, users do not exist
(i.e. they are not stored in etcd or anywhere else)
- Users can be created (and added to groups) independently of the API
- The Kubernetes API can be set up to use your custom CA to validate client certs
---
class: extra-details
## Viewing our admin certificate
- Let's inspect the certificate we've been using all this time!
.exercise[
- This command will show the `CN` and `O` fields for our certificate:
```bash
kubectl config view \
--raw \
-o json \
| jq -r .users[0].user[\"client-certificate-data\"] \
| openssl base64 -d -A \
| openssl x509 -text \
| grep Subject:
```
]
Let's break down that command together! 😅
---
class: extra-details
## Breaking down the command
- `kubectl config view` shows the Kubernetes user configuration
- `--raw` includes certificate information (which shows as REDACTED otherwise)
- `-o json` outputs the information in JSON format
- `| jq ...` extracts the field with the user certificate (in base64)
- `| openssl base64 -d -A` decodes the base64 format (now we have a PEM file)
- `| openssl x509 -text` parses the certificate and outputs it as plain text
- `| grep Subject:` shows us the line that interests us
→ We are user `kubernetes-admin`, in group `system:masters`.
(We will see later how and why this gives us the permissions that we have.)
---
## User certificates in practice
- The Kubernetes API server does not support certificate revocation
(see issue [#18982](https://github.com/kubernetes/kubernetes/issues/18982))
- As a result, we don't have an easy way to terminate someone's access
(if their key is compromised, or they leave the organization)
- Option 1: re-create a new CA and re-issue everyone's certificates
<br/>
→ Maybe OK if we only have a few users; no way otherwise
- Option 2: don't use groups; grant permissions to individual users
<br/>
→ Inconvenient if we have many users and teams; error-prone
- Option 3: issue short-lived certificates (e.g. 24 hours) and renew them often
<br/>
→ This can be facilitated by e.g. Vault or by the Kubernetes CSR API
---
## Authentication with tokens
- Tokens are passed as HTTP headers:
`Authorization: Bearer and-then-here-comes-the-token`
- Tokens can be validated through a number of different methods:
- static tokens hard-coded in a file on the API server
- [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/) (special case to create a cluster or join nodes)
- [OpenID Connect tokens](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) (to delegate authentication to compatible OAuth2 providers)
- service accounts (these deserve more details, coming right up!)
---
## Service accounts
- A service account is a user that exists in the Kubernetes API
(it is visible with e.g. `kubectl get serviceaccounts`)
- Service accounts can therefore be created / updated dynamically
(they don't require hand-editing a file and restarting the API server)
- A service account is associated with a set of secrets
(the kind that you can view with `kubectl get secrets`)
- Service accounts are generally used to grant permissions to applications, services...
(as opposed to humans)
---
class: extra-details
## Token authentication in practice
- We are going to list existing service accounts
- Then we will extract the token for a given service account
- And we will use that token to authenticate with the API
---
class: extra-details
## Listing service accounts
.exercise[
- The resource name is `serviceaccount` or `sa` for short:
```bash
kubectl get sa
```
]
There should be just one service account in the default namespace: `default`.
---
class: extra-details
## Finding the secret
.exercise[
- List the secrets for the `default` service account:
```bash
kubectl get sa default -o yaml
SECRET=$(kubectl get sa default -o json | jq -r .secrets[0].name)
```
]
It should be named `default-token-XXXXX`.
---
class: extra-details
## Extracting the token
- The token is stored in the secret, wrapped with base64 encoding
.exercise[
- View the secret:
```bash
kubectl get secret $SECRET -o yaml
```
- Extract the token and decode it:
```bash
TOKEN=$(kubectl get secret $SECRET -o json \
| jq -r .data.token | openssl base64 -d -A)
```
]
---
class: extra-details
## Using the token
- Let's send a request to the API, without and with the token
.exercise[
- Find the ClusterIP for the `kubernetes` service:
```bash
kubectl get svc kubernetes
API=$(kubectl get svc kubernetes -o json | jq -r .spec.clusterIP)
```
- Connect without the token:
```bash
curl -k https://$API
```
- Connect with the token:
```bash
curl -k -H "Authorization: Bearer $TOKEN" https://$API
```
]
---
class: extra-details
## Results
- In both cases, we will get a "Forbidden" error
- Without authentication, the user is `system:anonymous`
- With authentication, it is shown as `system:serviceaccount:default:default`
- The API "sees" us as a different user
- But neither user has any rights, so we can't do nothin'
- Let's change that!
---
## Authorization in Kubernetes
- There are multiple ways to grant permissions in Kubernetes, called [authorizers](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#authorization-modules):
- [Node Authorization](https://kubernetes.io/docs/reference/access-authn-authz/node/) (used internally by kubelet; we can ignore it)
- [Attribute-based access control](https://kubernetes.io/docs/reference/access-authn-authz/abac/) (powerful but complex and static; ignore it too)
- [Webhook](https://kubernetes.io/docs/reference/access-authn-authz/webhook/) (each API request is submitted to an external service for approval)
- [Role-based access control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) (associates permissions to users dynamically)
- The one we want is the last one, generally abbreviated as RBAC
---
## Role-based access control
- RBAC allows to specify fine-grained permissions
- Permissions are expressed as *rules*
- A rule is a combination of:
- [verbs](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb) like create, get, list, update, delete...
- resources (as in "API resource," like pods, nodes, services...)
- resource names (to specify e.g. one specific pod instead of all pods)
- in some case, [subresources](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources) (e.g. logs are subresources of pods)
---
## From rules to roles to rolebindings
- A *role* is an API object containing a list of *rules*
Example: role "external-load-balancer-configurator" can:
- [list, get] resources [endpoints, services, pods]
- [update] resources [services]
- A *rolebinding* associates a role with a user
Example: rolebinding "external-load-balancer-configurator":
- associates user "external-load-balancer-configurator"
- with role "external-load-balancer-configurator"
- Yes, there can be users, roles, and rolebindings with the same name
- It's a good idea for 1-1-1 bindings; not so much for 1-N ones
---
## Cluster-scope permissions
- API resources Role and RoleBinding are for objects within a namespace
- We can also define API resources ClusterRole and ClusterRoleBinding
- These are a superset, allowing us to:
- specify actions on cluster-wide objects (like nodes)
- operate across all namespaces
- We can create Role and RoleBinding resources within a namespace
- ClusterRole and ClusterRoleBinding resources are global
---
## Pods and service accounts
- A pod can be associated with a service account
- by default, it is associated with the `default` service account
- as we saw earlier, this service account has no permissions anyway
- The associated token is exposed to the pod's filesystem
(in `/var/run/secrets/kubernetes.io/serviceaccount/token`)
- Standard Kubernetes tooling (like `kubectl`) will look for it there
- So Kubernetes tools running in a pod will automatically use the service account
---
## In practice
- We are going to create a service account
- We will use a default cluster role (`view`)
- We will bind together this role and this service account
- Then we will run a pod using that service account
- In this pod, we will install `kubectl` and check our permissions
---
## Creating a service account
- We will call the new service account `viewer`
(note that nothing prevents us from calling it `view`, like the role)
.exercise[
- Create the new service account:
```bash
kubectl create serviceaccount viewer
```
- List service accounts now:
```bash
kubectl get serviceaccounts
```
]
---
## Binding a role to the service account
- Binding a role = creating a *rolebinding* object
- We will call that object `viewercanview`
(but again, we could call it `view`)
.exercise[
- Create the new role binding:
```bash
kubectl create rolebinding viewercanview \
--clusterrole=view \
--serviceaccount=default:viewer
```
]
It's important to note a couple of details in these flags...
---
## Roles vs Cluster Roles
- We used `--clusterrole=view`
- What would have happened if we had used `--role=view`?
- we would have bound the role `view` from the local namespace
<br/>(instead of the cluster role `view`)
- the command would have worked fine (no error)
- but later, our API requests would have been denied
- This is a deliberate design decision
(we can reference roles that don't exist, and create/update them later)
---
## Users vs Service Accounts
- We used `--serviceaccount=default:viewer`
- What would have happened if we had used `--user=default:viewer`?
- we would have bound the role to a user instead of a service account
- again, the command would have worked fine (no error)
- ...but our API requests would have been denied later
- What's about the `default:` prefix?
- that's the namespace of the service account
- yes, it could be inferred from context, but... `kubectl` requires it
---
## Testing
- We will run an `alpine` pod and install `kubectl` there
.exercise[
- Run a one-time pod:
```bash
kubectl run eyepod --rm -ti --restart=Never \
--serviceaccount=viewer \
--image alpine
```
- Install `curl`, then use it to install `kubectl`:
```bash
apk add --no-cache curl
URLBASE=https://storage.googleapis.com/kubernetes-release/release
KUBEVER=$(curl -s $URLBASE/stable.txt)
curl -LO $URLBASE/$KUBEVER/bin/linux/amd64/kubectl
chmod +x kubectl
```
]
---
## Running `kubectl` in the pod
- We'll try to use our `view` permissions, then to create an object
.exercise[
- Check that we can, indeed, view things:
```bash
./kubectl get all
```
- But that we can't create things:
```
./kubectl create deployment testrbac --image=nginx
```
- Exit the container with `exit` or `^D`
<!-- ```keys ^D``` -->
]
- We will see the pod has terminated with an error
---
## Testing directly with `kubectl`
- We can also check for permission with `kubectl auth can-i`:
```bash
kubectl auth can-i list nodes
kubectl auth can-i create pods
kubectl auth can-i get pod/name-of-pod
kubectl auth can-i get /url-fragment-of-api-request/
kubectl auth can-i '*' services
```
- And we can check permissions on behalf of other users:
```bash
kubectl auth can-i list nodes \
--as some-user
kubectl auth can-i list nodes \
--as system:serviceaccount:<namespace>:<name-of-service-account>
```
---
class: extra-details
## Where does this `view` role come from?
- Kubernetes defines a number of ClusterRoles intended to be bound to users
- `cluster-admin` can do *everything* (think `root` on UNIX)
- `admin` can do *almost everything* (except e.g. changing resource quotas and limits)
- `edit` is similar to `admin`, but cannot view or edit permissions
- `view` has read-only access to most resources, except permissions and secrets
*In many situations, these roles will be all you need.*
*You can also customize them!*
---
class: extra-details
## Customizing the default roles
- If you need to *add* permissions to these default roles (or others),
<br/>
you can do it through the [ClusterRole Aggregation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) mechanism
- This happens by creating a ClusterRole with the following labels:
```yaml
metadata:
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
```
- This ClusterRole permissions will be added to `admin`/`edit`/`view` respectively
- This is particulary useful when using CustomResourceDefinitions
(since Kubernetes cannot guess which resources are sensitive and which ones aren't)
---
class: extra-details
## Where do our permissions come from?
- When interacting with the Kubernetes API, we are using a client certificate
- We saw previously that this client certificate contained:
`CN=kubernetes-admin` and `O=system:masters`
- Let's look for these in existing ClusterRoleBindings:
```bash
kubectl get clusterrolebindings -o yaml |
grep -e kubernetes-admin -e system:masters
```
(`system:masters` should show up, but not `kubernetes-admin`.)
- Where does this match come from?
---
class: extra-details
## The `system:masters` group
- If we eyeball the output of `kubectl get clusterrolebindings -o yaml`, we'll find out!
- It is in the `cluster-admin` binding:
```bash
kubectl describe clusterrolebinding cluster-admin
```
- This binding associates `system:masters` with the cluster role `cluster-admin`
- And the `cluster-admin` is, basically, `root`:
```bash
kubectl describe clusterrole cluster-admin
```
---
class: extra-details
# Pod Security Policies
- If you'd like to check out pod-level controls in AKS, they are [available in preview](https://docs.microsoft.com/en-us/azure/aks/use-pod-security-policies)
- Experiment, but not in production!

2
slides/k8s/healthchecks.md
View File

@@ -120,7 +120,7 @@
## Example: HTTP probe
Here is a pod template for the `rng` web service of the DockerCoins app:
Here is a pod template for the `rng` web service of our DockerCoins sample app:
```yaml
apiVersion: v1

221
slides/k8s/kubercoins-k8s201.md Normal file
View File

@@ -0,0 +1,221 @@
# Deploying a sample application
- We will connect to our new Kubernetes cluster
- We will deploy a sample application, "DockerCoins"
- That app features multiple micro-services and a web UI
---
## Cloning some repos
- We will need two repositories:
- the first one has the "DockerCoins" demo app
- the second one has these slides, some scripts, more manifests ...
.exercise[
- Clone the kubercoins repository locally:
```bash
git clone https://github.com/jpetazzo/kubercoins
```
- Clone the container.training repository as well:
```bash
git clone https://@@GITREPO@@
```
]
---
## Running the application
Without further ado, let's start this application!
.exercise[
- Apply all the manifests from the kubercoins repository:
```bash
kubectl apply -f kubercoins/
```
]
---
## What's this application?
--
- It is a DockerCoin miner! .emoji[💰🐳📦🚢]
--
- No, you can't buy coffee with DockerCoins
--
- How DockerCoins works:
- generate a few random bytes
- hash these bytes
- increment a counter (to keep track of speed)
- repeat forever!
--
- DockerCoins is *not* a cryptocurrency
(the only common points are "randomness", "hashing", and "coins" in the name)
---
## DockerCoins in the microservices era
- DockerCoins is made of 5 services:
- `rng` = web service generating random bytes
- `hasher` = web service computing hash of POSTed data
- `worker` = background process calling `rng` and `hasher`
- `webui` = web interface to watch progress
- `redis` = data store (holds a counter updated by `worker`)
- These 5 services are visible in the application's Compose file,
[docker-compose.yml](
https://@@GITREPO@@/blob/master/dockercoins/docker-compose.yml)
---
## How DockerCoins works
- `worker` invokes web service `rng` to generate random bytes
- `worker` invokes web service `hasher` to hash these bytes
- `worker` does this in an infinite loop
- every second, `worker` updates `redis` to indicate how many loops were done
- `webui` queries `redis`, and computes and exposes "hashing speed" in our browser
*(See diagram on next slide!)*
---
class: pic
![Diagram showing the 5 containers of the applications](images/dockercoins-diagram.svg)
---
## Service discovery in container-land
How does each service find out the address of the other ones?
--
- We do not hard-code IP addresses in the code
- We do not hard-code FQDNs in the code, either
- We just connect to a service name, and container-magic does the rest
(And by container-magic, we mean "a crafty, dynamic, embedded DNS server")
---
## Example in `worker/worker.py`
```python
redis = Redis("`redis`")
def get_random_bytes():
r = requests.get("http://`rng`/32")
return r.content
def hash_bytes(data):
r = requests.post("http://`hasher`/",
data=data,
headers={"Content-Type": "application/octet-stream"})
```
(Full source code available [here](
https://@@GITREPO@@/blob/8279a3bce9398f7c1a53bdd95187c53eda4e6435/dockercoins/worker/worker.py#L17
))
---
## Show me the code!
- You can check the GitHub repository with all the materials of this workshop:
<br/>https://@@GITREPO@@
- The application is in the [dockercoins](
https://@@GITREPO@@/tree/master/dockercoins)
subdirectory
- The Compose file ([docker-compose.yml](
https://@@GITREPO@@/blob/master/dockercoins/docker-compose.yml))
lists all 5 services
- `redis` is using an official image from the Docker Hub
- `hasher`, `rng`, `worker`, `webui` are each built from a Dockerfile
- Each service's Dockerfile and source code is in its own directory
(`hasher` is in the [hasher](https://@@GITREPO@@/blob/master/dockercoins/hasher/) directory,
`rng` is in the [rng](https://@@GITREPO@@/blob/master/dockercoins/rng/)
directory, etc.)
---
## Our application at work
- We can check the logs of our application's pods
.exercise[
- Check the logs of the various components:
```bash
kubectl logs deploy/worker
kubectl logs deploy/hasher
```
]
---
## Connecting to the web UI
- "Logs are exciting and fun!" (No-one, ever)
- The `webui` container exposes a web dashboard; let's view it
.exercise[
- Check the NodePort allocated to the web UI:
```bash
kubectl get svc webui
```
- Open that in a web browser
]
A drawing area should show up, and after a few seconds, a blue
graph will appear.

181
slides/k8s/localkubeconfig-k8s201.md Normal file
View File

@@ -0,0 +1,181 @@
# Controlling a Kubernetes cluster remotely
- `kubectl` can be used either on cluster instances or outside the cluster
- Here, we are going to use `kubectl` from our local machine
---
## Requirements
.warning[The exercises in this chapter should be done *on your local machine*.]
- `kubectl` is officially available on Linux, macOS, Windows
(and unofficially anywhere we can build and run Go binaries)
- You may want to try Azure cloud shell if you are following along from:
- a tablet or phone
- a web-based terminal
- an environment where you can't install and run new binaries
---
## Installing `kubectl`
- If you already have `kubectl` on your local machine, you can skip this
.exercise[
<!-- ##VERSION## -->
- Download the `kubectl` binary from one of these links:
[Linux](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl)
|
[macOS](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/darwin/amd64/kubectl)
|
[Windows](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/windows/amd64/kubectl.exe)
- On Linux and macOS, make the binary executable with `chmod +x kubectl`
(And remember to run it with `./kubectl` or move it to your `$PATH`)
]
Note: if you are following along with a different platform (e.g. Linux on an architecture different from amd64, or with a phone or tablet), installing `kubectl` might be more complicated (or even impossible) so check with us about cloud shell.
---
## Testing `kubectl`
- Check that `kubectl` works correctly
(before even trying to connect to a remote cluster!)
.exercise[
- Ask `kubectl` to show its version number:
```bash
kubectl version --client
```
]
The output should look like this:
```
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0",
GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean",
BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc",
Platform:"darwin/amd64"}
```
---
## Preserving the existing `~/.kube/config`
- If you already have a `~/.kube/config` file, rename it
(we are going to overwrite it in the following slides!)
- If you never used `kubectl` on your machine before: nothing to do!
.exercise[
- Make a copy of `~/.kube/config`; if you are using macOS or Linux, you can do:
```bash
cp ~/.kube/config ~/.kube/config.before.training
```
- If you are using Windows, you will need to adapt this command
]
---
## Connecting to your AKS cluster
[fill in]
---
## Let's look at your cluster!
- First, inspect the config
```bash
kubectl config view
```
- Look for the `server:` address that matches your new cluster
```
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://aks-test-c-aks-test-group-0d35f7-28c7d691.hcp.eastus.azmk8s.io:443
name: aks-test-cluster
```
---
class: extra-details
## What if we get a certificate error?
- Generally, the Kubernetes API uses a certificate that is valid for:
- `kubernetes`
- `kubernetes.default`
- `kubernetes.default.svc`
- `kubernetes.default.svc.cluster.local`
- the ClusterIP address of the `kubernetes` service
- the hostname of the node hosting the control plane
- the IP address of the node hosting the control plane
- On most clouds, the IP address of the node is an internal IP address
- ... And we are going to connect over the external IP address
- ... And that external IP address was not used when creating the certificate!
---
class: extra-details
## Working around the certificate error
- We need to tell `kubectl` to skip TLS verification
(only do this with testing clusters, never in production!)
- The following command will do the trick:
```bash
kubectl config set-cluster <clustername> --insecure-skip-tls-verify
```
---
## Checking that we can connect to the cluster
- We can now run a couple of trivial commands to check that all is well
.exercise[
- Check the versions of the local client and remote server:
```bash
kubectl version
```
It is okay if you have a newer client than what is available on the server.
- View the nodes of the cluster:
```bash
kubectl get nodes
```
]
We can now utilize the cluster exactly as if we're logged into a node, except that it's remote.

4
slides/k8s/prereqs-k8s201.md
View File

@@ -44,7 +44,9 @@
- We will connect to these clusters with `kubectl`
(if you don't have `kubectl` installed, install it **now!**)
- If you don't have `kubectl` installed, we'll explain how to install it shortly
- You will also want to install `jq`
---