diff --git a/slides/k8s-201.yml b/slides/k8s-201.yml index 6c065d5f..c6566c69 100644 --- a/slides/k8s-201.yml +++ b/slides/k8s-201.yml @@ -21,13 +21,11 @@ chapters: - shared/about-slides.md - shared/toc.md - - k8s/prereqs-k8s201.md - - k8s/localkubeconfig.md + - k8s/localkubeconfig-k8s201.md - k8s/architecture-k8s201.md - - k8s/setup-managed.md - - k8s/healthchecks.md -# kubercoins? - - k8s/authn-authz.md - - k8s/podsecuritypolicy.md + - k8s/kubercoins-k8s201.md + - k8s/authn-authz-k8s201.md - - k8s/resource-limits.md - k8s/metrics-server.md - - k8s/cluster-sizing.md diff --git a/slides/k8s/architecture-k8s201.md b/slides/k8s/architecture-k8s201.md index db0b932a..06ce1d89 100644 --- a/slides/k8s/architecture-k8s201.md +++ b/slides/k8s/architecture-k8s201.md @@ -192,7 +192,12 @@ What does that mean? .exercise[ -- Create a namespace with the following command: +- List existing namespaces: + ```bash + kubectl get ns + ``` + +- Create a new namespace with the following command: ```bash kubectl create -f- < + → Maybe OK if we only have a few users; no way otherwise + +- Option 2: don't use groups; grant permissions to individual users +
+ → Inconvenient if we have many users and teams; error-prone + +- Option 3: issue short-lived certificates (e.g. 24 hours) and renew them often +
+ → This can be facilitated by e.g. Vault or by the Kubernetes CSR API + +--- + +## Authentication with tokens + +- Tokens are passed as HTTP headers: + + `Authorization: Bearer and-then-here-comes-the-token` + +- Tokens can be validated through a number of different methods: + + - static tokens hard-coded in a file on the API server + + - [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/) (special case to create a cluster or join nodes) + + - [OpenID Connect tokens](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) (to delegate authentication to compatible OAuth2 providers) + + - service accounts (these deserve more details, coming right up!) + +--- + +## Service accounts + +- A service account is a user that exists in the Kubernetes API + + (it is visible with e.g. `kubectl get serviceaccounts`) + +- Service accounts can therefore be created / updated dynamically + + (they don't require hand-editing a file and restarting the API server) + +- A service account is associated with a set of secrets + + (the kind that you can view with `kubectl get secrets`) + +- Service accounts are generally used to grant permissions to applications, services... + + (as opposed to humans) + +--- + +class: extra-details + +## Token authentication in practice + +- We are going to list existing service accounts + +- Then we will extract the token for a given service account + +- And we will use that token to authenticate with the API + +--- + +class: extra-details + +## Listing service accounts + +.exercise[ + +- The resource name is `serviceaccount` or `sa` for short: + ```bash + kubectl get sa + ``` + +] + +There should be just one service account in the default namespace: `default`. + +--- + +class: extra-details + +## Finding the secret + +.exercise[ + +- List the secrets for the `default` service account: + ```bash + kubectl get sa default -o yaml + SECRET=$(kubectl get sa default -o json | jq -r .secrets[0].name) + ``` + +] + +It should be named `default-token-XXXXX`. + +--- + +class: extra-details + +## Extracting the token + +- The token is stored in the secret, wrapped with base64 encoding + +.exercise[ + +- View the secret: + ```bash + kubectl get secret $SECRET -o yaml + ``` + +- Extract the token and decode it: + ```bash + TOKEN=$(kubectl get secret $SECRET -o json \ + | jq -r .data.token | openssl base64 -d -A) + ``` + +] + +--- + +class: extra-details + +## Using the token + +- Let's send a request to the API, without and with the token + +.exercise[ + +- Find the ClusterIP for the `kubernetes` service: + ```bash + kubectl get svc kubernetes + API=$(kubectl get svc kubernetes -o json | jq -r .spec.clusterIP) + ``` + +- Connect without the token: + ```bash + curl -k https://$API + ``` + +- Connect with the token: + ```bash + curl -k -H "Authorization: Bearer $TOKEN" https://$API + ``` + +] + +--- + +class: extra-details + +## Results + +- In both cases, we will get a "Forbidden" error + +- Without authentication, the user is `system:anonymous` + +- With authentication, it is shown as `system:serviceaccount:default:default` + +- The API "sees" us as a different user + +- But neither user has any rights, so we can't do nothin' + +- Let's change that! + +--- + +## Authorization in Kubernetes + +- There are multiple ways to grant permissions in Kubernetes, called [authorizers](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#authorization-modules): + + - [Node Authorization](https://kubernetes.io/docs/reference/access-authn-authz/node/) (used internally by kubelet; we can ignore it) + + - [Attribute-based access control](https://kubernetes.io/docs/reference/access-authn-authz/abac/) (powerful but complex and static; ignore it too) + + - [Webhook](https://kubernetes.io/docs/reference/access-authn-authz/webhook/) (each API request is submitted to an external service for approval) + + - [Role-based access control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) (associates permissions to users dynamically) + +- The one we want is the last one, generally abbreviated as RBAC + +--- + +## Role-based access control + +- RBAC allows to specify fine-grained permissions + +- Permissions are expressed as *rules* + +- A rule is a combination of: + + - [verbs](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb) like create, get, list, update, delete... + + - resources (as in "API resource," like pods, nodes, services...) + + - resource names (to specify e.g. one specific pod instead of all pods) + + - in some case, [subresources](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources) (e.g. logs are subresources of pods) + +--- + +## From rules to roles to rolebindings + +- A *role* is an API object containing a list of *rules* + + Example: role "external-load-balancer-configurator" can: + - [list, get] resources [endpoints, services, pods] + - [update] resources [services] + +- A *rolebinding* associates a role with a user + + Example: rolebinding "external-load-balancer-configurator": + - associates user "external-load-balancer-configurator" + - with role "external-load-balancer-configurator" + +- Yes, there can be users, roles, and rolebindings with the same name + +- It's a good idea for 1-1-1 bindings; not so much for 1-N ones + +--- + +## Cluster-scope permissions + +- API resources Role and RoleBinding are for objects within a namespace + +- We can also define API resources ClusterRole and ClusterRoleBinding + +- These are a superset, allowing us to: + + - specify actions on cluster-wide objects (like nodes) + + - operate across all namespaces + +- We can create Role and RoleBinding resources within a namespace + +- ClusterRole and ClusterRoleBinding resources are global + +--- + +## Pods and service accounts + +- A pod can be associated with a service account + + - by default, it is associated with the `default` service account + + - as we saw earlier, this service account has no permissions anyway + +- The associated token is exposed to the pod's filesystem + + (in `/var/run/secrets/kubernetes.io/serviceaccount/token`) + +- Standard Kubernetes tooling (like `kubectl`) will look for it there + +- So Kubernetes tools running in a pod will automatically use the service account + +--- + +## In practice + +- We are going to create a service account + +- We will use a default cluster role (`view`) + +- We will bind together this role and this service account + +- Then we will run a pod using that service account + +- In this pod, we will install `kubectl` and check our permissions + +--- + +## Creating a service account + +- We will call the new service account `viewer` + + (note that nothing prevents us from calling it `view`, like the role) + +.exercise[ + +- Create the new service account: + ```bash + kubectl create serviceaccount viewer + ``` + +- List service accounts now: + ```bash + kubectl get serviceaccounts + ``` + +] + +--- + +## Binding a role to the service account + +- Binding a role = creating a *rolebinding* object + +- We will call that object `viewercanview` + + (but again, we could call it `view`) + +.exercise[ + +- Create the new role binding: + ```bash + kubectl create rolebinding viewercanview \ + --clusterrole=view \ + --serviceaccount=default:viewer + ``` + +] + +It's important to note a couple of details in these flags... + +--- + +## Roles vs Cluster Roles + +- We used `--clusterrole=view` + +- What would have happened if we had used `--role=view`? + + - we would have bound the role `view` from the local namespace +
(instead of the cluster role `view`) + + - the command would have worked fine (no error) + + - but later, our API requests would have been denied + +- This is a deliberate design decision + + (we can reference roles that don't exist, and create/update them later) + +--- + +## Users vs Service Accounts + +- We used `--serviceaccount=default:viewer` + +- What would have happened if we had used `--user=default:viewer`? + + - we would have bound the role to a user instead of a service account + + - again, the command would have worked fine (no error) + + - ...but our API requests would have been denied later + +- What's about the `default:` prefix? + + - that's the namespace of the service account + + - yes, it could be inferred from context, but... `kubectl` requires it + +--- + +## Testing + +- We will run an `alpine` pod and install `kubectl` there + +.exercise[ + +- Run a one-time pod: + ```bash + kubectl run eyepod --rm -ti --restart=Never \ + --serviceaccount=viewer \ + --image alpine + ``` + +- Install `curl`, then use it to install `kubectl`: + ```bash + apk add --no-cache curl + URLBASE=https://storage.googleapis.com/kubernetes-release/release + KUBEVER=$(curl -s $URLBASE/stable.txt) + curl -LO $URLBASE/$KUBEVER/bin/linux/amd64/kubectl + chmod +x kubectl + ``` + +] + +--- + +## Running `kubectl` in the pod + +- We'll try to use our `view` permissions, then to create an object + +.exercise[ + +- Check that we can, indeed, view things: + ```bash + ./kubectl get all + ``` + +- But that we can't create things: + ``` + ./kubectl create deployment testrbac --image=nginx + ``` + +- Exit the container with `exit` or `^D` + + + +] + +- We will see the pod has terminated with an error + +--- + +## Testing directly with `kubectl` + +- We can also check for permission with `kubectl auth can-i`: + ```bash + kubectl auth can-i list nodes + kubectl auth can-i create pods + kubectl auth can-i get pod/name-of-pod + kubectl auth can-i get /url-fragment-of-api-request/ + kubectl auth can-i '*' services + ``` + +- And we can check permissions on behalf of other users: + ```bash + kubectl auth can-i list nodes \ + --as some-user + kubectl auth can-i list nodes \ + --as system:serviceaccount:: + ``` + +--- + +class: extra-details + +## Where does this `view` role come from? + +- Kubernetes defines a number of ClusterRoles intended to be bound to users + +- `cluster-admin` can do *everything* (think `root` on UNIX) + +- `admin` can do *almost everything* (except e.g. changing resource quotas and limits) + +- `edit` is similar to `admin`, but cannot view or edit permissions + +- `view` has read-only access to most resources, except permissions and secrets + +*In many situations, these roles will be all you need.* + +*You can also customize them!* + +--- + +class: extra-details + +## Customizing the default roles + +- If you need to *add* permissions to these default roles (or others), +
+ you can do it through the [ClusterRole Aggregation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) mechanism + +- This happens by creating a ClusterRole with the following labels: + ```yaml + metadata: + labels: + rbac.authorization.k8s.io/aggregate-to-admin: "true" + rbac.authorization.k8s.io/aggregate-to-edit: "true" + rbac.authorization.k8s.io/aggregate-to-view: "true" + ``` + +- This ClusterRole permissions will be added to `admin`/`edit`/`view` respectively + +- This is particulary useful when using CustomResourceDefinitions + + (since Kubernetes cannot guess which resources are sensitive and which ones aren't) + +--- + +class: extra-details + +## Where do our permissions come from? + +- When interacting with the Kubernetes API, we are using a client certificate + +- We saw previously that this client certificate contained: + + `CN=kubernetes-admin` and `O=system:masters` + +- Let's look for these in existing ClusterRoleBindings: + ```bash + kubectl get clusterrolebindings -o yaml | + grep -e kubernetes-admin -e system:masters + ``` + + (`system:masters` should show up, but not `kubernetes-admin`.) + +- Where does this match come from? + +--- + +class: extra-details + +## The `system:masters` group + +- If we eyeball the output of `kubectl get clusterrolebindings -o yaml`, we'll find out! + +- It is in the `cluster-admin` binding: + ```bash + kubectl describe clusterrolebinding cluster-admin + ``` + +- This binding associates `system:masters` with the cluster role `cluster-admin` + +- And the `cluster-admin` is, basically, `root`: + ```bash + kubectl describe clusterrole cluster-admin + ``` + +--- + +class: extra-details + +# Pod Security Policies + +- If you'd like to check out pod-level controls in AKS, they are [available in preview](https://docs.microsoft.com/en-us/azure/aks/use-pod-security-policies) + +- Experiment, but not in production! diff --git a/slides/k8s/healthchecks.md b/slides/k8s/healthchecks.md index a72fedbf..a647c9ce 100644 --- a/slides/k8s/healthchecks.md +++ b/slides/k8s/healthchecks.md @@ -120,7 +120,7 @@ ## Example: HTTP probe -Here is a pod template for the `rng` web service of the DockerCoins app: +Here is a pod template for the `rng` web service of our DockerCoins sample app: ```yaml apiVersion: v1 diff --git a/slides/k8s/kubercoins-k8s201.md b/slides/k8s/kubercoins-k8s201.md new file mode 100644 index 00000000..37449503 --- /dev/null +++ b/slides/k8s/kubercoins-k8s201.md @@ -0,0 +1,221 @@ +# Deploying a sample application + +- We will connect to our new Kubernetes cluster + +- We will deploy a sample application, "DockerCoins" + +- That app features multiple micro-services and a web UI + +--- + +## Cloning some repos + +- We will need two repositories: + + - the first one has the "DockerCoins" demo app + + - the second one has these slides, some scripts, more manifests ... + +.exercise[ + +- Clone the kubercoins repository locally: + ```bash + git clone https://github.com/jpetazzo/kubercoins + ``` + + +- Clone the container.training repository as well: + ```bash + git clone https://@@GITREPO@@ + ``` + +] + +--- + +## Running the application + +Without further ado, let's start this application! + +.exercise[ + +- Apply all the manifests from the kubercoins repository: + ```bash + kubectl apply -f kubercoins/ + ``` + +] + +--- + +## What's this application? + +-- + +- It is a DockerCoin miner! .emoji[💰🐳📦🚢] + +-- + +- No, you can't buy coffee with DockerCoins + +-- + +- How DockerCoins works: + + - generate a few random bytes + + - hash these bytes + + - increment a counter (to keep track of speed) + + - repeat forever! + +-- + +- DockerCoins is *not* a cryptocurrency + + (the only common points are "randomness", "hashing", and "coins" in the name) + +--- + +## DockerCoins in the microservices era + +- DockerCoins is made of 5 services: + + - `rng` = web service generating random bytes + + - `hasher` = web service computing hash of POSTed data + + - `worker` = background process calling `rng` and `hasher` + + - `webui` = web interface to watch progress + + - `redis` = data store (holds a counter updated by `worker`) + +- These 5 services are visible in the application's Compose file, + [docker-compose.yml]( + https://@@GITREPO@@/blob/master/dockercoins/docker-compose.yml) + +--- + +## How DockerCoins works + +- `worker` invokes web service `rng` to generate random bytes + +- `worker` invokes web service `hasher` to hash these bytes + +- `worker` does this in an infinite loop + +- every second, `worker` updates `redis` to indicate how many loops were done + +- `webui` queries `redis`, and computes and exposes "hashing speed" in our browser + +*(See diagram on next slide!)* + +--- + +class: pic + +![Diagram showing the 5 containers of the applications](images/dockercoins-diagram.svg) + +--- + +## Service discovery in container-land + +How does each service find out the address of the other ones? + +-- + +- We do not hard-code IP addresses in the code + +- We do not hard-code FQDNs in the code, either + +- We just connect to a service name, and container-magic does the rest + + (And by container-magic, we mean "a crafty, dynamic, embedded DNS server") + +--- + +## Example in `worker/worker.py` + +```python +redis = Redis("`redis`") + + +def get_random_bytes(): + r = requests.get("http://`rng`/32") + return r.content + + +def hash_bytes(data): + r = requests.post("http://`hasher`/", + data=data, + headers={"Content-Type": "application/octet-stream"}) +``` + +(Full source code available [here]( +https://@@GITREPO@@/blob/8279a3bce9398f7c1a53bdd95187c53eda4e6435/dockercoins/worker/worker.py#L17 +)) + +--- + +## Show me the code! + +- You can check the GitHub repository with all the materials of this workshop: +
https://@@GITREPO@@ + +- The application is in the [dockercoins]( + https://@@GITREPO@@/tree/master/dockercoins) + subdirectory + +- The Compose file ([docker-compose.yml]( + https://@@GITREPO@@/blob/master/dockercoins/docker-compose.yml)) + lists all 5 services + +- `redis` is using an official image from the Docker Hub + +- `hasher`, `rng`, `worker`, `webui` are each built from a Dockerfile + +- Each service's Dockerfile and source code is in its own directory + + (`hasher` is in the [hasher](https://@@GITREPO@@/blob/master/dockercoins/hasher/) directory, + `rng` is in the [rng](https://@@GITREPO@@/blob/master/dockercoins/rng/) + directory, etc.) + +--- + +## Our application at work + +- We can check the logs of our application's pods + +.exercise[ + +- Check the logs of the various components: + ```bash + kubectl logs deploy/worker + kubectl logs deploy/hasher + ``` + +] + +--- + +## Connecting to the web UI + +- "Logs are exciting and fun!" (No-one, ever) + +- The `webui` container exposes a web dashboard; let's view it + +.exercise[ + +- Check the NodePort allocated to the web UI: + ```bash + kubectl get svc webui + ``` + +- Open that in a web browser + +] + +A drawing area should show up, and after a few seconds, a blue +graph will appear. diff --git a/slides/k8s/localkubeconfig-k8s201.md b/slides/k8s/localkubeconfig-k8s201.md new file mode 100644 index 00000000..bc167bab --- /dev/null +++ b/slides/k8s/localkubeconfig-k8s201.md @@ -0,0 +1,181 @@ +# Controlling a Kubernetes cluster remotely + +- `kubectl` can be used either on cluster instances or outside the cluster + +- Here, we are going to use `kubectl` from our local machine + +--- + +## Requirements + +.warning[The exercises in this chapter should be done *on your local machine*.] + +- `kubectl` is officially available on Linux, macOS, Windows + + (and unofficially anywhere we can build and run Go binaries) + +- You may want to try Azure cloud shell if you are following along from: + + - a tablet or phone + + - a web-based terminal + + - an environment where you can't install and run new binaries + +--- + +## Installing `kubectl` + +- If you already have `kubectl` on your local machine, you can skip this + +.exercise[ + + + +- Download the `kubectl` binary from one of these links: + + [Linux](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl) + | + [macOS](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/darwin/amd64/kubectl) + | + [Windows](https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/windows/amd64/kubectl.exe) + +- On Linux and macOS, make the binary executable with `chmod +x kubectl` + + (And remember to run it with `./kubectl` or move it to your `$PATH`) + +] + +Note: if you are following along with a different platform (e.g. Linux on an architecture different from amd64, or with a phone or tablet), installing `kubectl` might be more complicated (or even impossible) so check with us about cloud shell. + +--- + +## Testing `kubectl` + +- Check that `kubectl` works correctly + + (before even trying to connect to a remote cluster!) + +.exercise[ + +- Ask `kubectl` to show its version number: + ```bash + kubectl version --client + ``` + +] + +The output should look like this: +``` +Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", +GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", +BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", +Platform:"darwin/amd64"} +``` + +--- + +## Preserving the existing `~/.kube/config` + +- If you already have a `~/.kube/config` file, rename it + + (we are going to overwrite it in the following slides!) + +- If you never used `kubectl` on your machine before: nothing to do! + +.exercise[ + +- Make a copy of `~/.kube/config`; if you are using macOS or Linux, you can do: + ```bash + cp ~/.kube/config ~/.kube/config.before.training + ``` + +- If you are using Windows, you will need to adapt this command + +] + +--- + +## Connecting to your AKS cluster + +[fill in] + +--- + +## Let's look at your cluster! + + +- First, inspect the config + ```bash + kubectl config view + ``` + +- Look for the `server:` address that matches your new cluster + +``` +- cluster: + certificate-authority-data: DATA+OMITTED + server: https://aks-test-c-aks-test-group-0d35f7-28c7d691.hcp.eastus.azmk8s.io:443 + name: aks-test-cluster +``` + +--- + +class: extra-details + +## What if we get a certificate error? + +- Generally, the Kubernetes API uses a certificate that is valid for: + + - `kubernetes` + - `kubernetes.default` + - `kubernetes.default.svc` + - `kubernetes.default.svc.cluster.local` + - the ClusterIP address of the `kubernetes` service + - the hostname of the node hosting the control plane + - the IP address of the node hosting the control plane + +- On most clouds, the IP address of the node is an internal IP address + +- ... And we are going to connect over the external IP address + +- ... And that external IP address was not used when creating the certificate! + +--- + +class: extra-details + +## Working around the certificate error + +- We need to tell `kubectl` to skip TLS verification + + (only do this with testing clusters, never in production!) + +- The following command will do the trick: + ```bash + kubectl config set-cluster --insecure-skip-tls-verify + ``` + +--- + +## Checking that we can connect to the cluster + +- We can now run a couple of trivial commands to check that all is well + +.exercise[ + +- Check the versions of the local client and remote server: + ```bash + kubectl version + ``` + +It is okay if you have a newer client than what is available on the server. + +- View the nodes of the cluster: + ```bash + kubectl get nodes + ``` + +] + +We can now utilize the cluster exactly as if we're logged into a node, except that it's remote. diff --git a/slides/k8s/prereqs-k8s201.md b/slides/k8s/prereqs-k8s201.md index 970ce03f..b000c7af 100644 --- a/slides/k8s/prereqs-k8s201.md +++ b/slides/k8s/prereqs-k8s201.md @@ -44,7 +44,9 @@ - We will connect to these clusters with `kubectl` - (if you don't have `kubectl` installed, install it **now!**) +- If you don't have `kubectl` installed, we'll explain how to install it shortly + +- You will also want to install `jq` ---