diff --git a/slides/k8s-201.yml b/slides/k8s-201.yml index 5380b6ac..8ce24cfe 100644 --- a/slides/k8s-201.yml +++ b/slides/k8s-201.yml @@ -28,7 +28,7 @@ chapters: - k8s/authn-authz-k8s201.md - - k8s/resource-limits-k8s201.md - k8s/metrics-server.md -- - k8s/cluster-sizing.md +- - k8s/cluster-sizing-k8s201.md - k8s/horizontal-pod-autoscaler.md - k8s/extending-api.md - k8s/operators.md diff --git a/slides/k8s/cluster-sizing-k8s201.md b/slides/k8s/cluster-sizing-k8s201.md new file mode 100644 index 00000000..2432a77b --- /dev/null +++ b/slides/k8s/cluster-sizing-k8s201.md @@ -0,0 +1,167 @@ +# Cluster sizing + +- What happens when the cluster gets full? + +- How can we scale up the cluster? + +- Can we do it automatically? + +- What are other methods to address capacity planning? + +--- + +## When are we out of resources? + +- kubelet monitors node resources: + + - memory + + - node disk usage (typically the root filesystem of the node) + + - image disk usage (where container images and RW layers are stored) + +- For each resource, we can provide two thresholds: + + - a hard threshold (if it's met, it provokes immediate action) + + - a soft threshold (provokes action only after a grace period) + +- Resource thresholds and grace periods are configurable + + (by passing kubelet command-line flags) + +--- + +## What happens then? + +- If disk usage is too high: + + - kubelet will try to remove terminated pods + + - then, it will try to *evict* pods + +- If memory usage is too high: + + - it will try to evict pods + +- The node is marked as "under pressure" + +- This temporarily prevents new pods from being scheduled on the node + +--- + +## Which pods get evicted? + +- kubelet looks at the pods' QoS and PriorityClass + +- First, pods with BestEffort QoS are considered + +- Then, pods with Burstable QoS exceeding their *requests* + + (but only if the exceeding resource is the one that is low on the node) + +- Finally, pods with Guaranteed QoS, and Burstable pods within their requests + +- Within each group, pods are sorted by PriorityClass + +- If there are pods with the same PriorityClass, they are sorted by usage excess + + (i.e. the pods whose usage exceeds their requests the most are evicted first) + +--- + +class: extra-details + +## Eviction of Guaranteed pods + +- *Normally*, pods with Guaranteed QoS should not be evicted + +- A chunk of resources is reserved for node processes (like kubelet) + +- It is expected that these processes won't use more than this reservation + +- If they do use more resources anyway, all bets are off! + +- If this happens, kubelet must evict Guaranteed pods to preserve node stability + + (or Burstable pods that are still within their requested usage) + +--- + +## What happens to evicted pods? + +- The pod is terminated + +- It is marked as `Failed` at the API level + +- If the pod was created by a controller, the controller will recreate it + +- The pod will be recreated on another node, *if there are resources available!* + +- For more details about the eviction process, see: + + - [this documentation page](https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/) about resource pressure and pod eviction, + + - [this other documentation page](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/) about pod priority and preemption. + +--- + +## What if there are no resources available? + +- Sometimes, a pod cannot be scheduled anywhere: + + - all the nodes are under pressure, + + - or the pod requests more resources than are available + +- The pod then remains in `Pending` state until the situation improves + +--- + +## Cluster scaling + +- One way to improve the situation is to add new nodes + +- This can be done automatically with the [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) + +- The autoscaler will automatically scale up: + + - if there are pods that failed to be scheduled + +- The autoscaler will automatically scale down: + + - if nodes have a low utilization for an extended period of time + +--- + +## Restrictions, gotchas ... + +- The Cluster Autoscaler only supports a few cloud infrastructures + + (see [here](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider) for a list) - ([in preview for AKS](https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler)) + +- The Cluster Autoscaler cannot scale down nodes that have pods using: + + - local storage + + - affinity/anti-affinity rules preventing them from being rescheduled + + - a restrictive PodDisruptionBudget + +--- + +## Other way to do capacity planning + +- "Running Kubernetes without nodes" + +- Systems like [Virtual Kubelet](https://virtual-kubelet.io/) or Kiyot can run pods using on-demand resources + + - Virtual Kubelet can leverage e.g. ACI or Fargate to run pods + + - Kiyot runs pods in ad-hoc EC2 instances (1 instance per pod) + +- Economic advantage (no wasted capacity) + +- Security advantage (stronger isolation between pods) + +Check [this blog post](http://jpetazzo.github.io/2019/02/13/running-kubernetes-without-nodes-with-kiyot/) for more details. diff --git a/slides/k8s/horizontal-pod-autoscaler.md b/slides/k8s/horizontal-pod-autoscaler.md index 519b5ef9..26edd4d5 100644 --- a/slides/k8s/horizontal-pod-autoscaler.md +++ b/slides/k8s/horizontal-pod-autoscaler.md @@ -88,9 +88,10 @@ kubectl expose deployment busyhttp --port=80 ``` -- Get the ClusterIP allocated to the service: +- Port-forward to our service ```bash - kubectl get svc busyhttp + kubectl port-forward service/busyhttp 8080:80 & + curl -k localhost:8080 ``` ] diff --git a/slides/k8s/kubercoins-k8s201.md b/slides/k8s/kubercoins-k8s201.md index 5a892c45..c70ea2c1 100644 --- a/slides/k8s/kubercoins-k8s201.md +++ b/slides/k8s/kubercoins-k8s201.md @@ -211,11 +211,11 @@ https://@@GITREPO@@/blob/8279a3bce9398f7c1a53bdd95187c53eda4e6435/dockercoins/wo kubectl proxy & ``` -- Open in a web browser: [our webui](http://localhost:8001/api/v1/namespaces/default/services/webui/proxy/index.html) +- Open in a web browser: [http://localhost:8001/api/v1/namespaces/default/services/webui/proxy/index.html](http://localhost:8001/api/v1/namespaces/default/services/webui/proxy/index.html) ] A drawing area should show up, and after a few seconds, a blue graph will appear. -- If using Cloud Shell, use the [Cloud Shell Web Preview](https://docs.microsoft.com/en-us/azure/cloud-shell/using-the-shell-window#web-preview) +- If using Cloud Shell, use the [Cloud Shell Web Preview](https://docs.microsoft.com/en-us/azure/cloud-shell/using-the-shell-window#web-preview) and append `/api/v1/namespaces/default/services/webui/proxy/index.html` to the existing path diff --git a/slides/k8s/localkubeconfig-k8s201.md b/slides/k8s/localkubeconfig-k8s201.md index ac50b8b9..78dbc0cb 100644 --- a/slides/k8s/localkubeconfig-k8s201.md +++ b/slides/k8s/localkubeconfig-k8s201.md @@ -31,6 +31,28 @@ - If you're going to use Cloud Shell, you can skip ahead +--- +class: extra-details + +## Preserving the existing `~/.kube/config` (optional) + +- If you already have a `~/.kube/config` file, rename it + + (we are going to overwrite it in the following slides!) + +- If you never used `kubectl` on your machine before: nothing to do! + +.exercise[ + +- Make a copy of `~/.kube/config`; if you are using macOS or Linux, you can do: + ```bash + cp ~/.kube/config ~/.kube/config.before.training + ``` + +- If you are using Windows, you will need to adapt this command + +] + --- ## Connecting to your AKS cluster via local tools @@ -107,27 +129,6 @@ class: extra-details Note: if you are following along with a different platform (e.g. Linux on an architecture different from amd64, or with a phone or tablet), installing `kubectl` might be more complicated (or even impossible) so check with us about cloud shell. ---- -class: extra-details - -## Preserving the existing `~/.kube/config` - -- If you already have a `~/.kube/config` file, rename it - - (we are going to overwrite it in the following slides!) - -- If you never used `kubectl` on your machine before: nothing to do! - -.exercise[ - -- Make a copy of `~/.kube/config`; if you are using macOS or Linux, you can do: - ```bash - cp ~/.kube/config ~/.kube/config.before.training - ``` - -- If you are using Windows, you will need to adapt this command - -] --- @@ -170,6 +171,7 @@ Platform:"darwin/amd64"} API_URL=$(kubectl config view -o json | jq -r ".clusters[] \ | select(.name == \"$AKS_NAME\") | .cluster.server") echo $API_URL + ``` ] --- diff --git a/slides/k8s/resource-limits-k8s201.md b/slides/k8s/resource-limits-k8s201.md index fed14f22..3ec0d87a 100644 --- a/slides/k8s/resource-limits-k8s201.md +++ b/slides/k8s/resource-limits-k8s201.md @@ -432,10 +432,6 @@ When a ResourceQuota is created, we can see how much of it is used: ``` -] - -.exercise[ - - Remove quota: ```bash kubectl delete quota my-resource-quota