diff --git a/content/runbooks/kubernetes/CPUThrottlingHigh.md b/content/runbooks/kubernetes/CPUThrottlingHigh.md new file mode 100644 index 0000000..0d89068 --- /dev/null +++ b/content/runbooks/kubernetes/CPUThrottlingHigh.md @@ -0,0 +1,45 @@ +--- +title: CPU Throttling High +weight: 20 +--- + +# CPU Throttling High + +## Meaning + +Processes experience elevated CPU throttling. + +## Impact + +The alert is purely informative and unless there is some other issue with +the application, it can be skipped. + +## Diagnosis + +- Check if application is performing normally +- Check if CPU resource requests are adjusted accordingly to the app usage +- Check kernel version in the node + +## Mitigation + +**Notice**: +User shouldn't increase CPU limits unless the application is behaving +erratically (another alert firing). + +For this particular reason, the alert is inhibited by default in +kube-prometheus and can be sent only if another alert in the same namespace +is firing. + +**When mixed with other alerts**: + +Give specific container in the pod more CPU limits. Requests can stay the same. + +In specific cases kubernetes node has too old kernel which is known to have +issues with assigning CPU resources to the process [see here](https://github.com/kubernetes/kubernetes/issues/67577) + +In certain scenarios ensure to use CPU Pinning and isolation - in short give +to the container full CPU cores. +Also ensure to update app so that it is aware it runs in cgropus, +or explicitly set number of CPU it can use, or limit number of threads. + +Longer and more detailed info - [PDF from Intel](https://builders.intel.com/docs/networkbuilders/cpu-pin-and-isolation-in-kubernetes-app-note.pdf) diff --git a/content/runbooks/kubernetes/KubeAPIDown.md b/content/runbooks/kubernetes/KubeAPIDown.md index 958c115..d9edb7f 100644 --- a/content/runbooks/kubernetes/KubeAPIDown.md +++ b/content/runbooks/kubernetes/KubeAPIDown.md @@ -1,3 +1,8 @@ +--- +title: Kube API Down +weight: 20 +--- + # KubeAPIDown ## Meaning @@ -10,13 +15,18 @@ been reachable by the monitoring system for more than 15 minutes. This is a critical alert. The Kubernetes API is not responding. The cluster may partially or fully non-functional. +Applications, which do not use kubernetes API directly, will continue to work. Changing kubernetes resources is not possible. +in the cluster. + +Services using Kubernetes API directly will start to behave erratically. + ## Diagnosis Check the status of the API server targets in the Prometheus UI. Then, confirm whether the API is also unresponsive for you: -```console +```shell $ kubectl cluster-info ``` @@ -24,10 +34,17 @@ If you can still reach the API server, there may be a network issue between the Prometheus instances and the API server pods. Check the status of the API server pods. -```console +```shell $ kubectl -n kube-system get pods $ kubectl -n kube-system logs -l 'app=kube-apiserver' ``` + +- Check networking on the node. +- Check firewall on the node. +- Investigate kube proxy logs. +- Investigate NetworkPolicies if prometheus/kubeApi was not filtered out. + + ## Mitigation If you can still reach the API server intermittently, you may be able treat this diff --git a/content/runbooks/kubernetes/KubeAPIErrorBudgetBurn.md b/content/runbooks/kubernetes/KubeAPIErrorBudgetBurn.md index cf74a9f..81932ce 100644 --- a/content/runbooks/kubernetes/KubeAPIErrorBudgetBurn.md +++ b/content/runbooks/kubernetes/KubeAPIErrorBudgetBurn.md @@ -1,8 +1,13 @@ +--- +title: Kube API Error Budget Burn +weight: 20 +--- + # KubeAPIErrorBudgetBurn ## Impact -The overall availability of your Kubernetes cluster isn't guaranteed anymore. +The overall availability of your Kubernetes cluster isn't guaranteed any more. There may be **too many errors** returned by the APIServer and/or **responses take too long** for guarantee proper reconciliation. **This is always important; the only deciding factor is how urgent it is at the current rate** @@ -43,17 +48,19 @@ _Example: If you have a 99% availability target this means that at the end of 30 1. If you don't see anything obvious with the error rates, it might be too many slow requests. [Check the queries below!](#example-queries-for-slow-requests) 1. Maybe it's some dependency of the APIServer? etcd? -### Example Queries for slow requests: +### Example Queries for slow requests Change the rate window according to your `long` label from the alert. Make sure to update the alert threshold too, like `> 0.01` to `> 14.4 * 0.01` for example. -#### Slow Read Requests: + +#### Slow Read Requests If you don't get any results back then there aren't too many slow requests - that's good. If you get results than you know what type of requests are too slow. Cluster scoped: -``` + +```promql ( sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="40",scope="cluster",verb=~"LIST|GET"}[3d])) - @@ -63,8 +70,10 @@ sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"LIST|GE sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) > 0.01 ``` + Namespace scoped: -``` + +```promql ( sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="5",scope="namespace",verb=~"LIST|GET"}[3d])) - @@ -76,7 +85,8 @@ sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) ``` Resource scoped: -``` + +```promql ( sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="1",scope=~"resource|",verb=~"LIST|GET"}[3d])) or vector(0) - @@ -89,7 +99,7 @@ sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d])) #### Slow Write Requests -``` +```promql ( sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d])) - diff --git a/content/runbooks/kubernetes/KubeAPITerminatedRequests.md b/content/runbooks/kubernetes/KubeAPITerminatedRequests.md new file mode 100644 index 0000000..bf9ff7d --- /dev/null +++ b/content/runbooks/kubernetes/KubeAPITerminatedRequests.md @@ -0,0 +1,25 @@ +--- +title: Kube API Terminated Requests +weight: 20 +--- + +# KubeAPITerminatedRequests + +## Meaning + +The apiserver has terminated over 20% of its incoming requests. + +## Impact + +Client will not be able to interact with the cluster. +Some in-cluster services this may degrade or make service unavailable. + +## Diagnosis + +Use the `apiserver_flowcontrol_rejected_requests_total` metric to determine +which flow schema is throttling the traffic to the API Server. +The flow schema also provides information on the affected resources and subjects. + +## Mitigation + +TODO diff --git a/content/runbooks/kubernetes/KubeAggregatedAPIDown.md b/content/runbooks/kubernetes/KubeAggregatedAPIDown.md new file mode 100644 index 0000000..5840be2 --- /dev/null +++ b/content/runbooks/kubernetes/KubeAggregatedAPIDown.md @@ -0,0 +1,30 @@ +--- +title: Kube Aggregated API Down +weight: 20 +--- + +# KubeAggregatedAPIDown + +## Meaning + +Kubernetes aggregated API has reported errors. +It has appeared unavailable X times averaged over the past 10m. + +## Impact + +From minor such as inability to see cluster metrics to more severe such as +unable to use custom metrics to scale or even unable to use cluster. + +## Diagnosis + +- Check networking on the node. +- Check firewall on the node. +- Investigate additional API logs. +- Investigate NetworkPolicies if kubeApi - additional API was not filtered out. +- Investigate NetworkPolicies if prometheus/additional api was not filtered out. + +## Mitigation + +TODO + +See [APIServer aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) diff --git a/content/runbooks/kubernetes/KubeAggregatedAPIErrors.md b/content/runbooks/kubernetes/KubeAggregatedAPIErrors.md new file mode 100644 index 0000000..eb11445 --- /dev/null +++ b/content/runbooks/kubernetes/KubeAggregatedAPIErrors.md @@ -0,0 +1,30 @@ +--- +title: Kube Aggregated API Errors +weight: 20 +--- + +# KubeAggregatedAPIErrors + +## Meaning + +Kubernetes aggregated API has reported errors. +It has appeared unavailable over 4 times averaged over the past 10m. + +## Impact + +From minor such as inability to see cluster metrics to more severe such as +unable to use custom metrics to scale or even unable to use cluster. + +## Diagnosis + +- Check networking on the node. +- Check firewall on the node. +- Investigate additional API logs. +- Investigate NetworkPolicies if kubeApi - additional API was not filtered out. +- Investigate NetworkPolicies if prometheus/additional API was not filtered out. + +## Mitigation + +TODO + +See [APIServer aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) diff --git a/content/runbooks/kubernetes/KubeClientCertificateExpiration.md b/content/runbooks/kubernetes/KubeClientCertificateExpiration.md new file mode 100644 index 0000000..b43b984 --- /dev/null +++ b/content/runbooks/kubernetes/KubeClientCertificateExpiration.md @@ -0,0 +1,26 @@ +--- +title: Kube Client Certificate Expiration +weight: 20 +--- + +# KubeClientCertificateExpiration + +## Meaning + +A client certificate used to authenticate to the apiserver is expiring in less than 7 days (warning alert) or 24 hours (critical alert). + +## Impact + +Client will not be able to interact with the cluster. +In cluster services communicating with Kubernetes API may degrade or become unavailable. + +## Diagnosis + +Check when certificate was issued and when it expires. +Check serviceAccounts and service account tokens. + +## Mitigation + +Update client certificate. + +For in-cluster clients recreate pods. diff --git a/content/runbooks/kubernetes/KubeClientErrors.md b/content/runbooks/kubernetes/KubeClientErrors.md new file mode 100644 index 0000000..ac1db5a --- /dev/null +++ b/content/runbooks/kubernetes/KubeClientErrors.md @@ -0,0 +1,32 @@ +--- +title: Kube Client Errors +weight: 20 +--- + +# KubeClientErrors + +## Meaning + +Kubernetes API server client is experiencing over 1% error rate in the last 15 minutes. + +## Impact + +Specific kubernetes client may malfunction. Service degradation. + +## Diagnosis + +Usual issues: + +- networking errors +- too low resources to perform given API calls (usually too low CPU/memory requests) +- wrong api client (old libraries) +- investigate if the app does not request more data than it really requires + from kubernetes API, for example it has too wide permissions and scans for + resources in all namespaces. + +Check logs from client side (sometimes app logs). + + +## Mitigation + +TODO diff --git a/content/runbooks/kubernetes/KubeContainerWaiting.md b/content/runbooks/kubernetes/KubeContainerWaiting.md new file mode 100644 index 0000000..428c94f --- /dev/null +++ b/content/runbooks/kubernetes/KubeContainerWaiting.md @@ -0,0 +1,26 @@ +--- +title: Kube Container Waiting +weight: 20 +--- + +# KubeContainerWaiting + +## Meaning + +Container in pod is in Waiting state for too long. + +## Impact + +Service degradation or unavailability. + +## Diagnosis + +- Check pod events via `kubectl -n $NAMESPACE describe pod $POD`. +- Check pod logs via `kubectl -n $NAMESPACE logs $POD -c $CONTAINER` +- Check for missing files such as configmaps/secrets/volumes +- Check for pod requests, especially special ones such as GPU. +- Check for node taints and capabilities. + +## Mitigation + +See [Container waiting](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#my-pod-stays-waiting) diff --git a/content/runbooks/kubernetes/KubeControllerManagerDown.md b/content/runbooks/kubernetes/KubeControllerManagerDown.md new file mode 100644 index 0000000..75fe4fd --- /dev/null +++ b/content/runbooks/kubernetes/KubeControllerManagerDown.md @@ -0,0 +1,29 @@ +--- +title: Kube Controller Manager Down +weight: 20 +--- + +# KubeControllerManagerDown + +## Meaning + +KubeControllerManager has disappeared from Prometheus target discovery. + +## Impact + +The cluster is not functional and Kubernetes resources cannot be reconciled. + +
+Full context + +More about kube-controller-manager function can be found at https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ + +
+ +## Diagnosis + +TODO + +## Mitigation + +See old CoreOS docs in [Web Archive](http://web.archive.org/web/20201026205154/https://coreos.com/tectonic/docs/latest/troubleshooting/controller-recovery.html)