Update runbooks/kubernetes (part 1)

2026-08-27 12:37:20 +00:00 · 2022-02-18 20:42:14 +01:00
parent 5ef0792031
commit 74ce294bb3
10 changed files with 279 additions and 9 deletions
@@ -0,0 +1,45 @@
+---
+title: CPU Throttling High
+weight: 20
+---
+
+# CPU Throttling High
+
+## Meaning
+
+Processes experience elevated CPU throttling.
+
+## Impact
+
+The alert is purely informative and unless there is some other issue with
+the application, it can be skipped.
+
+## Diagnosis
+
+- Check if application is performing normally
+- Check if CPU resource requests are adjusted accordingly to the app usage
+- Check kernel version in the node
+
+## Mitigation
+
+**Notice**:
+User shouldn't increase CPU limits unless the application is behaving
+erratically (another alert firing).
+
+For this particular reason, the alert is inhibited by default in
+kube-prometheus and can be sent only if another alert in the same namespace
+is firing.
+
+**When mixed with other alerts**:
+
+Give specific container in the pod more CPU limits. Requests can stay the same.
+
+In specific cases kubernetes node has too old kernel which is known to have
+issues with assigning CPU resources to the process [see here](https://github.com/kubernetes/kubernetes/issues/67577)
+
+In certain scenarios ensure to use CPU Pinning and isolation - in short give
+to the container full CPU cores.
+Also ensure to update app so that it is aware it runs in cgropus,
+or explicitly set number of CPU it can use, or limit number of threads.
+
+Longer and more detailed info - [PDF from Intel](https://builders.intel.com/docs/networkbuilders/cpu-pin-and-isolation-in-kubernetes-app-note.pdf)
@@ -1,3 +1,8 @@
+---
+title: Kube API Down
+weight: 20
+---
+
 # KubeAPIDown

 ## Meaning
@@ -10,13 +15,18 @@ been reachable by the monitoring system for more than 15 minutes.
 This is a critical alert. The Kubernetes API is not responding. The
 cluster may partially or fully non-functional.

+Applications, which do not use kubernetes API directly, will continue to work. Changing kubernetes resources is not possible.
+in the cluster.
+
+Services using Kubernetes API directly will start to behave erratically.
+
 ## Diagnosis

 Check the status of the API server targets in the Prometheus UI.

 Then, confirm whether the API is also unresponsive for you:

-```console
+```shell
 $ kubectl cluster-info
 ```

@@ -24,10 +34,17 @@ If you can still reach the API server, there may be a network issue between the
 Prometheus instances and the API server pods. Check the status of the API server
 pods.

-```console
+```shell
 $ kubectl -n kube-system get pods
 $ kubectl -n kube-system logs -l 'app=kube-apiserver'
 ```
+
+- Check networking on the node.
+- Check firewall on the node.
+- Investigate kube proxy logs.
+- Investigate NetworkPolicies if prometheus/kubeApi was not filtered out.
+
+
 ## Mitigation

 If you can still reach the API server intermittently, you may be able treat this
@@ -1,8 +1,13 @@
+---
+title: Kube API Error Budget Burn
+weight: 20
+---
+
 # KubeAPIErrorBudgetBurn

 ## Impact

-The overall availability of your Kubernetes cluster isn't guaranteed anymore.
+The overall availability of your Kubernetes cluster isn't guaranteed any more.
 There may be **too many errors** returned by the APIServer and/or **responses take too long** for guarantee proper reconciliation.

 **This is always important; the only deciding factor is how urgent it is at the current rate**
@@ -43,17 +48,19 @@ _Example: If you have a 99% availability target this means that at the end of 30
    1. If you don't see anything obvious with the error rates, it might be too many slow requests. [Check the queries below!](#example-queries-for-slow-requests)
 1. Maybe it's some dependency of the APIServer? etcd?

-### Example Queries for slow requests:
+### Example Queries for slow requests

 Change the rate window according to your `long` label from the alert.
 Make sure to update the alert threshold too, like `> 0.01` to `> 14.4 * 0.01` for example.
-#### Slow Read Requests:
+
+#### Slow Read Requests

 If you don't get any results back then there aren't too many slow requests - that's good.
 If you get results than you know what type of requests are too slow.

 Cluster scoped:
-```
+
+```promql
 (
 sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="40",scope="cluster",verb=~"LIST|GET"}[3d]))
 -
@@ -63,8 +70,10 @@ sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"LIST|GE
 sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
 > 0.01
 ```
+
 Namespace scoped:
-```
+
+```promql
 (
 sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="5",scope="namespace",verb=~"LIST|GET"}[3d]))
 -
@@ -76,7 +85,8 @@ sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))
 ```

 Resource scoped:
-```
+
+```promql
 (
 sum(rate(apiserver_request_duration_seconds_bucket{job="apiserver",le="1",scope=~"resource|",verb=~"LIST|GET"}[3d])) or vector(0)
 -
@@ -89,7 +99,7 @@ sum(rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET"}[3d]))

 #### Slow Write Requests

-```
+```promql
 (
 sum(rate(apiserver_request_duration_seconds_count{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}[3d]))
 -
@@ -0,0 +1,25 @@
+---
+title: Kube API Terminated Requests
+weight: 20
+---
+
+# KubeAPITerminatedRequests
+
+## Meaning
+
+The apiserver has terminated over 20% of its incoming requests.
+
+## Impact
+
+Client will not be able to interact with the cluster.
+Some in-cluster services this may degrade or make service unavailable.
+
+## Diagnosis
+
+Use the `apiserver_flowcontrol_rejected_requests_total` metric to determine
+which flow schema is throttling the traffic to the API Server.
+The flow schema also provides information on the affected resources and subjects.
+
+## Mitigation
+
+TODO
@@ -0,0 +1,30 @@
+---
+title: Kube Aggregated API Down
+weight: 20
+---
+
+# KubeAggregatedAPIDown
+
+## Meaning
+
+Kubernetes aggregated API has reported errors.
+It has appeared unavailable X times averaged over the past 10m.
+
+## Impact
+
+From minor such as inability to see cluster metrics to more severe such as
+unable to use custom metrics to scale or even unable to use cluster.
+
+## Diagnosis
+
+- Check networking on the node.
+- Check firewall on the node.
+- Investigate additional API logs.
+- Investigate NetworkPolicies if kubeApi - additional API was not filtered out.
+- Investigate NetworkPolicies if prometheus/additional api was not filtered out.
+
+## Mitigation
+
+TODO
+
+See [APIServer aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)
@@ -0,0 +1,30 @@
+---
+title: Kube Aggregated API Errors
+weight: 20
+---
+
+# KubeAggregatedAPIErrors
+
+## Meaning
+
+Kubernetes aggregated API has reported errors.
+It has appeared unavailable over 4 times averaged over the past 10m.
+
+## Impact
+
+From minor such as inability to see cluster metrics to more severe such as
+unable to use custom metrics to scale or even unable to use cluster.
+
+## Diagnosis
+
+- Check networking on the node.
+- Check firewall on the node.
+- Investigate additional API logs.
+- Investigate NetworkPolicies if kubeApi - additional API was not filtered out.
+- Investigate NetworkPolicies if prometheus/additional API was not filtered out.
+
+## Mitigation
+
+TODO
+
+See [APIServer aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)
@@ -0,0 +1,26 @@
+---
+title: Kube Client Certificate Expiration
+weight: 20
+---
+
+# KubeClientCertificateExpiration
+
+## Meaning
+
+A client certificate used to authenticate to the apiserver is expiring in less than 7 days (warning alert) or 24 hours (critical alert).
+
+## Impact
+
+Client will not be able to interact with the cluster.
+In cluster services communicating with Kubernetes API may degrade or become unavailable.
+
+## Diagnosis
+
+Check when certificate was issued and when it expires.
+Check serviceAccounts and service account tokens.
+
+## Mitigation
+
+Update client certificate.
+
+For in-cluster clients recreate pods.
@@ -0,0 +1,32 @@
+---
+title: Kube Client Errors
+weight: 20
+---
+
+# KubeClientErrors
+
+## Meaning
+
+Kubernetes API server client is experiencing over 1% error rate in the last 15 minutes.
+
+## Impact
+
+Specific kubernetes client may malfunction. Service degradation.
+
+## Diagnosis
+
+Usual issues:
+
+- networking errors
+- too low resources to perform given API calls (usually too low CPU/memory requests)
+- wrong api client (old libraries)
+- investigate if the app does not request more data than it really requires
+  from kubernetes API, for example it has too wide permissions and scans for
+  resources in all namespaces.
+
+Check logs from client side (sometimes app logs).
+
+
+## Mitigation
+
+TODO
@@ -0,0 +1,26 @@
+---
+title: Kube Container Waiting
+weight: 20
+---
+
+# KubeContainerWaiting
+
+## Meaning
+
+Container in pod is in Waiting state for too long.
+
+## Impact
+
+Service degradation or unavailability.
+
+## Diagnosis
+
+- Check pod events via `kubectl -n $NAMESPACE describe pod $POD`.
+- Check pod logs via `kubectl -n $NAMESPACE logs $POD -c $CONTAINER`
+- Check for missing files such as configmaps/secrets/volumes
+- Check for pod requests, especially special ones such as GPU.
+- Check for node taints and capabilities.
+
+## Mitigation
+
+See [Container waiting](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#my-pod-stays-waiting)
@@ -0,0 +1,29 @@
+---
+title: Kube Controller Manager Down
+weight: 20
+---
+
+# KubeControllerManagerDown
+
+## Meaning
+
+KubeControllerManager has disappeared from Prometheus target discovery.
+
+## Impact
+
+The cluster is not functional and Kubernetes resources cannot be reconciled.
+
+<details>
+<summary>Full context</summary>
+
+More about kube-controller-manager function can be found at https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
+
+</details>
+
+## Diagnosis
+
+TODO
+
+## Mitigation
+
+See old CoreOS docs in [Web Archive](http://web.archive.org/web/20201026205154/https://coreos.com/tectonic/docs/latest/troubleshooting/controller-recovery.html)