mirror of
https://github.com/prometheus-operator/runbooks.git
synced 2026-08-24 03:26:17 +00:00
Update runbooks/kubernetes (part 4)
This commit is contained in:
Michał Sochoń
@@ -0,0 +1,41 @@
|
||||
---
|
||||
title: Kube Memory Quota Overcommit
|
||||
weight: 20
|
||||
aliases:
|
||||
- /kubememquotaovercommit/
|
||||
---
|
||||
|
||||
# KubeMemoryQuotaOvercommit
|
||||
|
||||
## Meaning
|
||||
|
||||
Cluster has overcommitted memory resource requests for Namespaces.
|
||||
|
||||
## Impact
|
||||
|
||||
Various services degradation or unavailability in case of single node failure.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check if Memory resource requests are adjusted to the app usage
|
||||
- Check if some nodes are available and not cordoned
|
||||
- Check if cluster-autoscaler has issues with adding new nodes
|
||||
- Check if the given namespace usage grows in time more than expected
|
||||
|
||||
## Mitigation
|
||||
|
||||
- Review existing quota for given namespace and adjust it accordingly.
|
||||
|
||||
- Add more nodes to the cluster - usually it is better to have more smaller
|
||||
nodes, than few bigger.
|
||||
|
||||
- Add different node pools with different instance types to avoid problem
|
||||
when using only one instance type in the cloud.
|
||||
|
||||
- Use pod priorities to avoid important services from losing performance,
|
||||
see [pod priority and preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)
|
||||
|
||||
- Fine tune settings for special pods used with [cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-cluster-autoscaler-work-with-pod-priority-and-preemption)
|
||||
|
||||
- Prepare performance tests for the expected workload, plan cluster capacity
|
||||
accordingly.
|
||||
@@ -1,3 +1,8 @@
|
||||
---
|
||||
title: Kube Node Not Ready
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubeNodeNotReady
|
||||
|
||||
## Meaning
|
||||
@@ -24,7 +29,7 @@ The notification details should list the node that's not ready. For Example:
|
||||
|
||||
Login to the cluster. Check the status of that node:
|
||||
|
||||
```console
|
||||
```shell
|
||||
$ kubectl get node $NODE -o yaml
|
||||
```
|
||||
|
||||
@@ -36,4 +41,5 @@ API or kubelet).
|
||||
Once, the problem was resolved that prevented node from being replaced,
|
||||
the instance should be terminated.
|
||||
|
||||
[KubeNode]: https://kubernetes.io/docs/concepts/architecture/nodes/#condition
|
||||
See [KubeNode](https://kubernetes.io/docs/concepts/architecture/nodes/#condition)
|
||||
See [node problem detector](https://github.com/kubernetes/node-problem-detector)
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Kube Node Readiness Flapping
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubeNodeReadinessFlapping
|
||||
|
||||
## Meaning
|
||||
|
||||
The readiness status of node has changed few times in the last 15 minutes.
|
||||
|
||||
## Impact
|
||||
|
||||
The performance of the cluster deployments is affected, depending on the overall
|
||||
workload and the type of the node.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
The notification details should list the node that's not reachable. For Example:
|
||||
|
||||
```txt
|
||||
- alertname = KubeNodeUnreachable
|
||||
...
|
||||
- node = node1.example.com
|
||||
...
|
||||
```
|
||||
|
||||
Login to the cluster. Check the status of that node:
|
||||
|
||||
```shell
|
||||
$ kubectl get node $NODE -o yaml
|
||||
```
|
||||
|
||||
The output should describe why the node is not reachable.
|
||||
|
||||
Common failure scenarios:
|
||||
|
||||
- disruptive software upgrades
|
||||
- network patitioning due to hardware failures
|
||||
- firewall rules
|
||||
- virtual machines suspended due to storage area network problems
|
||||
- system crashes / freezes due to software or hardware malfunctions
|
||||
|
||||
## Mitigation
|
||||
|
||||
In case of maintenance ensure to [cordon and drain node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/).
|
||||
|
||||
In other cases ensure storage and networking redundancy if applicable.
|
||||
|
||||
See [KubeNode](https://kubernetes.io/docs/concepts/architecture/nodes/#condition)
|
||||
See [node problem detector](https://github.com/kubernetes/node-problem-detector)
|
||||
See [Watchdog timer](https://en.wikipedia.org/wiki/Watchdog_timer)
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Kube Node Unreachable
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubeNodeUnreachable
|
||||
|
||||
## Meaning
|
||||
|
||||
Kubernetes node is unreachable and some workloads may be rescheduled.
|
||||
|
||||
## Impact
|
||||
|
||||
The performance of the cluster deployments is affected, depending on the overall
|
||||
workload and the type of the node.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
The notification details should list the node that's not reachable. For Example:
|
||||
|
||||
```txt
|
||||
- alertname = KubeNodeUnreachable
|
||||
...
|
||||
- node = node1.example.com
|
||||
...
|
||||
```
|
||||
|
||||
Login to the cluster. Check the status of that node:
|
||||
|
||||
```shell
|
||||
$ kubectl get node $NODE -o yaml
|
||||
```
|
||||
|
||||
The output should describe why the node is not reachable.
|
||||
|
||||
Common failure scenarios:
|
||||
|
||||
- disruptive software upgrades
|
||||
- network patitioning due to hardware failures
|
||||
- firewall rules
|
||||
- virtual machines suspended due to storage area network problems
|
||||
- system crashes / freezes due to software or hardware malfunctions
|
||||
|
||||
## Mitigation
|
||||
|
||||
In case of maintenance ensure to [cordon and drain node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/).
|
||||
|
||||
In other cases ensure storage and networking redundancy if applicable.
|
||||
|
||||
See [KubeNode](https://kubernetes.io/docs/concepts/architecture/nodes/#condition)
|
||||
See [node problem detector](https://github.com/kubernetes/node-problem-detector)
|
||||
See [Watchdog timer](https://en.wikipedia.org/wiki/Watchdog_timer)
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: Kube Persistent Volume Errors
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubePersistentVolumeErrors
|
||||
|
||||
## Meaning
|
||||
|
||||
PersistentVolume is having issues with provisioning.
|
||||
|
||||
## Impact
|
||||
|
||||
Volue may be unavailable or have data erors (corrupted storage).
|
||||
|
||||
Service degradation, data loss.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check PV events via `kubectl describe pv $PV`.
|
||||
- Check storage provider for logs.
|
||||
- Check storage quotas in the cloud.
|
||||
|
||||
## Mitigation
|
||||
|
||||
In happy scenario storage is just not provisioned as fast as expected.
|
||||
In worst scenario there is data corruption or data loss. Restore from backup.
|
||||
@@ -1,14 +1,72 @@
|
||||
---
|
||||
title: Kube Persistent Volume Filling Up
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubePersistentVolumeFillingUp
|
||||
|
||||
There can be various reasons why a volume is filling up. This runbook does not cover application specific reasons, only mitigations for volumes that are legitimately filling.
|
||||
## Meaning
|
||||
|
||||
## Volume resizing
|
||||
There can be various reasons why a volume is filling up.
|
||||
This runbook does not cover application specific reasons, only mitigations
|
||||
for volumes that are legitimately filling.
|
||||
|
||||
If volume resizing is available, it's easiest to increase the capacity of the volume.
|
||||
As always refer to recommended scenarios for given service.
|
||||
|
||||
To check if volume expansion is available, run this with your namespace and PVC-name replaced.
|
||||
## Impact
|
||||
|
||||
```bash
|
||||
Service degradation, switching to read only mode.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
Check app usage in time.
|
||||
Check if there are any configurations such as snapshotting, automatic data retention.
|
||||
|
||||
## Mitigation
|
||||
|
||||
### Data retention
|
||||
|
||||
Deleting no longer needed data is the fastest and the cheapest solution.
|
||||
|
||||
Ask the service owner if specific old data can be deleted.
|
||||
Enable data retention especially for snapshots, if possible.
|
||||
|
||||
### Data export
|
||||
|
||||
If data is not needed in the service but needs to be processed later
|
||||
then send it to somewhere else, for example to S3 bucket.
|
||||
|
||||
### Data rebalance in the cluster
|
||||
|
||||
Some services automatically rebalance data on the cluster when one node
|
||||
fills up.
|
||||
Some allow to rebalance data across existing nodes, the other may require
|
||||
adding new nodes.
|
||||
If this is supported then increase number of replicas and wait for data
|
||||
migration or trigger it manually.
|
||||
|
||||
Example services that support this:
|
||||
|
||||
- cassandra
|
||||
- ceph
|
||||
- elasticsearch/opensearch
|
||||
- gluster
|
||||
- hadoop
|
||||
- kafka
|
||||
- minio
|
||||
|
||||
**Notice**: some services may require special scaling conditions such as
|
||||
adding twice more nodes than exist now.
|
||||
|
||||
### Direct Volume resizing
|
||||
|
||||
If volume resizing is available, it's easiest to increase the capacity of
|
||||
the volume.
|
||||
|
||||
To check if volume expansion is available, run this with your namespace
|
||||
and PVC-name replaced.
|
||||
|
||||
```shell
|
||||
$ kubectl get storageclass `kubectl -n <my-namespace> get pvc <my-pvc> -ojson | jq -r '.spec.storageClassName'`
|
||||
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
||||
standard (default) kubernetes.io/gce-pd Delete Immediate true 28d
|
||||
@@ -18,34 +76,71 @@ In this case `ALLOWVOLUMEEXPANSION` is true, so we can make use of the feature.
|
||||
|
||||
To resize the volume run:
|
||||
|
||||
```bash
|
||||
```shell
|
||||
$ kubectl -n <my-namespace> edit pvc <my-pvc>
|
||||
```
|
||||
|
||||
And edit `.spec.resources.requests.storage` to the new desired storage size. Eventually the PVC status will say "Waiting for user to (re-)start a pod to finish file system resize of volume on node."
|
||||
And edit `.spec.resources.requests.storage` to the new desired storage size.
|
||||
Eventually the PVC status will say "Waiting for user to (re-)start a pod to
|
||||
finish file system resize of volume on node."
|
||||
|
||||
You can check this with:
|
||||
|
||||
```bash
|
||||
```shell
|
||||
$ kubectl -n <my-namespace> get pvc <my-pvc>
|
||||
```
|
||||
|
||||
Once the PVC status says to restart the respective pod, run this to restart it (this automatically finds the pod that mounts the PVC and deletes it, if you know the pod name, you can also just simply delete that pod):
|
||||
Once the PVC status says to restart the respective pod, run this to restart it
|
||||
(this automatically finds the pod that mounts the PVC and deletes it,
|
||||
if you know the pod name, you can also just simply delete that pod):
|
||||
|
||||
```bash
|
||||
```shell
|
||||
$ kubectl -n <my-namespace> delete pod `kubectl -n <my-namespace> get pod -ojson | jq -r '.items[] | select(.spec.volumes[] .persistentVolumeClaim.claimName=="<my-pvc>") | .metadata.name'`
|
||||
```
|
||||
|
||||
## Migrate data to a new, larger volume
|
||||
### Migrate data to a new, larger volume
|
||||
|
||||
When resizing is not available and the data is not safe to be deleted, then the only way is to create a larger volume and migrate the data.
|
||||
When resizing is not available and the data is not safe to be deleted,
|
||||
then the only way is to create a larger volume and migrate the data.
|
||||
|
||||
TODO
|
||||
|
||||
## Purge volume
|
||||
### Purge volume
|
||||
|
||||
When the data is ephemeral and volume expansion is not available, it may be best to purge the volume.
|
||||
When the data is ephemeral and volume expansion is not available,
|
||||
it may be best to purge the volume.
|
||||
|
||||
WARNING/DANGER: This will permanently delete the data on the volume. Performing these steps is your responsibility.
|
||||
**WARNING/DANGER**: This will permanently delete the data on the volume.
|
||||
Performing these steps is your responsibility.
|
||||
|
||||
TODO
|
||||
|
||||
### Migrate data to new, larger instance pool in the same cluster
|
||||
|
||||
In very specific scenarios it is better to schedule data migration in the
|
||||
same cluster but to a new instances.
|
||||
This is sometimes hard to accomplish due to the way how certain resources
|
||||
are managed in kubernetes.
|
||||
|
||||
In general procedure is like this:
|
||||
|
||||
- add new nodes with bigger capacity than existing cluster
|
||||
- trigger data migration
|
||||
- scale in to 0 old instance pool and after that delete it.
|
||||
|
||||
### Migrate data to new, larger cluster
|
||||
|
||||
This is most common scenario, but is much more expensive and may be a bit
|
||||
time consuming.
|
||||
Also sometimes this causes split brain issues when writing.
|
||||
|
||||
In general procedure is like this, this is only a suggestion, though:
|
||||
|
||||
- create data snapshot on existing cluster
|
||||
- add new cluster with bigger capacity than existing cluster
|
||||
- start data restore on new cluster based on the snapshot
|
||||
- switch old cluster to read only mode
|
||||
- reconfigure networking to point to new cluster
|
||||
- trigger data migration from old cluster to new cluster to sync difference
|
||||
between snapshot and latest writes
|
||||
- remove old cluster
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Kube Pod Crash Looping
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubePodCrashLooping
|
||||
|
||||
## Meaning
|
||||
|
||||
Pod is in CrashLoop which means the app dies or is unresponsive and
|
||||
kubernetes tries to restart it automatically.
|
||||
|
||||
## Impact
|
||||
|
||||
Service degradation or unavailability.
|
||||
Inability to do rolling upgrades.
|
||||
Certain apps will not perform required tasks such as data migrations.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check template via `kubectl -n $NAMESPACE get pod $POD`.
|
||||
- Check pod events via `kubectl -n $NAMESPACE describe pod $POD`.
|
||||
- Check pod logs via `kubectl -n $NAMESPACE logs $POD -c $CONTAINER`
|
||||
- Check pod template parameters such as:
|
||||
- pod priority
|
||||
- resources - maybe it tries to use unavailable resource, such as GPU but
|
||||
there is limited number of nodes with GPU
|
||||
- readiness and liveness probes may be incorrect - wrong port or command,
|
||||
check is failing too fast due to short timeout for response
|
||||
|
||||
Other things to check:
|
||||
|
||||
- app responding extremely slow due to resource constraints such as memory too
|
||||
low, not enough CPU which is required on start
|
||||
- app waits for other services to start, such as database
|
||||
- misconfiguration causing app crash on start
|
||||
- missing files such as configmaps/secrets/volumes
|
||||
- read only filesystem
|
||||
- wrong user permissions in container
|
||||
- lack of special container capabilities (securityContext)
|
||||
- app is executed in different directory than expected
|
||||
(for example WORKDIR from Docerkfile is not used in OpenShift)
|
||||
|
||||
## Mitigation
|
||||
|
||||
Talk with developers or read documentation about the app, ensure to define
|
||||
sane default values to start the app.
|
||||
|
||||
See [Debugging Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-pods)
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: Kube Pod Not Ready
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubePodNotReady
|
||||
|
||||
## Meaning
|
||||
|
||||
Pod has been in a non-ready state for more than 15 minutes.
|
||||
|
||||
State Running but not ready means readiness probe fails.
|
||||
State Pending means pod can not be created for specific namespace and node.
|
||||
|
||||
<details>
|
||||
<summary>Full context</summary>
|
||||
|
||||
Pod failed to reach reay state, depending on the readiness/liveness probes.
|
||||
See [pod-lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
|
||||
|
||||
</details>
|
||||
|
||||
## Impact
|
||||
|
||||
Service degradation or unavailability.
|
||||
Pod not attached to service, thus not getting any traffic.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check template via `kubectl -n $NAMESPACE get pod $POD`.
|
||||
- Check pod events via `kubectl -n $NAMESPACE describe pod $POD`.
|
||||
- Check pod logs via `kubectl -n $NAMESPACE logs $POD -c $CONTAINER`
|
||||
- Check pod template parameters such as:
|
||||
- pod priority
|
||||
- resources - maybe it tries to use unavailable resource, such as GPU but
|
||||
there is limited number of nodes with GPU
|
||||
- readiness and liveness probes may be incorrect - wrong port or command,
|
||||
check is failing too fast due to short timeout for response
|
||||
- stuck or long running init containers
|
||||
|
||||
Other things to check:
|
||||
|
||||
- app responding extremely slow due to resource constraints such as memory too
|
||||
low, not enough CPU which is required on start
|
||||
- app waits for other services to start, such as database
|
||||
- misconfiguration causing app crash on start
|
||||
- missing files such as configmaps/secrets/volumes
|
||||
- read only filesystem
|
||||
- wrong user permissions in container
|
||||
- lack of special container capabilities (securityContext)
|
||||
- app is executed in different directory than expected
|
||||
(for example WORKDIR from Docerkfile is not used in OpenShift)
|
||||
|
||||
## Mitigation
|
||||
|
||||
Talk with developers or read documentation about the app, ensure to define
|
||||
sane default values to start the app.
|
||||
|
||||
See [Debugging Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-pods)
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: KubeProxy Down
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubeProxyDown
|
||||
|
||||
## Meaning
|
||||
|
||||
KubeProxy has disappeared from Prometheus target discovery.
|
||||
|
||||
## Impact
|
||||
|
||||
kube-proxy is a network proxy that runs on each node in your cluster,
|
||||
implementing part of the Kubernetes Service concept.
|
||||
|
||||
kube-proxy maintains network rules on nodes.
|
||||
These network rules allow network communication to your Pods
|
||||
from network sessions inside or outside of your cluster.
|
||||
|
||||
kube-proxy uses the operating system packet filtering layer if
|
||||
there is one and it's available. Otherwise, kube-proxy forwards the traffic
|
||||
itself.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
Check networking on the node.
|
||||
Check firewall on the node.
|
||||
Investigate kube proxy logs.
|
||||
Investigate NetworkPolicies if prometheus/kubeproxy was not filtered out.
|
||||
|
||||
## Mitigation
|
||||
|
||||
TODO
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Kube Quota Almost Full
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# KubeQuotaAlmostFull
|
||||
|
||||
## Meaning
|
||||
|
||||
Cluster reaches to the allowed limits for given namespace.
|
||||
|
||||
## Impact
|
||||
|
||||
In the future deployments may not be possbile.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check resource usage for the namespace in given time span
|
||||
|
||||
## Mitigation
|
||||
|
||||
- Review existing quota for given namespace and adjust it accordingly.
|
||||
- Review resources used by the quota and fine tune them.
|
||||
- Continue with standard capacity planning procedures.
|
||||
- See [Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/)
|
||||
|
||||
Reference in New Issue
Block a user