diff --git a/content/runbooks/general/InfoInhibitor.md b/content/runbooks/general/InfoInhibitor.md new file mode 100644 index 0000000..fd073a3 --- /dev/null +++ b/content/runbooks/general/InfoInhibitor.md @@ -0,0 +1,34 @@ +--- +title: Info Inhibitor +weight: 20 +--- + +# InfoInhibitor + +## Meaning + +This is an alert that is used to inhibit info alerts. + +By themselves, the info-level alerts are sometimes very noisy, +but they are relevant when combined with other alerts. + +
+Full context + +More information about the alert and design considerations can be found in a [kube-prometheus issue](https://github.com/prometheus-operator/kube-prometheus/issues/861) +
+## Impact + +Alert does not have any impact and it is used only as a workaround to a missing feature in alertmanager. + +## Diagnosis + +This alert fires whenever there's a `severity="info"` alert, +and stops firing when another alert with severity of `warning` or +`critical` starts firing on the same namespace. + + +## Mitigation + +This alert should be routed to a null receiver and configured to inhibit +alerts with `severity="info"`. Such configuration is available at https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/alertmanager-secret.yaml diff --git a/content/runbooks/general/NodeNetworkInterfaceFlapping.md b/content/runbooks/general/NodeNetworkInterfaceFlapping.md new file mode 100644 index 0000000..7673719 --- /dev/null +++ b/content/runbooks/general/NodeNetworkInterfaceFlapping.md @@ -0,0 +1,24 @@ +--- +title: Node Network Interface Flapping +weight: 20 +--- + +# NodeNetworkInterfaceFlapping + +## Meaning + +Network interface is often changing its status + +## Impact + +Applications on the node may no longer be able to operate with other services. +Network attached storage performance issues or even data loss. + +## Diagnosis + +Investigate networkng issues on the node and to connected hardware. +Check physical cables, check networking firewall rules and so on. + +## Mitigation + +Cordon and drain node to migrate apps from it. diff --git a/content/runbooks/general/Watchdog.md b/content/runbooks/general/Watchdog.md new file mode 100644 index 0000000..4f377db --- /dev/null +++ b/content/runbooks/general/Watchdog.md @@ -0,0 +1,28 @@ +--- +title: Watchdog +weight: 20 +--- + +# Watchdog + +## Meaning + +This is an alert meant to ensure that the entire alerting pipeline is functional. +This alert is always firing, therefore it should always be firing in Alertmanager +and always fire against a receiver. + +## Impact + +If not firing then it should alert external systems that this alerting system +is no longer working. + +## Diagnosis + +Misconfigured alertmanager, bad credentials, bad endpoint, firewalls.. +Check alertmanager logs. + +## Mitigation + +There are integrations with various notification +mechanisms that send a notification when this alert is not firing. +For example the `DeadMansSnitch` integration in PagerDuty.