Update runbooks/general

content/runbooks/general/InfoInhibitor.md

@@ -0,0 +1,27 @@
+---
+title: Info Inhibitor
+weight: 20
+---
+
+# InfoInhibitor
+
+## Meaning
+
+This is an alert that is used to inhibit info alerts.
+
+By themselves, the info-level alerts are sometimes very noisy,
+but they are relevant when combined with other alerts.
+          
+## Impact
+
+## Diagnosis
+
+This alert fires whenever there's a `severity="info"` alert,
+and stops firing when another alert with a severity of `warning` or
+`critical` starts firing on the same namespace.
+
+
+## Mitigation
+
+This alert should be routed to a null receiver and configured to inhibit
+alerts with `severity="info"`.

content/runbooks/general/NodeNetworkInterfaceFlapping.md

@@ -0,0 +1,24 @@
+---
+title: Node Network Interface Flapping
+weight: 20
+---
+
+# NodeNetworkInterfaceFlapping
+
+## Meaning
+
+Network interface is often changing its status
+
+## Impact
+
+Applications on the node may no longer be able to operate with other services.
+Network attached storage performance issues or even data loss.
+
+## Diagnosis
+
+Investigate networkng issues on the node and to connected hardware.
+Check physical cables, check networking firewall rules and so on.
+
+## Mitigation
+
+Cordon and drain node to migrate apps from it.

content/runbooks/general/Watchdog.md

@@ -0,0 +1,28 @@
+---
+title: Watchdog
+weight: 20
+---
+
+# Watchdog
+
+## Meaning
+
+This is an alert meant to ensure that the entire alerting pipeline is functional.
+This alert is always firing, therefore it should always be firing in Alertmanager
+and always fire against a receiver.
+
+## Impact
+
+If not firing then it should alert external systems that this alerting system
+is no longer working.
+
+## Diagnosis
+
+Misconfigured alertmanager, bad credentials, bad endpoint, firewalls..
+Check alertmanager logs.
+
+## Mitigation
+
+There are integrations with various notification
+mechanisms that send a notification when this alert is not firing.
+For example the `DeadMansSnitch` integration in PagerDuty.