Merge pull request #19 from nvtkaszpir/runbooks-general

Update runbooks/general

content/runbooks/general/InfoInhibitor.md

@@ -0,0 +1,34 @@
+---
+title: Info Inhibitor
+weight: 20
+---
+
+# InfoInhibitor
+
+## Meaning
+
+This is an alert that is used to inhibit info alerts.
+
+By themselves, the info-level alerts are sometimes very noisy,
+but they are relevant when combined with other alerts.
+
+<details>
+<summary>Full context</summary>
+
+More information about the alert and design considerations can be found in a [kube-prometheus issue](https://github.com/prometheus-operator/kube-prometheus/issues/861)
+</details>
+## Impact
+
+Alert does not have any impact and it is used only as a workaround to a missing feature in alertmanager.
+
+## Diagnosis
+
+This alert fires whenever there's a `severity="info"` alert,
+and stops firing when another alert with severity of `warning` or
+`critical` starts firing on the same namespace.
+
+
+## Mitigation
+
+This alert should be routed to a null receiver and configured to inhibit
+alerts with `severity="info"`. Such configuration is available at https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/alertmanager-secret.yaml

content/runbooks/general/NodeNetworkInterfaceFlapping.md

@@ -0,0 +1,24 @@
+---
+title: Node Network Interface Flapping
+weight: 20
+---
+
+# NodeNetworkInterfaceFlapping
+
+## Meaning
+
+Network interface is often changing its status
+
+## Impact
+
+Applications on the node may no longer be able to operate with other services.
+Network attached storage performance issues or even data loss.
+
+## Diagnosis
+
+Investigate networkng issues on the node and to connected hardware.
+Check physical cables, check networking firewall rules and so on.
+
+## Mitigation
+
+Cordon and drain node to migrate apps from it.

content/runbooks/general/Watchdog.md

@@ -0,0 +1,28 @@
+---
+title: Watchdog
+weight: 20
+---
+
+# Watchdog
+
+## Meaning
+
+This is an alert meant to ensure that the entire alerting pipeline is functional.
+This alert is always firing, therefore it should always be firing in Alertmanager
+and always fire against a receiver.
+
+## Impact
+
+If not firing then it should alert external systems that this alerting system
+is no longer working.
+
+## Diagnosis
+
+Misconfigured alertmanager, bad credentials, bad endpoint, firewalls..
+Check alertmanager logs.
+
+## Mitigation
+
+There are integrations with various notification
+mechanisms that send a notification when this alert is not firing.
+For example the `DeadMansSnitch` integration in PagerDuty.