github/runbooks
1
0
Fork 0
mirror of https://github.com/prometheus-operator/runbooks.git synced 2026-05-08 16:26:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #6 from swissquote/AlertmanagerClusterDown

Browse Source 
add doc for AlertmanagerClusterDown
This commit is contained in:
Tigran Tch
2021-11-25 16:27:05 +01:00
committed by GitHub
parent 6dd0987395 cbf59bda0f
commit 84e35e9cfd
1 changed files with 28 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
content/runbooks/alertmanager/AlertmanagerClusterDown.md Normal file
View File

@@ -0,0 +1,28 @@
---
title: Alertmanager Cluster Down
weight: 20
---
# AlertmanagerClusterDown
## Meaning
Half or more of the Alertmanager instances within the same cluster are down.
## Impact
You have an unstable cluster, if everything goes wrong you will lose the whole cluster.
## Diagnosis
Verify why pods are not running.
You can get a big picture with `events`.
```bash
$ kubectl get events --field-selector involvedObject.kind=Pod | grep alertmanager
```
## Mitigation
There are no cheap options to mitigate this risk.
Verifying any new changes in preprod before production environment should improve stability.