github/runbooks
1
0
Fork 0
mirror of https://github.com/prometheus-operator/runbooks.git synced 2026-08-27 12:37:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2 from swissquote/AlertmanagerFailedToSendAlerts

Browse Source 
add doc to AlertmanagerFailedToSendAlerts
This commit is contained in:
Tigran Tch
2021-11-26 10:18:53 +01:00
committed by GitHub
parent 82060d7106 5a57a87ba3
commit d175bbb21e
1 changed files with 35 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md
+35
View File
@@ -0,0 +1,35 @@
---
title: Alertmanager Failed To Send Alerts
weight: 20
---
# AlertmanagerFailedToSendAlerts
## Meaning
At least one instance is unable to routed alert to the corresponding integration.
## Impact
No impact since another instance should be able to send the notification, unless `AlertmanagerClusterFailedToSendAlerts` is also triggerd for the same integration.
## Diagnosis
Verify the amount of failed notification per alert-manager-[instance] for a specific integration.
You can look metrics exposed in prometheus console using promQL. For exemple the following query will display the number of failed notifications per instance for pager duty integration. We have 3 instances involved in the example bellow.
```
rate(alertmanager_notifications_total{integration="pagerduty"}[5m])
```
![image](https://user-images.githubusercontent.com/3153333/143552468-ff573f1a-19a6-44ea-9c85-631687d01bf9.png)
## Mitigation
Depending on the integration, you can have a look to alert-manager logs and act (network, authorization token, firewall...)
```
kubectl -n monitoring logs -l 'alertmanager=main' -c alertmanager
```