github/runbooks
1
0
Fork 0
mirror of https://github.com/prometheus-operator/runbooks.git synced 2026-08-27 12:37:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update AlertmanagerFailedToSendAlerts.md

Browse Source
This commit is contained in:
Tigran Tch
2021-11-26 09:47:59 +01:00
committed by GitHub
parent 5fbac9e822
commit 5a57a87ba3
1 changed files with 16 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md
+16 -3
View File
@@ -11,12 +11,25 @@ At least one instance is unable to routed alert to the corresponding integration
## Impact
No impact since another instance will be able to send the notification.
No impact since another instance should be able to send the notification, unless `AlertmanagerClusterFailedToSendAlerts` is also triggerd for the same integration.
## Diagnosis
Verify that alerts send by each instance have equivalent alert distribution per integration.
Verify the amount of failed notification per alert-manager-[instance] for a specific integration.
You can look metrics exposed in prometheus console using promQL. For exemple the following query will display the number of failed notifications per instance for pager duty integration. We have 3 instances involved in the example bellow.
```
rate(alertmanager_notifications_total{integration="pagerduty"}[5m])
```
![image](https://user-images.githubusercontent.com/3153333/143552468-ff573f1a-19a6-44ea-9c85-631687d01bf9.png)
## Mitigation
Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...)
Depending on the integration, you can have a look to alert-manager logs and act (network, authorization token, firewall...)
```
kubectl -n monitoring logs -l 'alertmanager=main' -c alertmanager
```