From 5a57a87ba36da79c1dd077050d34360d9f0fb80a Mon Sep 17 00:00:00 2001
From: Tigran Tch <3153333+NargiT@users.noreply.github.com>
Date: Fri, 26 Nov 2021 09:47:59 +0100
Subject: [PATCH] Update AlertmanagerFailedToSendAlerts.md

---
 .../AlertmanagerFailedToSendAlerts.md         | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md
index c5fdf3f..b41c26a 100644
--- a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md
+++ b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md
@@ -11,12 +11,25 @@ At least one instance is unable to routed alert to the corresponding integration
 
 ## Impact
 
-No impact since another instance will be able to send the notification.
+No impact since another instance should be able to send the notification, unless `AlertmanagerClusterFailedToSendAlerts` is also triggerd for the same integration.
 
 ## Diagnosis
 
-Verify that alerts send by each instance have equivalent alert distribution per integration.
+Verify the amount of failed notification per alert-manager-[instance] for a specific integration.
+
+You can look metrics exposed in prometheus console using promQL. For exemple the following query will display the number of failed notifications per instance for pager duty integration. We have 3 instances involved in the example bellow.
+
+```
+rate(alertmanager_notifications_total{integration="pagerduty"}[5m])
+```
+
+![image](https://user-images.githubusercontent.com/3153333/143552468-ff573f1a-19a6-44ea-9c85-631687d01bf9.png)
+
 
 ## Mitigation
 
-Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...)
+Depending on the integration, you can have a look to alert-manager logs and act (network, authorization token, firewall...)
+
+```
+kubectl -n monitoring logs -l 'alertmanager=main' -c alertmanager
+```