From 90f5b891a58498ff6979ce163e5f0c57153527eb Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Wed, 10 Nov 2021 15:08:27 +0100 Subject: [PATCH 01/15] doc added for AlertmanagerFailedReload --- .../alertmanager/AlertmanagerFailedReload.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerFailedReload.md diff --git a/content/runbooks/alertmanager/AlertmanagerFailedReload.md b/content/runbooks/alertmanager/AlertmanagerFailedReload.md new file mode 100644 index 0000000..436e376 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerFailedReload.md @@ -0,0 +1,33 @@ +--- +title: AlertmanagerFailedReload +weight: 20 +--- + +# AlertmanagerFailedReload + +## Meaning + +At least one worker failed to load the configuration. + +## Impact + +The impact depends on the type of the error you will find in the logs. +Most of the time, previous configuration is still working, so avoid deleting existing pods. + +## Diagnosis + +Verify if there is an error in `config-reloader` container logs. +Here an example with network issues. + +```bash +$ kubectl logs sts/alertmanager-main -c config-reloader + +level=error ts=2021-09-24T11:24:52.69629226Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9093/alertmanager/-/reload\": dial tcp [::1]:9093: connect: connection refused" +``` + +You can also verify directly `alertmanager.yaml` file (default: `/etc/alertmanager/config/alertmanager.yaml`). + +## Mitigation + +Running [amtool check-config alertmanager.yaml](https://github.com/prometheus/alertmanager#amtool) on your configuration file will help you detect problem related to syntax. +You could also rollback `alertmanager.yaml` to the previous version in order to get back to a stable version. From a25b3afd4ad1189b4bb7dbf1ac0bb05afc2e7a11 Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Thu, 11 Nov 2021 13:58:40 +0100 Subject: [PATCH 02/15] add doc to AlertmanagerFailedToSendAlerts --- .../AlertmanagerFailedToSendAlerts.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md diff --git a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md new file mode 100644 index 0000000..f64cf65 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md @@ -0,0 +1,21 @@ +--- +title: Alertmanager Failed To Send Alerts weight: 20 +--- + +# AlertmanagerFailedToSendAlerts + +## Meaning + +At least one instance is unable to routed alert to the corresponding integration. + +## Impact + +No impact since another instance will be able to send the notification. + +## Diagnosis + +Verify that alerts send by each instance have equivalent alert distribution per integration. + +## Mitigation + +Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) From 5fbac9e82239ea1971eff98777b25e8cddf6fbfd Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Thu, 11 Nov 2021 14:11:32 +0100 Subject: [PATCH 03/15] Update AlertmanagerFailedToSendAlerts.md --- .../runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md index f64cf65..c5fdf3f 100644 --- a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md +++ b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md @@ -1,5 +1,6 @@ --- -title: Alertmanager Failed To Send Alerts weight: 20 +title: Alertmanager Failed To Send Alerts +weight: 20 --- # AlertmanagerFailedToSendAlerts From f6f31eba186e4eb48b1caf9ce52c64956200e2d3 Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Thu, 11 Nov 2021 14:28:45 +0100 Subject: [PATCH 04/15] add doc for AlertmanagerClusterFailedToSendAlerts --- .../AlertmanagerClusterFailedToSendAlerts.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md diff --git a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md new file mode 100644 index 0000000..f70b63a --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md @@ -0,0 +1,22 @@ +--- +title: Alertmanager Cluster Failed To Send Alerts +weight: 20 +--- + +# Alertmanager Cluster Failed To Send Alerts + +## Meaning + +All instances failed to send notification to a critical integration. + +## Impact + +You will not receive notification when an alert is raised. + +## Diagnosis + +No alerts are received at the integration level from the cluster. + +## Mitigation + +Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) \ No newline at end of file From a35d153274a124e2d90cd34f500e4e548e6e4ee0 Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Fri, 12 Nov 2021 09:01:57 +0100 Subject: [PATCH 05/15] same alert exists for non critical --- .../alertmanager/AlertmanagerClusterFailedToSendAlerts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md index f70b63a..de831e7 100644 --- a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md +++ b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md @@ -7,7 +7,7 @@ weight: 20 ## Meaning -All instances failed to send notification to a critical integration. +All instances failed to send notification to an integration. ## Impact From 2f88d7d32980686f7aed12a188dfe7e2bf7ebe93 Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Fri, 12 Nov 2021 12:08:47 +0100 Subject: [PATCH 06/15] add doc for AlertmanagerConfigInconsistent --- .../AlertmanagerConfigInconsistent.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md diff --git a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md new file mode 100644 index 0000000..ce9cf86 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md @@ -0,0 +1,24 @@ +--- +title: Alertmanager ConfigInconsistent +weight: 20 +--- + +# Alertmanager Config Inconsistent + +## Meaning + +The configuration between instances inside a cluster is inconsistent. + +## Impact + +Configuration inconsistency can be multiple and impact is hard to predict. +Nevertheless, most of the case the alert might be lost or routed to the incorrect integration. + +## Diagnosis + +Run a `diff` tool between all `alertmanager.yml` that are deployed to find what is wrong. +You could run a job within your CI to avoid this issue in the future. + +## Mitigation + +Delete the incorrect secret and deploy the correct one. From 7fb999b4bcbe92337336ab265096f0941ad1a2fe Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Fri, 12 Nov 2021 13:29:48 +0100 Subject: [PATCH 07/15] Update AlertmanagerClusterFailedToSendAlerts.md --- .../alertmanager/AlertmanagerClusterFailedToSendAlerts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md index de831e7..2ecbb5c 100644 --- a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md +++ b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md @@ -3,7 +3,7 @@ title: Alertmanager Cluster Failed To Send Alerts weight: 20 --- -# Alertmanager Cluster Failed To Send Alerts +# AlertmanagerClusterFailedToSendAlerts ## Meaning @@ -19,4 +19,4 @@ No alerts are received at the integration level from the cluster. ## Mitigation -Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) \ No newline at end of file +Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) From 71e2271213225667afe9cb404ebae156c8ec2b9b Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Fri, 12 Nov 2021 13:30:09 +0100 Subject: [PATCH 08/15] Update AlertmanagerConfigInconsistent.md --- content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md index ce9cf86..3a507da 100644 --- a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md +++ b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md @@ -3,7 +3,7 @@ title: Alertmanager ConfigInconsistent weight: 20 --- -# Alertmanager Config Inconsistent +# AlertmanagerConfigInconsistent ## Meaning From cbf59bda0f7804185ae60a307a3d27ee30baa6de Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Fri, 12 Nov 2021 16:37:08 +0100 Subject: [PATCH 09/15] add doc for AlertmanagerClusterDown --- .../alertmanager/AlertmanagerClusterDown.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerClusterDown.md diff --git a/content/runbooks/alertmanager/AlertmanagerClusterDown.md b/content/runbooks/alertmanager/AlertmanagerClusterDown.md new file mode 100644 index 0000000..6048601 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterDown.md @@ -0,0 +1,28 @@ +--- +title: Alertmanager Cluster Down +weight: 20 +--- + +# AlertmanagerClusterDown + +## Meaning + +Half or more of the Alertmanager instances within the same cluster are down. + +## Impact + +You have an unstable cluster, if everything goes wrong you will lose the whole cluster. + +## Diagnosis + +Verify why pods are not running. +You can get a big picture with `events`. + +```bash +$ kubectl get events --field-selector involvedObject.kind=Pod | grep alertmanager +``` + +## Mitigation + +There are no cheap options to mitigate this risk. +Verifying any new changes in preprod before production environment should improve stability. From 558a0f9b72b7973ae09940ec783d735331458bb4 Mon Sep 17 00:00:00 2001 From: Tigran Tch Date: Fri, 12 Nov 2021 16:54:52 +0100 Subject: [PATCH 10/15] add doc AlertmanagerClusterCrashlooping --- .../AlertmanagerClusterCrashlooping.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md diff --git a/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md new file mode 100644 index 0000000..78e0c6c --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md @@ -0,0 +1,29 @@ +--- +title: Alertmanager Cluster Crashlooping +weight: 20 +--- + +# AlertmanagerClusterCrashlooping + +## Meaning + +Half or more of the Alertmanager instances within the same cluster are crashlooping. + +## Impact + +Alerts could be notified multiple time unless pods are crashing to fast and no alerts can be sent. + +## Diagnosis + +```bash +kubectl get pod -l app=alertmanager + +NAMESPACE NAME READY STATUS RESTARTS AGE +default alertmanager-main-0 1/2 CrashLoopBackOff 37107 2d +default alertmanager-main-1 2/2 Running 0 43d +default alertmanager-main-2 2/2 Running 0 43d +``` + +## Mitigation + +Make sure pods have enough resources (CPU, MEM) to work correctly. From 3ac613a118593d8c1ea7c8e6c65e011b9c6a9613 Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Thu, 25 Nov 2021 16:27:21 +0100 Subject: [PATCH 11/15] Update content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md Co-authored-by: Drew Boswell --- content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md index 3a507da..3cc5e6b 100644 --- a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md +++ b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md @@ -12,7 +12,7 @@ The configuration between instances inside a cluster is inconsistent. ## Impact Configuration inconsistency can be multiple and impact is hard to predict. -Nevertheless, most of the case the alert might be lost or routed to the incorrect integration. +Nevertheless, in most cases the alert might be lost or routed to the incorrect integration. ## Diagnosis From d9c25a2c6146340af15d88f7917346cabdd9a733 Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Thu, 25 Nov 2021 16:27:36 +0100 Subject: [PATCH 12/15] Update content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md Co-authored-by: Drew Boswell --- .../alertmanager/AlertmanagerClusterFailedToSendAlerts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md index 2ecbb5c..7e04b7e 100644 --- a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md +++ b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md @@ -11,7 +11,7 @@ All instances failed to send notification to an integration. ## Impact -You will not receive notification when an alert is raised. +You will not receive a notification when an alert is raised. ## Diagnosis From 1910ab97bd5ae296bff9a45d6bd7343e9d2ffc61 Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Fri, 26 Nov 2021 09:21:54 +0100 Subject: [PATCH 13/15] Update AlertmanagerFailedReload.md --- content/runbooks/alertmanager/AlertmanagerFailedReload.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/runbooks/alertmanager/AlertmanagerFailedReload.md b/content/runbooks/alertmanager/AlertmanagerFailedReload.md index 436e376..0097a5a 100644 --- a/content/runbooks/alertmanager/AlertmanagerFailedReload.md +++ b/content/runbooks/alertmanager/AlertmanagerFailedReload.md @@ -7,12 +7,12 @@ weight: 20 ## Meaning -At least one worker failed to load the configuration. +The alert `AlertmanagerFailedReload` is triggered when the Alertmanager instance for the cluster monitoring stack has consistently failed to reload its configuration for a certain period. ## Impact The impact depends on the type of the error you will find in the logs. -Most of the time, previous configuration is still working, so avoid deleting existing pods. +Most of the time, previous configuration is still working, thanks to multiple instances, so avoid deleting existing pods. ## Diagnosis From 5a57a87ba36da79c1dd077050d34360d9f0fb80a Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Fri, 26 Nov 2021 09:47:59 +0100 Subject: [PATCH 14/15] Update AlertmanagerFailedToSendAlerts.md --- .../AlertmanagerFailedToSendAlerts.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md index c5fdf3f..b41c26a 100644 --- a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md +++ b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md @@ -11,12 +11,25 @@ At least one instance is unable to routed alert to the corresponding integration ## Impact -No impact since another instance will be able to send the notification. +No impact since another instance should be able to send the notification, unless `AlertmanagerClusterFailedToSendAlerts` is also triggerd for the same integration. ## Diagnosis -Verify that alerts send by each instance have equivalent alert distribution per integration. +Verify the amount of failed notification per alert-manager-[instance] for a specific integration. + +You can look metrics exposed in prometheus console using promQL. For exemple the following query will display the number of failed notifications per instance for pager duty integration. We have 3 instances involved in the example bellow. + +``` +rate(alertmanager_notifications_total{integration="pagerduty"}[5m]) +``` + +![image](https://user-images.githubusercontent.com/3153333/143552468-ff573f1a-19a6-44ea-9c85-631687d01bf9.png) + ## Mitigation -Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) +Depending on the integration, you can have a look to alert-manager logs and act (network, authorization token, firewall...) + +``` +kubectl -n monitoring logs -l 'alertmanager=main' -c alertmanager +``` From fac64f1b1ba5ff1371ca65de70d509c639a413ad Mon Sep 17 00:00:00 2001 From: Tigran Tch <3153333+NargiT@users.noreply.github.com> Date: Fri, 26 Nov 2021 09:57:06 +0100 Subject: [PATCH 15/15] Update AlertmanagerClusterCrashlooping.md --- .../alertmanager/AlertmanagerClusterCrashlooping.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md index 78e0c6c..f0cc848 100644 --- a/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md +++ b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md @@ -24,6 +24,12 @@ default alertmanager-main-1 2/2 Running 0 43d default alertmanager-main-2 2/2 Running 0 43d ``` +Find the root cause by looking to events for a given pod/deployement + +``` +kubectl get events --field-selector involvedObject.name=alertmanager-main-0 +``` + ## Mitigation Make sure pods have enough resources (CPU, MEM) to work correctly.