diff --git a/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md new file mode 100644 index 0000000..f0cc848 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md @@ -0,0 +1,35 @@ +--- +title: Alertmanager Cluster Crashlooping +weight: 20 +--- + +# AlertmanagerClusterCrashlooping + +## Meaning + +Half or more of the Alertmanager instances within the same cluster are crashlooping. + +## Impact + +Alerts could be notified multiple time unless pods are crashing to fast and no alerts can be sent. + +## Diagnosis + +```bash +kubectl get pod -l app=alertmanager + +NAMESPACE NAME READY STATUS RESTARTS AGE +default alertmanager-main-0 1/2 CrashLoopBackOff 37107 2d +default alertmanager-main-1 2/2 Running 0 43d +default alertmanager-main-2 2/2 Running 0 43d +``` + +Find the root cause by looking to events for a given pod/deployement + +``` +kubectl get events --field-selector involvedObject.name=alertmanager-main-0 +``` + +## Mitigation + +Make sure pods have enough resources (CPU, MEM) to work correctly. diff --git a/content/runbooks/alertmanager/AlertmanagerClusterDown.md b/content/runbooks/alertmanager/AlertmanagerClusterDown.md new file mode 100644 index 0000000..6048601 --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterDown.md @@ -0,0 +1,28 @@ +--- +title: Alertmanager Cluster Down +weight: 20 +--- + +# AlertmanagerClusterDown + +## Meaning + +Half or more of the Alertmanager instances within the same cluster are down. + +## Impact + +You have an unstable cluster, if everything goes wrong you will lose the whole cluster. + +## Diagnosis + +Verify why pods are not running. +You can get a big picture with `events`. + +```bash +$ kubectl get events --field-selector involvedObject.kind=Pod | grep alertmanager +``` + +## Mitigation + +There are no cheap options to mitigate this risk. +Verifying any new changes in preprod before production environment should improve stability. diff --git a/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md new file mode 100644 index 0000000..7e04b7e --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerClusterFailedToSendAlerts.md @@ -0,0 +1,22 @@ +--- +title: Alertmanager Cluster Failed To Send Alerts +weight: 20 +--- + +# AlertmanagerClusterFailedToSendAlerts + +## Meaning + +All instances failed to send notification to an integration. + +## Impact + +You will not receive a notification when an alert is raised. + +## Diagnosis + +No alerts are received at the integration level from the cluster. + +## Mitigation + +Depending on the integration, correct the integration with the faulty instance (network, authorization token, firewall...) diff --git a/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md new file mode 100644 index 0000000..3cc5e6b --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerConfigInconsistent.md @@ -0,0 +1,24 @@ +--- +title: Alertmanager ConfigInconsistent +weight: 20 +--- + +# AlertmanagerConfigInconsistent + +## Meaning + +The configuration between instances inside a cluster is inconsistent. + +## Impact + +Configuration inconsistency can be multiple and impact is hard to predict. +Nevertheless, in most cases the alert might be lost or routed to the incorrect integration. + +## Diagnosis + +Run a `diff` tool between all `alertmanager.yml` that are deployed to find what is wrong. +You could run a job within your CI to avoid this issue in the future. + +## Mitigation + +Delete the incorrect secret and deploy the correct one. diff --git a/content/runbooks/alertmanager/AlertmanagerFailedReload.md b/content/runbooks/alertmanager/AlertmanagerFailedReload.md index 55b9f0b..aa4ad7e 100644 --- a/content/runbooks/alertmanager/AlertmanagerFailedReload.md +++ b/content/runbooks/alertmanager/AlertmanagerFailedReload.md @@ -1,3 +1,8 @@ +--- +title: AlertmanagerFailedReload +weight: 20 +--- + # AlertmanagerFailedReload ## Meaning @@ -8,16 +13,23 @@ configuration for a certain period. ## Impact -Alerts for cluster components may not be delivered as expected. +The impact depends on the type of the error you will find in the logs. +Most of the time, previous configuration is still working, thanks to multiple instances, so avoid deleting existing pods. ## Diagnosis -Check the logs for the `alertmanager-main` pods in the `monitoring` namespace: +Verify if there is an error in `config-reloader` container logs. +Here an example with network issues. -```console -$ kubectl -n monitoring logs -l 'alertmanager=main' +```bash +$ kubectl logs sts/alertmanager-main -c config-reloader + +level=error ts=2021-09-24T11:24:52.69629226Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9093/alertmanager/-/reload\": dial tcp [::1]:9093: connect: connection refused" ``` +You can also verify directly `alertmanager.yaml` file (default: `/etc/alertmanager/config/alertmanager.yaml`). + ## Mitigation -The resolution depends on the particular issue reported in the logs. +Running [amtool check-config alertmanager.yaml](https://github.com/prometheus/alertmanager#amtool) on your configuration file will help you detect problem related to syntax. +You could also rollback `alertmanager.yaml` to the previous version in order to get back to a stable version. diff --git a/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md new file mode 100644 index 0000000..b41c26a --- /dev/null +++ b/content/runbooks/alertmanager/AlertmanagerFailedToSendAlerts.md @@ -0,0 +1,35 @@ +--- +title: Alertmanager Failed To Send Alerts +weight: 20 +--- + +# AlertmanagerFailedToSendAlerts + +## Meaning + +At least one instance is unable to routed alert to the corresponding integration. + +## Impact + +No impact since another instance should be able to send the notification, unless `AlertmanagerClusterFailedToSendAlerts` is also triggerd for the same integration. + +## Diagnosis + +Verify the amount of failed notification per alert-manager-[instance] for a specific integration. + +You can look metrics exposed in prometheus console using promQL. For exemple the following query will display the number of failed notifications per instance for pager duty integration. We have 3 instances involved in the example bellow. + +``` +rate(alertmanager_notifications_total{integration="pagerduty"}[5m]) +``` + +![image](https://user-images.githubusercontent.com/3153333/143552468-ff573f1a-19a6-44ea-9c85-631687d01bf9.png) + + +## Mitigation + +Depending on the integration, you can have a look to alert-manager logs and act (network, authorization token, firewall...) + +``` +kubectl -n monitoring logs -l 'alertmanager=main' -c alertmanager +```