From b7867287927ae16fca3d0d2217e9cb6bdc344dfd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Socho=C5=84?= Date: Fri, 18 Feb 2022 20:19:27 +0100 Subject: [PATCH] Update runbooks/prometheus --- .../prometheus/PrometheusBadConfig.md | 21 +++++++++--- .../PrometheusDuplicateTimestamps.md | 21 +++++++++--- ...heusErrorSendingAlertsToAnyAlertmanager.md | 24 ++++++++++++++ ...usErrorSendingAlertsToSomeAlertmanagers.md | 24 ++++++++++++++ .../prometheus/PrometheusLabelLimitHit.md | 22 +++++++++++++ .../PrometheusMissingRuleEvaluations.md | 22 +++++++++++++ .../PrometheusNotConnectedToAlertmanagers.md | 24 ++++++++++++++ .../PrometheusNotIngestingSamples.md | 22 +++++++++++++ .../PrometheusNotificationQueueRunningFull.md | 26 +++++++++++++++ .../PrometheusOutOfOrderTimestamps.md | 7 +++- .../PrometheusRemoteStorageFailures.md | 24 ++++++++++++++ .../prometheus/PrometheusRemoteWriteBehind.md | 29 ++++++++++++++++ .../PrometheusRemoteWriteDesiredShards.md | 33 +++++++++++++++++++ .../prometheus/PrometheusRuleFailures.md | 26 +++++++++++++-- .../PrometheusTSDBCompactionsFailing.md | 28 ++++++++++++++++ .../PrometheusTSDBReloadsFailing.md | 24 ++++++++++++++ .../prometheus/PrometheusTargetLimitHit.md | 22 +++++++++++++ .../prometheus/PrometheusTargetSyncFailure.md | 7 +++- 18 files changed, 393 insertions(+), 13 deletions(-) create mode 100644 content/runbooks/prometheus/PrometheusErrorSendingAlertsToAnyAlertmanager.md create mode 100644 content/runbooks/prometheus/PrometheusErrorSendingAlertsToSomeAlertmanagers.md create mode 100644 content/runbooks/prometheus/PrometheusLabelLimitHit.md create mode 100644 content/runbooks/prometheus/PrometheusMissingRuleEvaluations.md create mode 100644 content/runbooks/prometheus/PrometheusNotConnectedToAlertmanagers.md create mode 100644 content/runbooks/prometheus/PrometheusNotIngestingSamples.md create mode 100644 content/runbooks/prometheus/PrometheusNotificationQueueRunningFull.md create mode 100644 content/runbooks/prometheus/PrometheusRemoteStorageFailures.md create mode 100644 content/runbooks/prometheus/PrometheusRemoteWriteBehind.md create mode 100644 content/runbooks/prometheus/PrometheusRemoteWriteDesiredShards.md create mode 100644 content/runbooks/prometheus/PrometheusTSDBCompactionsFailing.md create mode 100644 content/runbooks/prometheus/PrometheusTSDBReloadsFailing.md create mode 100644 content/runbooks/prometheus/PrometheusTargetLimitHit.md diff --git a/content/runbooks/prometheus/PrometheusBadConfig.md b/content/runbooks/prometheus/PrometheusBadConfig.md index a0c18c2..5cb4184 100644 --- a/content/runbooks/prometheus/PrometheusBadConfig.md +++ b/content/runbooks/prometheus/PrometheusBadConfig.md @@ -1,17 +1,30 @@ +--- +title: Prometheus Bad Config +weight: 20 +--- + # PrometheusBadConfig ## Meaning -Alert fires when Prometheus cannot successfully reload the configuration file due to the file having incorrect content. +Alert fires when Prometheus cannot successfully reload the configuration file +due to the file having incorrect content. ## Impact -Configuration cannot be reloaded and prometheus operates with last known good configuration. Configuration changes in any of Prometheus, Probe, PodMonitor, or ServiceMonitor objects may not be picked up by prometheus server. +Configuration cannot be reloaded and prometheus operates with last known good +configuration. +Configuration changes in any of Prometheus, Probe, PodMonitor, +or ServiceMonitor objects may not be picked up by prometheus server. ## Diagnosis -Check prometheus container logs for an explanation of which part of the configuration is problematic. Usually this can occur when ServiceMonitors or PodMonitors share the same job label. +Check prometheus container logs for an explanation of which part of the +configuration is problematic. + +Usually this can occur when ServiceMonitors or +PodMonitors share the same job label. ## Mitigation -Remove conflicting configuration option. \ No newline at end of file +Remove conflicting configuration option. diff --git a/content/runbooks/prometheus/PrometheusDuplicateTimestamps.md b/content/runbooks/prometheus/PrometheusDuplicateTimestamps.md index a24fe61..82b4b5b 100644 --- a/content/runbooks/prometheus/PrometheusDuplicateTimestamps.md +++ b/content/runbooks/prometheus/PrometheusDuplicateTimestamps.md @@ -1,8 +1,13 @@ +--- +title: Prometheus Duplicate Timestamps +weight: 20 +--- + # PrometheusDuplicateTimestamps Find the Prometheus Pod that concerns this. -```bash +```shell $ kubectl -n get pod prometheus-k8s-0 2/2 Running 1 122m prometheus-k8s-1 2/2 Running 1 122m @@ -10,14 +15,20 @@ prometheus-k8s-1 2/2 Running 1 122m Look at the logs of each of them, there should be a log line such as: -```bash +```shell $ kubectl -n logs prometheus-k8s-0 level=warn ts=2021-01-04T15:08:55.613Z caller=scrape.go:1372 component="scrape manager" scrape_pool=default/main-ingress-nginx-controller/0 target=http://10.0.7.3:10254/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=16 ``` Now there is a judgement call to make, this could be the result of: -* Faulty configuration, which could be resolved by removing the offending `ServiceMonitor` or `PodMonitor` object, which can be identified through the `scrape_pool` label in the log line, which is in the format of `//`. -* The target is reporting faulty data, sometimes this can be resolved by restarting the target, or it might need to be fixed in code of the offending application. +* Faulty configuration, which could be resolved by removing the offending + `ServiceMonitor` or `PodMonitor` object, which can be identified through + the `scrape_pool` label in the log line, which is in the format of + `//`. -Further reading: https://www.robustperception.io/debugging-out-of-order-samples \ No newline at end of file +* The target is reporting faulty data, sometimes this can be resolved by + restarting the target, or it might need to be fixed in code of the offending + application. + +Further reading [blog](https://www.robustperception.io/debugging-out-of-order-samples) diff --git a/content/runbooks/prometheus/PrometheusErrorSendingAlertsToAnyAlertmanager.md b/content/runbooks/prometheus/PrometheusErrorSendingAlertsToAnyAlertmanager.md new file mode 100644 index 0000000..7e30617 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusErrorSendingAlertsToAnyAlertmanager.md @@ -0,0 +1,24 @@ +--- +title: Prometheus Error Sending Alerts To Any Alertmanager +weight: 20 +--- + +# PrometheusErrorSendingAlertsToAnyAlertmanager + +## Meaning + +Prometheus has encountered errors sending alerts to a any Alertmanager. + +## Impact + +All alerts may be lost. + +## Diagnosis + +Check connectivity issues between Prometheus and AlertManager cluster. +Check NetworkPolicies, network saturation. +Check if AlertManager is not overloaded or has not enough resources. + +## Mitigation + +Set multiple AlertManager instances, spread them across nodes. diff --git a/content/runbooks/prometheus/PrometheusErrorSendingAlertsToSomeAlertmanagers.md b/content/runbooks/prometheus/PrometheusErrorSendingAlertsToSomeAlertmanagers.md new file mode 100644 index 0000000..2a1c005 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusErrorSendingAlertsToSomeAlertmanagers.md @@ -0,0 +1,24 @@ +--- +title: Prometheus Error Sending Alerts To Some Alertmanagers +weight: 20 +--- + +# PrometheusErrorSendingAlertsToSomeAlertmanagers + +## Meaning + +Prometheus has encountered more than 1% errors sending alerts to a specific Alertmanager. + +## Impact + +Some alerts may be lost. + +## Diagnosis + +Check connectivity issues between Prometheus and AlertManager. +Check NetworkPolicies, network saturation. +Check if AlertManager is not overloaded or has not enough resources. + +## Mitigation + +Set multiple AlertManager instances, spread them across nodes. diff --git a/content/runbooks/prometheus/PrometheusLabelLimitHit.md b/content/runbooks/prometheus/PrometheusLabelLimitHit.md new file mode 100644 index 0000000..76a1420 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusLabelLimitHit.md @@ -0,0 +1,22 @@ +--- +title: Prometheus Label LimitHit +weight: 20 +--- + +# PrometheusLabelLimitHit + +## Meaning + +Prometheus has dropped targets because some scrape configs have exceeded the labels limit. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + + +## Mitigation + +Start thinking about sharding prometheus. +Increase scrape times to perform it less frequently. diff --git a/content/runbooks/prometheus/PrometheusMissingRuleEvaluations.md b/content/runbooks/prometheus/PrometheusMissingRuleEvaluations.md new file mode 100644 index 0000000..41c2eca --- /dev/null +++ b/content/runbooks/prometheus/PrometheusMissingRuleEvaluations.md @@ -0,0 +1,22 @@ +--- +title: Prometheus Missing Rule Evaluations +weight: 20 +--- + +# PrometheusMissingRuleEvaluations + +## Meaning + +Prometheus is missing rule evaluations due to slow rule group evaluation. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + +Check which rules fail, try to calcuate them differently. + +## Mitigation + +Sometimes giving more CPU is the only way to fix it. diff --git a/content/runbooks/prometheus/PrometheusNotConnectedToAlertmanagers.md b/content/runbooks/prometheus/PrometheusNotConnectedToAlertmanagers.md new file mode 100644 index 0000000..bf148e5 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusNotConnectedToAlertmanagers.md @@ -0,0 +1,24 @@ +--- +title: Prometheus Not Connected To Alertmanagers +weight: 20 +--- + +# PrometheusNotConnectedToAlertmanagers + +## Meaning + +Prometheus is not connected to any Alertmanagers. + +## Impact + +Sending alerts is not possible. + +## Diagnosis + +Check connectivity issues between Prometheus and AlertManager. +Check NetworkPolicies, network saturation. +Check if AlertManager is not overloaded or has not enough resources. + +## Mitigation + +Set multiple AlertManager instances, spread them across nodes. diff --git a/content/runbooks/prometheus/PrometheusNotIngestingSamples.md b/content/runbooks/prometheus/PrometheusNotIngestingSamples.md new file mode 100644 index 0000000..9f791da --- /dev/null +++ b/content/runbooks/prometheus/PrometheusNotIngestingSamples.md @@ -0,0 +1,22 @@ +--- +title: Prometheus Not Ingesting Samples +weight: 20 +--- + +# PrometheusNotIngestingSamples + +## Meaning + +Prometheus is not ingesting samples. + +## Impact + +Missing metrics. + +## Diagnosis + +TODO + +## Mitigation + +TODO diff --git a/content/runbooks/prometheus/PrometheusNotificationQueueRunningFull.md b/content/runbooks/prometheus/PrometheusNotificationQueueRunningFull.md new file mode 100644 index 0000000..444e717 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusNotificationQueueRunningFull.md @@ -0,0 +1,26 @@ +--- +title: Prometheus Notification Queue Running Full +weight: 20 +--- + +# PrometheusNotificationQueueRunningFull + +## Meaning + +Prometheus alert notification queue predicted to run full in less than 30m. + +## Impact + +Fail to send alerts. + +## Diagnosis + +Check prometheus container logs for an explanation of which part of the +configuration is problematic. + +## Mitigation + +Remove conflicting configuration option. + +Check if there is an option to decrease number of alerts firing, +for example by sharding prometheus. diff --git a/content/runbooks/prometheus/PrometheusOutOfOrderTimestamps.md b/content/runbooks/prometheus/PrometheusOutOfOrderTimestamps.md index 852f335..b7910a1 100644 --- a/content/runbooks/prometheus/PrometheusOutOfOrderTimestamps.md +++ b/content/runbooks/prometheus/PrometheusOutOfOrderTimestamps.md @@ -1,3 +1,8 @@ +--- +title: Prometheus Out Of Order Timestamps +weight: 20 +--- + # PrometheusOutOfOrderTimestamps -More information in https://www.robustperception.io/debugging-out-of-order-samples +More information in [blog](https://www.robustperception.io/debugging-out-of-order-samples) diff --git a/content/runbooks/prometheus/PrometheusRemoteStorageFailures.md b/content/runbooks/prometheus/PrometheusRemoteStorageFailures.md new file mode 100644 index 0000000..c1502da --- /dev/null +++ b/content/runbooks/prometheus/PrometheusRemoteStorageFailures.md @@ -0,0 +1,24 @@ +--- +title: Prometheus Remote Storage Failures +weight: 20 +--- + +# PrometheusRemoteStorageFailures + +## Meaning + +Prometheus fails to send samples to remote storage. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + +Check prometheus logs and remote storage logs. +Investigate network issues. +Check configs and credentials. + +## Mitigation + +TODO diff --git a/content/runbooks/prometheus/PrometheusRemoteWriteBehind.md b/content/runbooks/prometheus/PrometheusRemoteWriteBehind.md new file mode 100644 index 0000000..9e9896c --- /dev/null +++ b/content/runbooks/prometheus/PrometheusRemoteWriteBehind.md @@ -0,0 +1,29 @@ +--- +title: Prometheus Remote Write Behind +weight: 20 +--- + +# PrometheusRemoteStorageFailures + +## Meaning + +Prometheus remote write is behind. + +## Impact + +Metrics and alerts may be missing or inaccurate. +Increased data lag between locations. + +## Diagnosis + +Check prometheus logs and remote storage logs. +Investigate network issues. +Check configs and credentials. + +## Mitigation + +Probbaly amout of data sent to remote system is too high +for given network connectivity speed. +You may need to limit which metrics to send to minimize transfers. + +See [Prometheus Remote Storage Failures]({{< ref "./PrometheusRemoteStorageFailures.md" >}}) diff --git a/content/runbooks/prometheus/PrometheusRemoteWriteDesiredShards.md b/content/runbooks/prometheus/PrometheusRemoteWriteDesiredShards.md new file mode 100644 index 0000000..24448bd --- /dev/null +++ b/content/runbooks/prometheus/PrometheusRemoteWriteDesiredShards.md @@ -0,0 +1,33 @@ +--- +title: PrometheusRemoteWriteDesiredShards +weight: 20 +--- + +# PrometheusRemoteWriteDesiredShards + +## Meaning + +Prometheus remote write desired shards calculation wants to run +more than configured max shards. + + +## Impact + +Metrics and alerts may be missing or inaccurate. + + +## Diagnosis + +Check metrics cardinality. + +Check prometheus logs and remote storage logs. +Investigate network issues. +Check configs and credentials. + +## Mitigation + +Probbaly amout of data sent to remote system is too high +for given network connectivity speed. +You may need to limit which metrics to send to minimize transfers. + +See [Prometheus Remote Storage Failures]({{< ref "./PrometheusRemoteStorageFailures.md" >}}) diff --git a/content/runbooks/prometheus/PrometheusRuleFailures.md b/content/runbooks/prometheus/PrometheusRuleFailures.md index e1ded31..c354e82 100644 --- a/content/runbooks/prometheus/PrometheusRuleFailures.md +++ b/content/runbooks/prometheus/PrometheusRuleFailures.md @@ -1,5 +1,27 @@ +--- +title: Prometheus Rule Failures +weight: 20 +--- + # PrometheusRuleFailures -Your best starting point is the rules page of the Prometheus UI (:9090/rules). It will show the error. +## Meaning -You can also evaluate the rule expression yourself, using the UI, or maybe using PromLens to help debug expression issues. +Prometheus is failing rule evaluations. +Prometheus rules are incorrect or failed to calculate. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + +Your best starting point is the rules page of the Prometheus UI (:9090/rules). +It will show the error. + +You can also evaluate the rule expression yourself, using the UI, or maybe +using PromLens to help debug expression issues. + +## Mitigation + +Fix rules. diff --git a/content/runbooks/prometheus/PrometheusTSDBCompactionsFailing.md b/content/runbooks/prometheus/PrometheusTSDBCompactionsFailing.md new file mode 100644 index 0000000..5b9d299 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusTSDBCompactionsFailing.md @@ -0,0 +1,28 @@ +--- +title: Prometheus TSDB Compactions Failing +weight: 20 +--- + +# PrometheusTSDBCompactionsFailing + +## Meaning + +Prometheus has issues compacting blocks. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + +Check storage used by the pod. +This can happen if there is a lot of going on in the cluster and +prometheus did not manage to compact data. + +## Mitigation + +At first just wait, it may fix itself after some time. + +Increase Prometheus pod memory so that it caches more from disk. +Try expanding volumes if they are too small or too slow. +Change PVC storageClass to a more performant one. diff --git a/content/runbooks/prometheus/PrometheusTSDBReloadsFailing.md b/content/runbooks/prometheus/PrometheusTSDBReloadsFailing.md new file mode 100644 index 0000000..1c9c492 --- /dev/null +++ b/content/runbooks/prometheus/PrometheusTSDBReloadsFailing.md @@ -0,0 +1,24 @@ +--- +title: Prometheus TSDB Reloads Failing +weight: 20 +--- + +# PrometheusTSDBReloadsFailing + +## Meaning + +Prometheus has issues reloading blocks from disk. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + +Check storage used by the pod. + +## Mitigation + +Increase Prometheus pod memory so that it caches more from disk. +Try expanding volumes if they are too small or too slow. +Change PVC storageClass to a more performant one. diff --git a/content/runbooks/prometheus/PrometheusTargetLimitHit.md b/content/runbooks/prometheus/PrometheusTargetLimitHit.md new file mode 100644 index 0000000..320baae --- /dev/null +++ b/content/runbooks/prometheus/PrometheusTargetLimitHit.md @@ -0,0 +1,22 @@ +--- +title: Prometheus Target Limit Hit +weight: 20 +--- + +# PrometheusTargetLimitHit + +## Meaning + +Prometheus has dropped targets because some scrape configs have exceeded the targets limit. + +## Impact + +Metrics and alerts may be missing or inaccurate. + +## Diagnosis + + +## Mitigation + +Start thinking about sharding prometheus. +Increase scrape times to perform it less frequently. diff --git a/content/runbooks/prometheus/PrometheusTargetSyncFailure.md b/content/runbooks/prometheus/PrometheusTargetSyncFailure.md index e38062d..a1c37ed 100644 --- a/content/runbooks/prometheus/PrometheusTargetSyncFailure.md +++ b/content/runbooks/prometheus/PrometheusTargetSyncFailure.md @@ -1,3 +1,8 @@ +--- +title: Prometheus Target Sync Failure +weight: 20 +--- + # PrometheusTargetSyncFailure ## Meaning @@ -16,7 +21,7 @@ inspecting the alert's `namespace` label. Check the logs for the appropriate Prometheus instance: -```console +```shell $ NAMESPACE='' $ oc -n $NAMESPACE logs -l 'app=prometheus'