Two new value maps under `prometheusRules`:
- `alertExprOverrides` — alertName -> custom Prometheus expression
that replaces the default `expr:`. Closes#253: multi-cluster setups
can inject `max by (cluster, …)` aggregations or label filters
without having to disableBuiltinAlertGroup and reconstruct everything.
- `alertForOverrides` — alertName -> `for:` duration that replaces
the default. Schema-validated against Prometheus's duration syntax.
Both maps key on bare alert names (no `rulePrefix`). Overrides are
taken whole — no merge with the default. Single-line and multi-line
strings both render correctly: the template now emits each `expr:`
via `| quote` which wraps in a YAML double-quoted scalar (newlines
serialise to \n; Prometheus parses the resulting string identically
to the multi-line form).
Test ratchets added alongside:
- test/schema/valid/prometheusrules-alert-overrides.yaml — valid case
covering both maps, with a multi-line expr to exercise the quoting
path.
- test/schema/invalid/prometheusrules-alertforoverrides-bad-duration
— paired with .expect.txt, ensures the duration regex rejects
natural-language inputs ("30 minutes").
- test/render/prometheusrules-default-alerts.{yaml,expect-pass.txt} —
new ratchet under the existing TestHelmRender machinery (extended
to support "positive substring" checks via .expect-pass.txt). Locks
in the exact 13 alert names the chart ships by default: any rename
or accidental removal breaks the test at PR time.
Docs:
- chart/README.md.gotmpl alert table refreshed — was still on the v3
layout (X509ExporterReadErrors + 3 cert alerts). Now lists all 13.
- values.yaml gets the two new map keys with descriptions and
commented examples.
MAJOR CHANGE: PrometheusRule alert names and chart value keys
renamed. Users with AlertManager routing or silencing rules that
reference the old names must update them on upgrade. Migration map:
X509ExporterReadErrors -> SourceErrors (+ SourceErrorsSustained)
alertOnReadErrors -> alertOnSourceErrors
readErrorsSeverity -> sourceErrorsSeverity (+ ...SustainedSeverity)
alertOnPassphraseFailures -> alertOnKeystorePassphraseFailures
passphraseFailuresSeverity -> keystorePassphraseFailuresSeverity
alertOnCertificateErrors -> alertOnCertificateError (singular)
certificateErrorsSeverity -> certificateErrorSeverity (singular)
alertOnCertificateCollisions -> alertOnCertificateCollision (singular)
certificateCollisionsSeverity -> certificateCollisionSeverity
alertOnCRLs -> (removed; CRL alerts always-on like Renewal/Expiration)
Substantive changes alongside the renames:
- SourceErrors gets the same two-band shape as KubeTransportErrors
(warning at >5/15min for 5m, critical for 30m). The previous "fire
on any single error" expression paged on a single malformed PEM,
which was indistinguishable from a real outage.
- SourceErrors and KubeTransportErrors both aggregate over
reason/resource via 'sum without (...)', so the alert routes by
source. The breakdown stays available on the metric for triage.
- CRLNeedsRefresh and CRLStale no longer require alertOnCRLs — same
policy as CertificateRenewal/Expiration. The underlying x509_crl_*
series only exist when a CRL is actually observed, so installs that
don't watch CRLs get nothing.
- CertificateRenewal expression now anchors to (not_after - now) > 0,
so it doesn't keep firing alongside CertificateExpiration once a
cert is past its NotAfter (those two alerts had identical truth
values in the late-life window).
- CertificateExpiration description distinguishes 'expires in X' from
'expired X ago' via humanizeDuration's sign, so the message stays
accurate after the cert is past NotAfter.
- CertificateError, CertificateNotYetValid, CertificateRenewal and
CertificateExpiration each get inline comments explaining their
dependency or always-on status — consistent with the recent
KubeTransportErrors / CertificateCollision additions.
Template hygiene:
- Three helper templates in _helpers.tpl: alertExtraLabels and
alertExtraAnnotations (replace 12 duplicated if-blocks);
alertLocationSuffix (replaces the 5 sites that re-emit the
"{{if $labels.secret_name}}in Kubernetes secret …{{else}}at
location …{{end}}" literal — when we flipped quotes to backticks
recently, 5 edits were required; the helper would have made that
one).
- All alerts stay in a single PrometheusRule group. Groups in
PrometheusRule only matter for sequential recording-rule ordering,
per-group evaluation intervals, or execution isolation — none of
which apply here. AlertManager routes on labels (alertname,
severity), not on group names, so splitting by domain would
fragment without buying anything; worse, it would trap users who
add recording rules via extraAlertGroups into the wrong group.
Category-based routing on the consumer side is straightforward via
alertname regex (Source.*, Cert.*, CRL.*, Kube.*).
Thibault VINCENT