Two new value maps under `prometheusRules`:
- `alertExprOverrides` — alertName -> custom Prometheus expression
that replaces the default `expr:`. Closes#253: multi-cluster setups
can inject `max by (cluster, …)` aggregations or label filters
without having to disableBuiltinAlertGroup and reconstruct everything.
- `alertForOverrides` — alertName -> `for:` duration that replaces
the default. Schema-validated against Prometheus's duration syntax.
Both maps key on bare alert names (no `rulePrefix`). Overrides are
taken whole — no merge with the default. Single-line and multi-line
strings both render correctly: the template now emits each `expr:`
via `| quote` which wraps in a YAML double-quoted scalar (newlines
serialise to \n; Prometheus parses the resulting string identically
to the multi-line form).
Test ratchets added alongside:
- test/schema/valid/prometheusrules-alert-overrides.yaml — valid case
covering both maps, with a multi-line expr to exercise the quoting
path.
- test/schema/invalid/prometheusrules-alertforoverrides-bad-duration
— paired with .expect.txt, ensures the duration regex rejects
natural-language inputs ("30 minutes").
- test/render/prometheusrules-default-alerts.{yaml,expect-pass.txt} —
new ratchet under the existing TestHelmRender machinery (extended
to support "positive substring" checks via .expect-pass.txt). Locks
in the exact 13 alert names the chart ships by default: any rename
or accidental removal breaks the test at PR time.
Docs:
- chart/README.md.gotmpl alert table refreshed — was still on the v3
layout (X509ExporterReadErrors + 3 cert alerts). Now lists all 13.
- values.yaml gets the two new map keys with descriptions and
commented examples.
MAJOR CHANGE: PrometheusRule alert names and chart value keys
renamed. Users with AlertManager routing or silencing rules that
reference the old names must update them on upgrade. Migration map:
X509ExporterReadErrors -> SourceErrors (+ SourceErrorsSustained)
alertOnReadErrors -> alertOnSourceErrors
readErrorsSeverity -> sourceErrorsSeverity (+ ...SustainedSeverity)
alertOnPassphraseFailures -> alertOnKeystorePassphraseFailures
passphraseFailuresSeverity -> keystorePassphraseFailuresSeverity
alertOnCertificateErrors -> alertOnCertificateError (singular)
certificateErrorsSeverity -> certificateErrorSeverity (singular)
alertOnCertificateCollisions -> alertOnCertificateCollision (singular)
certificateCollisionsSeverity -> certificateCollisionSeverity
alertOnCRLs -> (removed; CRL alerts always-on like Renewal/Expiration)
Substantive changes alongside the renames:
- SourceErrors gets the same two-band shape as KubeTransportErrors
(warning at >5/15min for 5m, critical for 30m). The previous "fire
on any single error" expression paged on a single malformed PEM,
which was indistinguishable from a real outage.
- SourceErrors and KubeTransportErrors both aggregate over
reason/resource via 'sum without (...)', so the alert routes by
source. The breakdown stays available on the metric for triage.
- CRLNeedsRefresh and CRLStale no longer require alertOnCRLs — same
policy as CertificateRenewal/Expiration. The underlying x509_crl_*
series only exist when a CRL is actually observed, so installs that
don't watch CRLs get nothing.
- CertificateRenewal expression now anchors to (not_after - now) > 0,
so it doesn't keep firing alongside CertificateExpiration once a
cert is past its NotAfter (those two alerts had identical truth
values in the late-life window).
- CertificateExpiration description distinguishes 'expires in X' from
'expired X ago' via humanizeDuration's sign, so the message stays
accurate after the cert is past NotAfter.
- CertificateError, CertificateNotYetValid, CertificateRenewal and
CertificateExpiration each get inline comments explaining their
dependency or always-on status — consistent with the recent
KubeTransportErrors / CertificateCollision additions.
Template hygiene:
- Three helper templates in _helpers.tpl: alertExtraLabels and
alertExtraAnnotations (replace 12 duplicated if-blocks);
alertLocationSuffix (replaces the 5 sites that re-emit the
"{{if $labels.secret_name}}in Kubernetes secret …{{else}}at
location …{{end}}" literal — when we flipped quotes to backticks
recently, 5 edits were required; the helper would have made that
one).
- All alerts stay in a single PrometheusRule group. Groups in
PrometheusRule only matter for sequential recording-rule ordering,
per-group evaluation intervals, or execution isolation — none of
which apply here. AlertManager routes on labels (alertname,
severity), not on group names, so splitting by domain would
fragment without buying anything; worse, it would trap users who
add recording rules via extraAlertGroups into the wrong group.
Category-based routing on the consumer side is straightforward via
alertname regex (Source.*, Cert.*, CRL.*, Kube.*).
The Kubernetes source's transport layer had 9 distinct failure sites
that surfaced only as logs: LIST failure (with retry/backoff), WATCH
start failure, WATCH stream Error event, watch flap (close within 5s
of open), and namespace informer sync timeout — once for secrets, once
for configmaps. None had a corresponding metric, so an exporter that
was functionally up but operationally degraded (token rotation drift,
apiserver throttling, intermittent network) was invisible to ops.
Introduce x509_kube_transport_errors_total{source_name,resource,reason}
incremented at each of the 9 sites via a new Recorder interface on
k8s.Options. The interface keeps the source decoupled from
*registry.Registry; nil is valid (the unmetered path keeps the old
log-only behaviour) and tests plug a fake. main.go wires the actual
registry through.
Ship a paired KubeTransportErrors PrometheusRule alert (opt-in via
alertOnKubeTransportErrors, default true). docs/metrics.md documents
the new metric, the five reason codes, and cross-links the alert.
Four new opt-in alerts close observability holes the existing rules
left uncovered:
- SourceDown (critical) — x509_source_up == 0 for 5m. Catches RBAC
failures, persistent K8s API errors, unreadable file paths. A source
that never converges would otherwise be silent while certs it should
watch are not checked.
- KeystorePassphraseFailures (warning) — increase on
x509_{pkcs12,jks}_passphrase_failures_total. A misconfigured
passphraseKey / passphraseSecretRef previously only leaked into
logs and an unwatched counter.
- CertificateNotYetValid (warning) — x509_cert_not_before > time().
Depends on exposeNotBeforeMetric: true; gated for explicit opt-in.
- CertificateCollision (warning) — increase on x509_cert_collision_total.
Two certs sharing a label set means one is silently invisible.
Each alert ships with an alertOnXxx toggle (default true) and its own
severity in values.yaml. Schema + README regenerated. helm lint, helm
template, schema fixtures and helm-examples all pass.
