MAJOR CHANGE: PrometheusRule alert names and chart value keys
renamed. Users with AlertManager routing or silencing rules that
reference the old names must update them on upgrade. Migration map:
X509ExporterReadErrors -> SourceErrors (+ SourceErrorsSustained)
alertOnReadErrors -> alertOnSourceErrors
readErrorsSeverity -> sourceErrorsSeverity (+ ...SustainedSeverity)
alertOnPassphraseFailures -> alertOnKeystorePassphraseFailures
passphraseFailuresSeverity -> keystorePassphraseFailuresSeverity
alertOnCertificateErrors -> alertOnCertificateError (singular)
certificateErrorsSeverity -> certificateErrorSeverity (singular)
alertOnCertificateCollisions -> alertOnCertificateCollision (singular)
certificateCollisionsSeverity -> certificateCollisionSeverity
alertOnCRLs -> (removed; CRL alerts always-on like Renewal/Expiration)
Substantive changes alongside the renames:
- SourceErrors gets the same two-band shape as KubeTransportErrors
(warning at >5/15min for 5m, critical for 30m). The previous "fire
on any single error" expression paged on a single malformed PEM,
which was indistinguishable from a real outage.
- SourceErrors and KubeTransportErrors both aggregate over
reason/resource via 'sum without (...)', so the alert routes by
source. The breakdown stays available on the metric for triage.
- CRLNeedsRefresh and CRLStale no longer require alertOnCRLs — same
policy as CertificateRenewal/Expiration. The underlying x509_crl_*
series only exist when a CRL is actually observed, so installs that
don't watch CRLs get nothing.
- CertificateRenewal expression now anchors to (not_after - now) > 0,
so it doesn't keep firing alongside CertificateExpiration once a
cert is past its NotAfter (those two alerts had identical truth
values in the late-life window).
- CertificateExpiration description distinguishes 'expires in X' from
'expired X ago' via humanizeDuration's sign, so the message stays
accurate after the cert is past NotAfter.
- CertificateError, CertificateNotYetValid, CertificateRenewal and
CertificateExpiration each get inline comments explaining their
dependency or always-on status — consistent with the recent
KubeTransportErrors / CertificateCollision additions.
Template hygiene:
- Three helper templates in _helpers.tpl: alertExtraLabels and
alertExtraAnnotations (replace 12 duplicated if-blocks);
alertLocationSuffix (replaces the 5 sites that re-emit the
"{{if $labels.secret_name}}in Kubernetes secret …{{else}}at
location …{{end}}" literal — when we flipped quotes to backticks
recently, 5 edits were required; the helper would have made that
one).
- All alerts stay in a single PrometheusRule group. Groups in
PrometheusRule only matter for sequential recording-rule ordering,
per-group evaluation intervals, or execution isolation — none of
which apply here. AlertManager routes on labels (alertname,
severity), not on group names, so splitting by domain would
fragment without buying anything; worse, it would trap users who
add recording rules via extraAlertGroups into the wrong group.
Category-based routing on the consumer side is straightforward via
alertname regex (Source.*, Cert.*, CRL.*, Kube.*).
Thibault VINCENTandThibault VINCENT