Two new value maps under `prometheusRules`:
- `alertExprOverrides` — alertName -> custom Prometheus expression
that replaces the default `expr:`. Closes#253: multi-cluster setups
can inject `max by (cluster, …)` aggregations or label filters
without having to disableBuiltinAlertGroup and reconstruct everything.
- `alertForOverrides` — alertName -> `for:` duration that replaces
the default. Schema-validated against Prometheus's duration syntax.
Both maps key on bare alert names (no `rulePrefix`). Overrides are
taken whole — no merge with the default. Single-line and multi-line
strings both render correctly: the template now emits each `expr:`
via `| quote` which wraps in a YAML double-quoted scalar (newlines
serialise to \n; Prometheus parses the resulting string identically
to the multi-line form).
Test ratchets added alongside:
- test/schema/valid/prometheusrules-alert-overrides.yaml — valid case
covering both maps, with a multi-line expr to exercise the quoting
path.
- test/schema/invalid/prometheusrules-alertforoverrides-bad-duration
— paired with .expect.txt, ensures the duration regex rejects
natural-language inputs ("30 minutes").
- test/render/prometheusrules-default-alerts.{yaml,expect-pass.txt} —
new ratchet under the existing TestHelmRender machinery (extended
to support "positive substring" checks via .expect-pass.txt). Locks
in the exact 13 alert names the chart ships by default: any rename
or accidental removal breaks the test at PR time.
Docs:
- chart/README.md.gotmpl alert table refreshed — was still on the v3
layout (X509ExporterReadErrors + 3 cert alerts). Now lists all 13.
- values.yaml gets the two new map keys with descriptions and
commented examples.
MAJOR CHANGE: PrometheusRule alert names and chart value keys
renamed. Users with AlertManager routing or silencing rules that
reference the old names must update them on upgrade. Migration map:
X509ExporterReadErrors -> SourceErrors (+ SourceErrorsSustained)
alertOnReadErrors -> alertOnSourceErrors
readErrorsSeverity -> sourceErrorsSeverity (+ ...SustainedSeverity)
alertOnPassphraseFailures -> alertOnKeystorePassphraseFailures
passphraseFailuresSeverity -> keystorePassphraseFailuresSeverity
alertOnCertificateErrors -> alertOnCertificateError (singular)
certificateErrorsSeverity -> certificateErrorSeverity (singular)
alertOnCertificateCollisions -> alertOnCertificateCollision (singular)
certificateCollisionsSeverity -> certificateCollisionSeverity
alertOnCRLs -> (removed; CRL alerts always-on like Renewal/Expiration)
Substantive changes alongside the renames:
- SourceErrors gets the same two-band shape as KubeTransportErrors
(warning at >5/15min for 5m, critical for 30m). The previous "fire
on any single error" expression paged on a single malformed PEM,
which was indistinguishable from a real outage.
- SourceErrors and KubeTransportErrors both aggregate over
reason/resource via 'sum without (...)', so the alert routes by
source. The breakdown stays available on the metric for triage.
- CRLNeedsRefresh and CRLStale no longer require alertOnCRLs — same
policy as CertificateRenewal/Expiration. The underlying x509_crl_*
series only exist when a CRL is actually observed, so installs that
don't watch CRLs get nothing.
- CertificateRenewal expression now anchors to (not_after - now) > 0,
so it doesn't keep firing alongside CertificateExpiration once a
cert is past its NotAfter (those two alerts had identical truth
values in the late-life window).
- CertificateExpiration description distinguishes 'expires in X' from
'expired X ago' via humanizeDuration's sign, so the message stays
accurate after the cert is past NotAfter.
- CertificateError, CertificateNotYetValid, CertificateRenewal and
CertificateExpiration each get inline comments explaining their
dependency or always-on status — consistent with the recent
KubeTransportErrors / CertificateCollision additions.
Template hygiene:
- Three helper templates in _helpers.tpl: alertExtraLabels and
alertExtraAnnotations (replace 12 duplicated if-blocks);
alertLocationSuffix (replaces the 5 sites that re-emit the
"{{if $labels.secret_name}}in Kubernetes secret …{{else}}at
location …{{end}}" literal — when we flipped quotes to backticks
recently, 5 edits were required; the helper would have made that
one).
- All alerts stay in a single PrometheusRule group. Groups in
PrometheusRule only matter for sequential recording-rule ordering,
per-group evaluation intervals, or execution isolation — none of
which apply here. AlertManager routes on labels (alertname,
severity), not on group names, so splitting by domain would
fragment without buying anything; worse, it would trap users who
add recording rules via extraAlertGroups into the wrong group.
Category-based routing on the consumer side is straightforward via
alertname regex (Source.*, Cert.*, CRL.*, Kube.*).
The Kubernetes source's transport layer had 9 distinct failure sites
that surfaced only as logs: LIST failure (with retry/backoff), WATCH
start failure, WATCH stream Error event, watch flap (close within 5s
of open), and namespace informer sync timeout — once for secrets, once
for configmaps. None had a corresponding metric, so an exporter that
was functionally up but operationally degraded (token rotation drift,
apiserver throttling, intermittent network) was invisible to ops.
Introduce x509_kube_transport_errors_total{source_name,resource,reason}
incremented at each of the 9 sites via a new Recorder interface on
k8s.Options. The interface keeps the source decoupled from
*registry.Registry; nil is valid (the unmetered path keeps the old
log-only behaviour) and tests plug a fake. main.go wires the actual
registry through.
Ship a paired KubeTransportErrors PrometheusRule alert (opt-in via
alertOnKubeTransportErrors, default true). docs/metrics.md documents
the new metric, the five reason codes, and cross-links the alert.
Four new opt-in alerts close observability holes the existing rules
left uncovered:
- SourceDown (critical) — x509_source_up == 0 for 5m. Catches RBAC
failures, persistent K8s API errors, unreadable file paths. A source
that never converges would otherwise be silent while certs it should
watch are not checked.
- KeystorePassphraseFailures (warning) — increase on
x509_{pkcs12,jks}_passphrase_failures_total. A misconfigured
passphraseKey / passphraseSecretRef previously only leaked into
logs and an unwatched counter.
- CertificateNotYetValid (warning) — x509_cert_not_before > time().
Depends on exposeNotBeforeMetric: true; gated for explicit opt-in.
- CertificateCollision (warning) — increase on x509_cert_collision_total.
Two certs sharing a label set means one is silently invisible.
Each alert ships with an alertOnXxx toggle (default true) and its own
severity in values.yaml. Schema + README regenerated. helm lint, helm
template, schema fixtures and helm-examples all pass.
The X509ExporterReadErrors PrometheusRule alert matched x509_read_errors,
the v3 metric name. The v4 exporter emits x509_source_errors_total
instead — the alert never fired since the v4 rewrite. Same stale name
lingered in the values.yaml extraAlertGroups example and across 8
Grafana dashboard panels. All three sites renamed in lockstep.
Switch the alert's range function from delta() to increase() while
we're here. delta() is documented for gauges; for monotonic counters
increase() is the canonical choice and correctly handles counter
resets across exporter restarts.
Complete rewrite of the codebase, the build pipeline, the dev loop,
and the release pipeline.
For the exporter itself, refer to the updated README and Helm chart
documentation to discover the new functionality and assess the impact
of the breaking changes on your existing setup.
Build & release:
- QA/CI pipelines now run through a Dagger Module, wrapped by
Taskfile.yml for the developer interface.
- Releases run through GoReleaser: cross-compiled binaries × OS/arch,
archives, checksums, multi-arch container images (busybox + scratch
variants on linux/amd64,arm64,riscv64), pushed to ghcr/quay/docker.io.
- Everything is cosign-signed (binaries, images, Helm chart). Image
CycloneDX SBOMs are attached as cosign attestations. SLSA-3
provenance is attached to every GitHub Release.
- The Helm chart is published as a cosign-signed OCI artifact.
- Versioning and changelog are automated by release-please from
Conventional Commits.
Dev experience:
- Local loop driven by Tilt + k3d + Dagger; one command brings up an
exporter with seeded fixtures and a Prometheus scraping it.
- End-to-end tests run on a throwaway k3d cluster against the real
rendered chart.
BREAKING CHANGE: the Helm chart is now published exclusively as an OCI
artifact at oci://quay.io/enix/charts/x509-certificate-exporter. The
legacy Helm repository at https://charts.enix.io is no longer updated;
users must switch to the OCI reference (Helm 3.8+ required).
Installation: `helm install x509-certificate-exporter
oci://quay.io/enix/charts/x509-certificate-exporter --version <vX.Y.Z>`.
BREAKING CHANGE: the Helm chart's values schema may diverge from v3 in
edge cases despite a best-effort to preserve backwards compatibility.
Review your existing values against the updated chart/values.yaml
before upgrading. A JSON schema (chart/values.schema.json) is shipped
with the chart so `helm install` / `helm upgrade` will reject any
values that no longer match the expected shape, surfacing regressions
early instead of at runtime.
BREAKING CHANGE: Alpine-based container images are no longer published.
The release pipeline now ships only the `busybox` and `scratch` variants
on linux/amd64,arm64,riscv64. Users pulling `*-alpine` tags must switch
to one of the new variants — `busybox` is the closest functional
replacement (still has a shell), `scratch` is the minimal distroless
option.
Thibault VINCENT
enix-renovate[bot]