Five user-facing counters had a UX hit: prometheus.NewCounterVec only
materialises a series the first time WithLabelValues(...).Inc() is
called, so an exporter running cleanly produced no series at all for
x509_source_errors_total, x509_kube_transport_errors_total,
x509_cert_collision_dropped_total and the two
x509_{pkcs12,jks}_passphrase_failures_total. Dashboards couldn't tell
'healthy' from 'metric not reporting', rate()/increase() needed two
real events to compute anything, and the chart's
SourceErrors[Sustained] / KubeTransportErrors[Sustained] alerts had
ambiguous PromQL semantics on first event.
PreInitBundleSource(kind, name) and PreInitKubeTransport(name,
resources, namespaceInformer) on Registry materialise the expected
series at zero by calling WithLabelValues without .Inc(). The static
reason sets they iterate (cert.BundleReasons, cert.KubeTransportPerResourceReasons,
cert.ReasonNamespaceSyncFail) live in pkg/cert/reason.go for a single
source of truth — dynamic 'http_NNN' reasons are deliberately
excluded since they can't be enumerated.
cmd/x509-certificate-exporter/main.go calls the right combination
for each config.Source kind right after buildSource: kubernetes
sources get both bundle init (kube-secret, kube-configmap) and
transport init (per-resource + namespace informer when label rules
require it); file/kubeconfig/cabundle sources just get the bundle
init for their respective kind. Cardinality cost is bounded by
declared reasons × source count — about 50 series per source — and
negligible against the per-cert series the exporter produces in
normal operation.
TestPreInitMaterializesZeroSeries in pkg/registry locks in the exact
series count materialised by a representative call and asserts every
emitted counter value is 0.
Two new value maps under `prometheusRules`:
- `alertExprOverrides` — alertName -> custom Prometheus expression
that replaces the default `expr:`. Closes#253: multi-cluster setups
can inject `max by (cluster, …)` aggregations or label filters
without having to disableBuiltinAlertGroup and reconstruct everything.
- `alertForOverrides` — alertName -> `for:` duration that replaces
the default. Schema-validated against Prometheus's duration syntax.
Both maps key on bare alert names (no `rulePrefix`). Overrides are
taken whole — no merge with the default. Single-line and multi-line
strings both render correctly: the template now emits each `expr:`
via `| quote` which wraps in a YAML double-quoted scalar (newlines
serialise to \n; Prometheus parses the resulting string identically
to the multi-line form).
Test ratchets added alongside:
- test/schema/valid/prometheusrules-alert-overrides.yaml — valid case
covering both maps, with a multi-line expr to exercise the quoting
path.
- test/schema/invalid/prometheusrules-alertforoverrides-bad-duration
— paired with .expect.txt, ensures the duration regex rejects
natural-language inputs ("30 minutes").
- test/render/prometheusrules-default-alerts.{yaml,expect-pass.txt} —
new ratchet under the existing TestHelmRender machinery (extended
to support "positive substring" checks via .expect-pass.txt). Locks
in the exact 13 alert names the chart ships by default: any rename
or accidental removal breaks the test at PR time.
Docs:
- chart/README.md.gotmpl alert table refreshed — was still on the v3
layout (X509ExporterReadErrors + 3 cert alerts). Now lists all 13.
- values.yaml gets the two new map keys with descriptions and
commented examples.
MAJOR CHANGE: PrometheusRule alert names and chart value keys
renamed. Users with AlertManager routing or silencing rules that
reference the old names must update them on upgrade. Migration map:
X509ExporterReadErrors -> SourceErrors (+ SourceErrorsSustained)
alertOnReadErrors -> alertOnSourceErrors
readErrorsSeverity -> sourceErrorsSeverity (+ ...SustainedSeverity)
alertOnPassphraseFailures -> alertOnKeystorePassphraseFailures
passphraseFailuresSeverity -> keystorePassphraseFailuresSeverity
alertOnCertificateErrors -> alertOnCertificateError (singular)
certificateErrorsSeverity -> certificateErrorSeverity (singular)
alertOnCertificateCollisions -> alertOnCertificateCollision (singular)
certificateCollisionsSeverity -> certificateCollisionSeverity
alertOnCRLs -> (removed; CRL alerts always-on like Renewal/Expiration)
Substantive changes alongside the renames:
- SourceErrors gets the same two-band shape as KubeTransportErrors
(warning at >5/15min for 5m, critical for 30m). The previous "fire
on any single error" expression paged on a single malformed PEM,
which was indistinguishable from a real outage.
- SourceErrors and KubeTransportErrors both aggregate over
reason/resource via 'sum without (...)', so the alert routes by
source. The breakdown stays available on the metric for triage.
- CRLNeedsRefresh and CRLStale no longer require alertOnCRLs — same
policy as CertificateRenewal/Expiration. The underlying x509_crl_*
series only exist when a CRL is actually observed, so installs that
don't watch CRLs get nothing.
- CertificateRenewal expression now anchors to (not_after - now) > 0,
so it doesn't keep firing alongside CertificateExpiration once a
cert is past its NotAfter (those two alerts had identical truth
values in the late-life window).
- CertificateExpiration description distinguishes 'expires in X' from
'expired X ago' via humanizeDuration's sign, so the message stays
accurate after the cert is past NotAfter.
- CertificateError, CertificateNotYetValid, CertificateRenewal and
CertificateExpiration each get inline comments explaining their
dependency or always-on status — consistent with the recent
KubeTransportErrors / CertificateCollision additions.
Template hygiene:
- Three helper templates in _helpers.tpl: alertExtraLabels and
alertExtraAnnotations (replace 12 duplicated if-blocks);
alertLocationSuffix (replaces the 5 sites that re-emit the
"{{if $labels.secret_name}}in Kubernetes secret …{{else}}at
location …{{end}}" literal — when we flipped quotes to backticks
recently, 5 edits were required; the helper would have made that
one).
- All alerts stay in a single PrometheusRule group. Groups in
PrometheusRule only matter for sequential recording-rule ordering,
per-group evaluation intervals, or execution isolation — none of
which apply here. AlertManager routes on labels (alertname,
severity), not on group names, so splitting by domain would
fragment without buying anything; worse, it would trap users who
add recording rules via extraAlertGroups into the wrong group.
Category-based routing on the consumer side is straightforward via
alertname regex (Source.*, Cert.*, CRL.*, Kube.*).
The Kubernetes source's transport layer had 9 distinct failure sites
that surfaced only as logs: LIST failure (with retry/backoff), WATCH
start failure, WATCH stream Error event, watch flap (close within 5s
of open), and namespace informer sync timeout — once for secrets, once
for configmaps. None had a corresponding metric, so an exporter that
was functionally up but operationally degraded (token rotation drift,
apiserver throttling, intermittent network) was invisible to ops.
Introduce x509_kube_transport_errors_total{source_name,resource,reason}
incremented at each of the 9 sites via a new Recorder interface on
k8s.Options. The interface keeps the source decoupled from
*registry.Registry; nil is valid (the unmetered path keeps the old
log-only behaviour) and tests plug a fake. main.go wires the actual
registry through.
Ship a paired KubeTransportErrors PrometheusRule alert (opt-in via
alertOnKubeTransportErrors, default true). docs/metrics.md documents
the new metric, the five reason codes, and cross-links the alert.
Detect-only x509_cert_collision_total ticks on every scrape under the
default CollisionAuto policy, even when the discriminator label
silently resolves the overlap and no certificate is dropped. Alerting
on it produced perma-firing pages for a benign config quirk.
Introduce x509_cert_collision_dropped_total: counts only the items
the registry actually threw away (CollisionNever policy). The
CertificateCollision PrometheusRule now points at the new counter so
the alert fires only when data is genuinely lost. The detect-only
counter stays available for dashboards and diagnostics.
Tests in pkg/registry cement the contract: dropped stays at 0 under
CollisionAuto and is positive under CollisionNever. docs/metrics.md
documents both counters and cross-links them.
Four new opt-in alerts close observability holes the existing rules
left uncovered:
- SourceDown (critical) — x509_source_up == 0 for 5m. Catches RBAC
failures, persistent K8s API errors, unreadable file paths. A source
that never converges would otherwise be silent while certs it should
watch are not checked.
- KeystorePassphraseFailures (warning) — increase on
x509_{pkcs12,jks}_passphrase_failures_total. A misconfigured
passphraseKey / passphraseSecretRef previously only leaked into
logs and an unwatched counter.
- CertificateNotYetValid (warning) — x509_cert_not_before > time().
Depends on exposeNotBeforeMetric: true; gated for explicit opt-in.
- CertificateCollision (warning) — increase on x509_cert_collision_total.
Two certs sharing a label set means one is silently invisible.
Each alert ships with an alertOnXxx toggle (default true) and its own
severity in values.yaml. Schema + README regenerated. helm lint, helm
template, schema fixtures and helm-examples all pass.
The X509ExporterReadErrors PrometheusRule alert matched x509_read_errors,
the v3 metric name. The v4 exporter emits x509_source_errors_total
instead — the alert never fired since the v4 rewrite. Same stale name
lingered in the values.yaml extraAlertGroups example and across 8
Grafana dashboard panels. All three sites renamed in lockstep.
Switch the alert's range function from delta() to increase() while
we're here. delta() is documented for gauges; for monotonic counters
increase() is the canonical choice and correctly handles counter
resets across exporter restarts.
