Reloader/CLAUDE.md

Stakater Reloader Project Memory

Project Purpose

Reloader is a Kubernetes operator that automatically triggers rolling restarts of workloads when the ConfigMaps or Secrets they reference are updated. Without it, Kubernetes does not restart pods when configuration changes — operators must do it manually or rely on GitOps pipelines.

What it watches: ConfigMaps, Secrets, Namespaces, and (optionally) SecretProviderClassPodStatus (CSI-mounted secrets).

Workload types it can reload: Deployment, StatefulSet, DaemonSet, CronJob, Job, Argo Rollout, and OpenShift DeploymentConfig.

How restarts are triggered: Two strategies (selected via --reload-strategy):

1. env-vars (default) — injects an environment variable (STAKATER_{NAME}_{TYPE}) into every container with the SHA1 hash of the resource's data. A change in data changes the env var value, causing Kubernetes to restart pods.
2. annotations — writes the SHA1 hash into the pod template's annotations, which also forces a rollout.

The core problem it solves: ConfigMaps and Secrets are decoupled from pod lifecycle in Kubernetes. Applications reading config at startup see stale data after a config update unless pods are restarted. Reloader closes that gap automatically and selectively.

Potential improvements observed:

• Duplicate reload suppression: If a workload references both a ConfigMap and a Secret that are updated in the same controller reconcile cycle, it may get reloaded twice. Could be solved with a per-workload debounce map keyed by namespace/name/resourceVersion, flushed after a short TTL.
• CronJob/Job reload is destructive: Jobs are deleted and recreated on change, which loses run history. Could instead only annotate the CronJob template without spawning a new Job.
• No per-resource reload rate limiting: A rapid-fire ConfigMap update (e.g., from a CI pipeline) can trigger many restarts. A cooldown window per resource would help.
• CSI integration gap: CSI volumes are watched at the SecretProviderClassPodStatus level but the link back to the workload is indirect and may miss edge cases. Needs a direct map from SecretProviderClass → workloads that mount it.

---

Repo Map

Path	Owns	Inspect when
main.go	Entry point, delegates to app.Run()	Never needs changes
internal/pkg/app/	Run() bootstrap, Cobra command wiring	Startup sequence changes
internal/pkg/cmd/	CLI flags parsing, startReloader(), controller/HA wiring	Adding new flags or startup behavior
internal/pkg/controller/	Informer/queue per resource type, event handlers (Add/Update/Delete)	Watching new resource types, queue tuning
internal/pkg/handler/	Per-event handlers (create, update, delete), doRollingUpgrade(), pause deployment	Core reload logic changes
internal/pkg/callbacks/	Workload-specific get/list/update/patch functions, RollingUpgradeFuncs struct	Adding new workload types
internal/pkg/options/	All CLI flag variables, defaults, ArgoRolloutStrategy type	Adding or renaming flags
internal/pkg/constants/	Constants: env var postfixes, annotation prefix, strategy names, HA lock name	Renaming global identifiers
internal/pkg/metrics/	Prometheus Collectors struct, all metric registration and recording helpers	Adding metrics
internal/pkg/alerts/	Slack/Teams/GChat/raw webhook alerting, env var config	Alert sink changes
internal/pkg/util/	SHA generation via crypto/sha.go, env var name conversion, namespace/label utilities	Utility/hash changes
internal/pkg/crypto/	GenerateSHA(data) — SHA1 hex digest	Hash algorithm changes
internal/pkg/leadership/	Leader election via Kubernetes Lease, HA stop/start of controllers	HA behavior changes
internal/pkg/testutil/	Fake Kubernetes objects for unit tests	Writing new tests
pkg/common/	ReloadCheckResult, ReloaderOptions, ShouldReload() logic, Config struct	Reload decision logic, annotation precedence
pkg/kube/	Clients struct (k8s + OpenShift + Argo + CSI), GetKubernetesClient(), ResourceMap	Client initialization, new CRD clients
deployments/	Helm chart (deployments/kubernetes/chart/reloader/), Kustomize manifests	Helm values, RBAC, deployment config
docs/	User-facing annotation documentation, architecture notes	Writing docs or confirming annotation behavior
scripts/	Shell scripts used by CI and Makefile	Build/release pipeline
test/loadtest/	Load test CLI (cmd/loadtest), 13 scenarios (S1–S13), Kind cluster setup	Performance testing, regression benchmarks
.github/	CI workflows: lint, test, Kind e2e, multi-arch Docker build, release	CI changes

---

Core Runtime Flow

1. Entry — main.go:10 calls app.Run().

2. CLI Init — internal/pkg/app/app.go calls cmd.NewReloaderCommand() which registers all Cobra flags from options/flags.go and runs startReloader().

3. Client Setup — pkg/kube/client.go: builds kube.Clients with:

• kubernetes.Interface — standard k8s client
• appsclient.Interface — OpenShift client (auto-detected by probing deploymentconfigs)
• argorollout.Interface — if --is-Argo-Rollouts=true
• csiclient.Interface — if --enable-csi-integration

4. Controller Creation — startReloader() iterates kube.ResourceMap (configmaps, secrets, namespaces, and optionally secretproviderclasspodstatuses) and calls controller.NewController() for each resource in each watched namespace.

5. Informer/Queue — controller.NewController():

• Creates a cache.NewFilteredListWatchFromClient with label/field selectors.
• Registers Add, Update, Delete event handlers.
• Creates a workqueue.TypedRateLimitingQueue for async processing.

6. Event Detection:

• Add — enqueues only if ReloadOnCreate is enabled (skips during initial sync unless SyncAfterRestart).
• Update — compares SHA of old vs new object data; enqueues only on real changes.
• Delete — enqueues only if ReloadOnDelete is enabled.
• Namespace events update selectedNamespacesCache for namespace-selector filtering.

7. Handler Dispatch — The queue worker calls handler.Handle() on the dequeued item. Three handler types:

• ResourceCreatedHandler (create.go) — fires doRollingUpgrade or sends webhook.
• ResourceUpdatedHandler (update.go) — fires doRollingUpgrade or sends webhook.
• ResourceDeleteHandler (delete.go) — calls invokeDeleteStrategy (removes env vars or clears annotation).

8. Workload Discovery — doRollingUpgrade() (upgrade.go:181) calls rollingUpgrade() for each workload type. For each type, ItemsFunc lists all workloads in the namespace, then pkg/common.ShouldReload() checks annotations to decide which ones need reloading.

9. Reload Execution — invokeReloadStrategy() either:

• env-vars: mutates container env vars; uses JSON patch if SupportsPatch=true, full update otherwise.
• annotations: writes SHA to pod template annotations; same patch/update split.

10. Post-reload — optionally pauses the Deployment via pause_deployment.go, records Kubernetes Events via recorder, updates Prometheus metrics, sends alert webhooks.

HA Mode: if --enable-ha, internal/pkg/leadership/ runs Kubernetes Lease-based leader election. Only the leader runs controllers; losing leadership stops them and marks the pod unhealthy.

HTTP Server: port :9090 serves /metrics (Prometheus) and liveness/readiness probes.

---

Reload Behavior And Annotations

All annotation names are configurable via CLI flags; the values below are defaults.

Trigger Annotations (on workloads)

Annotation	Value	Behavior
reloader.stakater.com/auto	"true"	Reload on change to any ConfigMap or Secret referenced by the workload (via envFrom, env valueFrom, or volumes)
configmap.reloader.stakater.com/auto	"true"	Reload on change to any referenced ConfigMap only
secret.reloader.stakater.com/auto	"true"	Reload on change to any referenced Secret only
secretproviderclass.reloader.stakater.com/auto	"true"	Reload on change to any referenced SecretProviderClass only
configmap.reloader.stakater.com/reload	"cm1,cm2"	Reload only when the named ConfigMaps change (regex supported)
secret.reloader.stakater.com/reload	"sec1,sec2"	Reload only when the named Secrets change (regex supported)
secretproviderclass.reloader.stakater.com/reload	"spc1"	Reload only when the named SecretProviderClass changes
reloader.stakater.com/search	"true"	Reload when any ConfigMap/Secret tagged with reloader.stakater.com/match: "true" changes

Exclude Annotations (on workloads)

Annotation	Value	Behavior
reloader.stakater.com/ignore	"true"	Skip this workload entirely
configmaps.exclude.reloader.stakater.com/reload	"cm1,cm2"	Exclude these named ConfigMaps from triggering reload
secrets.exclude.reloader.stakater.com/reload	"sec1,sec2"	Exclude these named Secrets
secretproviderclasses.exclude.reloader.stakater.com/reload	"spc1"	Exclude these named SecretProviderClasses

Behavior Annotations (on workloads)

Annotation	Value	Behavior
reloader.stakater.com/rollout-strategy	"restart" or "rollout"	For Argo Rollouts: "restart" uses restartAt, "rollout" (default) uses full rollout update
deployment.reloader.stakater.com/pause-period	Go duration e.g. "30s"	Pause Deployment for this duration after reload
deployment.reloader.stakater.com/paused-at	RFC3339 timestamp	Set by Reloader to track pause start time; do not set manually

Search/Match Pattern

The reloader.stakater.com/search annotation on a workload pairs with reloader.stakater.com/match: "true" on a ConfigMap or Secret. Any workload with search: true will reload when any match: true resource changes.

Global Flag Overrides

• --auto-reload-all — reload all workloads on any ConfigMap/Secret change; annotation not required.
• --resources-to-ignore=configMaps or =secrets — skip one type entirely.
• --ignored-workload-types=jobs,cronjobs — skip Job and CronJob reload.
• --namespaces-to-ignore — comma-separated namespace names to skip.
• --namespace-selector — only watch namespaces with matching labels.
• --resource-label-selector — only watch ConfigMaps/Secrets with matching labels.

Precedence Rules

1. reloader.stakater.com/ignore: "true" wins everything — workload is skipped.
2. Exclude annotations override include annotations for specific named resources.
3. Named annotations (.../reload) are checked before auto annotations.
4. --auto-reload-all is the lowest-priority fallback (only applies if no annotation matches).
5. Annotations are checked on both the workload and its pod template (pod template takes precedence in some paths — verify in pkg/common/common.go:ShouldReload()).

---

Workload Support

Workload	SupportsPatch	Update Mechanism	Key files
Deployment	Yes	JSON patch or full update	callbacks/rolling_upgrade.go, handler/upgrade.go:38
StatefulSet	Yes	JSON patch or full update	callbacks/rolling_upgrade.go, handler/upgrade.go:109
DaemonSet	Yes	JSON patch or full update	callbacks/rolling_upgrade.go, handler/upgrade.go:91
CronJob	No	Creates a new Job from CronJob spec (adds cronjob.kubernetes.io/instantiate: manual)	callbacks.CreateJobFromCronjob, handler/upgrade.go:55
Job	No	Deletes old Job, creates new one (strips ResourceVersion, UID, Status, controller labels)	callbacks.ReCreateJobFromjob, handler/upgrade.go:73
Argo Rollout	No	Full update via Argo Rollouts client	callbacks.UpdateRollout, handler/upgrade.go:127; requires --is-Argo-Rollouts=true
DeploymentConfig	Yes	OpenShift DeploymentConfigs API	callbacks/rolling_upgrade.go; auto-detected by probing deploymentconfigs

Reload flow per workload: doRollingUpgrade() → rollingUpgrade() per type → ItemsFunc lists workloads → ShouldReload() filters → invokeReloadStrategy() patches or updates → optional pause + metrics + alert.

---

CSI Support

Enabled by: --enable-csi-integration

What is watched: SecretProviderClassPodStatus resources (from sigs.k8s.io/secrets-store-csi-driver). Resource name constant: constants.SecretProviderClassController = "secretproviderclasspodstatuses".

How it works:

1. The CSI driver injects secrets into pods as volume mounts and tracks injection state via SecretProviderClassPodStatus objects.
2. Reloader watches these objects for version changes.
3. When a version change is detected, it computes a SHA of the object's IDs and versions.
4. It then looks up the referenced SecretProviderClass and treats the event like a Secret update, triggering workload reloads.

Workload annotation: secretproviderclass.reloader.stakater.com/reload: "my-spc" or secretproviderclass.reloader.stakater.com/auto: "true".

Required: CSI CRDs must be installed in the cluster. Reloader auto-detects their presence at startup.

Env var postfix: STAKATER_{NAME}_SECRETPROVIDERCLASS.

Known limitations:

• Only works for secrets mounted as volumes via CSI, not env-var-based CSI injection.
• The link from SecretProviderClassPodStatus → workload is indirect; edge cases may be missed.
• Requires the CSI driver CRDs to be pre-installed; Reloader won't start CSI controller if CRDs are absent.

---

Build, Test, And Run Commands

Go version: go 1.26.2 (from go.mod)

Purpose	Command
Run locally	go run ./main.go
Build binary	make build → go build -o Reloader
Unit tests	make test → go test -timeout 1800s -v ./...
Lint	make lint → golangci-lint run ./... (v2.6.1)
Docker build (single arch)	make build-image ARCH=amd64
Docker push	make push
Full release (build+push+manifest)	make release ARCH=amd64
Multi-arch release	make release-all
Generate k8s manifests	make k8s-manifests (Kustomize v5.3.0)
Load test (quick)	make loadtest-quick LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=... (runs S1, S4, S6)
Load test (full)	make loadtest-full LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...
Load test (custom)	make loadtest LOADTEST_SCENARIOS=S1,S3 LOADTEST_DURATION=120

Docker image: ghcr.io/stakater/reloader — multi-arch (amd64, arm64, arm), distroless nonroot base.

Helm chart: deployments/kubernetes/chart/reloader/ — install via Helm or kubectl apply -f deployments/kubernetes/reloader.yaml.

---

Coding Conventions

Package boundaries: Each internal/pkg/<name> package has a single clear responsibility. Cross-package access goes through exported types/functions only.

Error handling: logrus.Errorf(...) for non-fatal, logrus.Fatalf(...) for startup failures. Errors are returned up the call stack and logged at the point of action, not at every layer. Retry uses k8s.io/client-go/util/retry.RetryOnConflict.

Logging: logrus with structured fields. Format controlled by --log-format=json flag. Log level controlled by --log-level. Messages follow the pattern: "Changes detected in '%s' of type '%s' in namespace '%s'".

Kubernetes client patterns: All k8s operations go through the kube.Clients struct. Use context.TODO() for context (no request-scoped contexts). List/watch via informers, not polling.

Callback pattern: Workload-specific logic is encapsulated in callbacks.RollingUpgradeFuncs structs returned by handler.Get*RollingUpgradeFuncs(). Adding a new workload type = add a new RollingUpgradeFuncs factory function and call it in doRollingUpgrade().

Test style: Standard testing.T, testify/assert. Fake k8s objects via testutil/kube.go. Tests live alongside source in the same package. Large integration-style tests in handler/upgrade_test.go.

Naming patterns:

• Annotation variables: XxxUpdateOnChangeAnnotation, XxxReloaderAutoAnnotation
• Callback funcs: GetXxxItem, GetXxxItems, UpdateXxx, PatchXxx
• Handler factories: GetXxxRollingUpgradeFuncs()

Adding new behavior: Add flag to options/flags.go + common.ReloaderOptions struct → wire in cmd/reloader.go → implement logic in handler/ or callbacks/ → add metrics recording → write tests in *_test.go.

---

Gotchas And Risks

Duplicate reloads: If a workload references multiple ConfigMaps/Secrets and all change simultaneously, each change event fires a separate reload. No deduplication exists within a reconcile window. This can cause unnecessary rolling restarts.

Controller init guard: secretControllerInitialized and configmapControllerInitialized booleans in controller/controller.go prevent processing Add events during the initial list/sync (to avoid reloading everything on startup). If --sync-after-restart is set, both are pre-set to true, bypassing the guard. Be careful when this interacts with --reload-on-create.

Namespace filtering: --namespaces-to-ignore does a name match; --namespace-selector watches namespaces by label and caches them in selectedNamespacesCache. The cache is updated on Namespace Add/Update/Delete events. A race between cache population and first ConfigMap event could cause missed reloads on startup in label-selected deployments.

RBAC: Reloader requires get/list/watch on secrets and configmaps, and get/list/watch/update/patch on all workload types it manages. Missing RBAC silently causes no reloads (not an error — just empty lists). Check ClusterRole in deployments/kubernetes/chart/reloader/templates/.

GitOps drift: If a GitOps tool (Flux, ArgoCD) manages the same Deployments, annotation or env var changes made by Reloader will be detected as drift and reverted. Use --reload-strategy=annotations with care in GitOps setups; env-vars strategy is generally safer since it modifies the pod template rather than workload-level annotations.

Annotation precedence edge case: Annotations are checked first on the workload object, then on the pod template. If both are set to conflicting values, the behavior depends on which path ShouldReload() hits first. Verify in pkg/common/common.go.

CronJob/Job destructive reload: Job recreation deletes the old Job. Any in-flight pod from that Job will be terminated. This is intentional but surprising. There is no protection for long-running jobs.

OpenShift DeploymentConfig: Auto-detected by probing for the deploymentconfigs resource. If the probe fails at startup, OpenShift support is silently disabled. Check pkg/kube/client.go.

Argo Rollouts: Must be explicitly enabled via --is-Argo-Rollouts=true. Without it, Rollout objects are never listed. The SupportsPatch=false means full object updates are used — be aware of potential conflicts with Argo's own controller.

CSI rotation behavior: SecretProviderClassPodStatus is updated by the CSI driver when secrets rotate. Reloader reacts to those updates. However, if the CSI driver updates the status in a way that doesn't change the versions Reloader tracks, the reload will be missed.

Backward compatibility: Annotation names are configurable, so changing defaults would break existing clusters. Never change default annotation values without a migration path.

Tests to update for risky changes: handler/upgrade_test.go (large suite covering all workload types), controller/controller_test.go (event handling), pkg/common/common_test.go (reload decision logic).

---

Open Questions

• Exact ShouldReload() precedence: The code in pkg/common/common.go checks annotations in a specific order. The exact tie-breaking when both workload-level and pod-template-level annotations are set should be verified by reading that function fully before making annotation behavior changes.
• CSI → workload mapping: How exactly does Reloader map a SecretProviderClassPodStatus change back to workloads? Is it via the SecretProviderClass name matching an annotation on the workload, or via volume reference scanning? Needs confirmation before adding CSI-related features.
• ContainerPatchPathFunc field: RollingUpgradeFuncs has a ContainerPatchPathFunc field but it is not documented — unclear if/how it differs from ContainersFunc in patch scenarios.
• Webhook vs alert: --webhook-url replaces reloading with a POST request. ALERT_WEBHOOK_URL env var sends an alert after reloading. These are two different mechanisms; the naming is confusing and easy to conflate.
• Load test scenarios S7–S13: Only S1, S4, and S6 are confirmed from CI. The behavior and coverage of the remaining scenarios is unknown without reading test/loadtest/ in full.
• SyncAfterRestart semantics: Flag docs say it "syncs add events after restart" but only if ReloadOnCreate is also true. The interaction between these two flags in HA mode (where controllers restart on leader change) needs verification.

---

Important Files

File	Description
internal/pkg/cmd/reloader.go	startReloader() — main wiring of clients, controllers, HA, and HTTP server
internal/pkg/handler/upgrade.go	doRollingUpgrade() + all Get*RollingUpgradeFuncs() factories
internal/pkg/callbacks/rolling_upgrade.go	All workload-specific get/update/patch implementations
pkg/common/common.go	ShouldReload() — the annotation decision tree
internal/pkg/options/flags.go	Every configurable option with defaults
internal/pkg/controller/controller.go	Informer setup, queue, event handlers
pkg/kube/client.go	Multi-client initialization and OpenShift/CSI detection
internal/pkg/handler/pause_deployment.go	Pause/resume deployment logic with timers
internal/pkg/leadership/leadership.go	HA leader election
internal/pkg/metrics/prometheus.go	All Prometheus collector definitions
internal/pkg/alerts/alert.go	Slack/Teams/GChat alerting
internal/pkg/constants/constants.go	Global constants (env var prefixes, annotation prefix, strategy names)
deployments/kubernetes/chart/reloader/values.yaml	Helm chart defaults — source of truth for production config
handler/upgrade_test.go	Largest test suite; must be updated for any reload logic change
Makefile	All build/test/release/loadtest commands