silverwindandsilverwind
bc8161c673
Jobs occasionally go silent ([example](https://gitea.com/gitea/runner/actions/runs/805045/jobs/1055123)) mid-run and Gitea reaped them after `ZOMBIE_TASK_TIMEOUT`, with no error in the log. This contains a number of related fixes, all with full test coverage: 1. Bound every RPC to Gitea with a timeout, a stalled report otherwise parked logs and heartbeats for the whole job. 2. Cap `runner.fetch_timeout` at that ceiling. 3. Let only the daemon loop close its own channel, the race panicked the process. 4. Stop the job on any terminal server result, not just `RESULT_CANCELLED`. 5. Report that result instead of relabelling it as cancelled. 6. Log reporting failures once at each end of an outage instead of discarding them. 7. Clamp the acknowledged log index, a too-large ack panicked on a slice bound. 8. Stop reading server health from a `FetchTask` deadline, it marked the runner healthy and reset the error backoff on a timeout. 9. Return an error from the Docker version probe instead of a `logrus` panic. 10. Pass the context to go-git's fetch and pull. 11. Fail the clone when a refresh dies on a cancelled context. 12. Set `terminationGracePeriodSeconds` in the Kubernetes examples. Also contains a deprecation fix for goreleaser. Reviewed-on: https://gitea.com/gitea/runner/pulls/1174 Reviewed-by: bircni <bircni@icloud.com> Co-authored-by: silverwind <me@silverwind.io>
Kubernetes Docker in Docker Deployment with gitea-runner
NOTE: Docker in Docker (dind) requires elevated privileges on Kubernetes. The current way to achieve this is to set the pod SecurityContext to privileged. Keep in mind that this is a potential security issue that has the potential for a malicious application to break out of the container context.
NOTE: dind-docker.yaml uses the native sidecar pattern (init container with restartPolicy: Always), which requires Kubernetes 1.29+ (or 1.28 with the SidecarContainers feature gate).
NOTE: A helm chart for gitea-runner also exists for easier deployments https://gitea.com/gitea/helm-actions
Each example persists two things, and it is worth knowing which is which:
-
/datais the runner's working directory. It holds the.runnerregistration file and, optionally, the config file — so the runner re-attaches to the server instead of registering again. -
The Docker daemon's data root holds the images pulled for jobs (
/var/lib/dockerfor the dind sidecar,/home/rootless/.local/share/dockerfordind-rootless). It is not under/data. If you drop this volume, the examples still work, but the image cache is discarded whenever the pod is recreated and every job re-pulls its images. -
Kubernetes SIGKILLs a pod 30s after SIGTERM by default, long before a job finishes and reports its result, which leaves tasks the server can only reap as zombies. The manifests raise
terminationGracePeriodSecondsto three hours, matching the systemd example and therunner.timeoutjob ceiling; setrunner.shutdown_timeoutbelow that so the runner drains jobs within the window rather than being killed mid-cleanup.
Files in this directory:
-
dind-docker.yamlHow to create a Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted. -
rootless-docker.yamlHow to create a rootless Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted. -
statefulset-dind.yamlStatefulSet variant of the dind example. Each replica gets a stable identity and its own persistent volume viavolumeClaimTemplates, so the runner keeps its.runnerregistration across restarts and reschedules instead of trying to register again.