mirror of
https://github.com/helm/charts.git
synced 2026-08-23 22:37:45 +00:00
[stable/concourse] Worker lifecycle management (#2109)
* Make Concourse resilient to worker pod restarts. Add time between worker stop and start. A shell is used to start the worker and sleep after the process terminates. SIGTERM is not propagated to the concourse process as a result, but this is actually desired for the shutdown process, below. Gracefully de-register a worker using `retire-worker` instead of `land-worker` in the `preStop` hook. This tells concourse that the worker and its container volumes will not be coming back, which is the case when worker pods restart. Because SIGTERM doesn't reach the concourse process, `retire-worker` is the only mechanism used for graceful shutdown. If all containers are cleaned up within `worker.terminationGracePeriodSeconds`, the concourse process terminates on its own and the pod finishes terminating, otherwise the pod's container is killed. In either case, the worker disappears on ATC's end. See https://concourse.ci/worker-internals.html for more info. Fixes https://github.com/kubernetes/charts/issues/2103 * Add a livenessProbe to Concourse workers that detects fatal errors. Logs seem to be the only reliable way of detecting issues that prevent workers from taking on work. Tee the log to a file, and truncate it each time the livenessProbe runs. Fixes https://github.com/kubernetes/charts/issues/2104 * Bump version to 0.7.0 and add myself to maintainers list.
This commit is contained in:
Will Tran
Vic Iglesias
@@ -1,5 +1,5 @@
|
||||
name: concourse
|
||||
version: 0.6.0
|
||||
version: 0.7.0
|
||||
appVersion: 3.3.2
|
||||
description: Concourse is a simple and scalable CI system.
|
||||
icon: https://avatars1.githubusercontent.com/u/7809479
|
||||
@@ -16,4 +16,6 @@ maintainers:
|
||||
email: frodenas@gmail.com
|
||||
- name: viglesiasce
|
||||
email: viglesias@google.com
|
||||
- name: william-tran
|
||||
email: will@autonomic.ai
|
||||
engine: gotpl
|
||||
|
||||
@@ -55,27 +55,11 @@ $ kubectl scale statefulset my-release-worker --replicas=3
|
||||
|
||||
### Restarting workers
|
||||
|
||||
If worker pods go down, their persistent volumes are changed, or if you're having other issues with them, you'll need to restart the workers. Concourse workers were designed to be deployed onto infrastructure VMs which are less "ephemeral" than pods, so it isn't good at detecting when a worker goes down and comes back under the same hostname.
|
||||
If a worker isn't taking on work, you can restart the worker with `kubectl delete pod`. This will initiate a graceful shutdown by "retiring" the worker, with some waiting time before the worker starts up again to ensure concourse doesn't try looking for old volumes on the new worker. The values `worker.postStopDelaySeconds` and `worker.terminationGracePeriodSeconds` can be used to tune this.
|
||||
|
||||
Scale the workers down to 0:
|
||||
### Worker Liveness Probe
|
||||
|
||||
```
|
||||
kubectl scale statefulset concourse-worker --replicas=0
|
||||
|
||||
```
|
||||
|
||||
And then `fly workers` until the workers are detected to be `stalled`. Then for each worker
|
||||
```
|
||||
fly prune-worker -w concourse-worker-0
|
||||
fly prune-worker -w concourse-worker-1
|
||||
...
|
||||
|
||||
```
|
||||
And finally
|
||||
|
||||
```
|
||||
kubectl scale statefulset concourse-worker --replicas=3
|
||||
```
|
||||
The worker's Liveness Probe will trigger a restart of the worker if it detects unrecoverable errors, by looking at the worker's logs. The set of strings used to identify such errors could change in the future, but can be tuned with `worker.fatalErrors`. See [values.yaml](values.yaml) for the defaults.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -135,9 +119,12 @@ The following tables lists the configurable parameters of the Concourse chart an
|
||||
| `web.ingress.tls` | Concourse Web Ingress TLS configuration | `[]` |
|
||||
| `worker.nameOverride` | Override the Concourse Worker components name| `worker` |
|
||||
| `worker.replicas` | Number of Concourse Worker replicas | `2` |
|
||||
| `worker.minAvailable` | Minimun number of workers available after an eviction | `1` |
|
||||
| `worker.minAvailable` | Minimum number of workers available after an eviction | `1` |
|
||||
| `worker.resources` | Concourse Worker resource requests and limits | `{requests: {cpu: "100m", memory: "512Mi"}}` |
|
||||
| `worker.additionalAffinities` | Additional affinities to apply to worker pods. E.g: node affinity | `nil` |
|
||||
| `worker.postStopDelaySeconds` | Time to wait after graceful shutdown of worker before starting up again | `60` |
|
||||
| `worker.terminationGracePeriodSeconds` | Upper bound for graceful shutdown, including `worker.postStopDelaySeconds` | `120` |
|
||||
| `worker.fatalErrors` | Newline delimited strings which, when logged, should trigger a restart of the worker | *See [values.yaml](values.yaml)* |
|
||||
| `persistence.enabled` | Enable Concourse persistence using Persistent Volume Claims | `true` |
|
||||
| `persistence.worker.class` | Concourse Worker Persistent Volume Storage Class | `generic` |
|
||||
| `persistence.worker.accessMode` | Concourse Worker Persistent Volume Access Mode | `ReadWriteOnce` |
|
||||
|
||||
@@ -37,3 +37,6 @@ data:
|
||||
generic-oauth-auth-url-param: {{ default "" .Values.concourse.genericOauthAuthUrlParam | quote }}
|
||||
generic-oauth-scope: {{ default "" .Values.concourse.genericOauthScope | quote }}
|
||||
generic-oauth-token-url: {{ default "" .Values.concourse.genericOauthTokenUrl | quote }}
|
||||
worker-post-stop-delay-seconds: {{ .Values.worker.postStopDelaySeconds | quote }}
|
||||
worker-fatal-errors: {{ default "" .Values.worker.fatalErrors | quote }}
|
||||
|
||||
@@ -20,20 +20,42 @@ spec:
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
spec:
|
||||
terminationGracePeriodSeconds: 60
|
||||
terminationGracePeriodSeconds: {{ .Values.worker.terminationGracePeriodSeconds }}
|
||||
containers:
|
||||
- name: {{ template "concourse.worker.fullname" . }}
|
||||
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
|
||||
imagePullPolicy: {{ default "" .Values.imagePullPolicy | quote }}
|
||||
command:
|
||||
- /bin/sh
|
||||
args:
|
||||
- "worker"
|
||||
- -c
|
||||
- |-
|
||||
cp /dev/null /concourse-work-dir/.liveness_probe
|
||||
concourse worker --name=${HOSTNAME} | tee -a /concourse-work-dir/.liveness_probe
|
||||
sleep ${POST_STOP_DELAY_SECONDS}
|
||||
livenessProbe:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |-
|
||||
FATAL_ERRORS=$( echo "${LIVENESS_PROBE_FATAL_ERRORS}" | grep -q '\S' && \
|
||||
grep -F "${LIVENESS_PROBE_FATAL_ERRORS}" /concourse-work-dir/.liveness_probe )
|
||||
cp /dev/null /concourse-work-dir/.liveness_probe
|
||||
if [ ! -z "${FATAL_ERRORS}" ]; then
|
||||
>&2 echo "Fatal error detected: ${FATAL_ERRORS}"
|
||||
exit 1
|
||||
fi
|
||||
failureThreshold: 1
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command:
|
||||
- "concourse"
|
||||
- "land-worker"
|
||||
- "--name=${HOSTNAME}"
|
||||
- "/bin/sh"
|
||||
- "-c"
|
||||
- "concourse retire-worker --name=${HOSTNAME}"
|
||||
env:
|
||||
- name: CONCOURSE_TSA_HOST
|
||||
valueFrom:
|
||||
@@ -66,6 +88,16 @@ spec:
|
||||
configMapKeyRef:
|
||||
name: {{ template "concourse.concourse.fullname" . }}
|
||||
key: concourse-baggageclaim-driver
|
||||
- name: POST_STOP_DELAY_SECONDS
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: {{ template "concourse.concourse.fullname" . }}
|
||||
key: worker-post-stop-delay-seconds
|
||||
- name: LIVENESS_PROBE_FATAL_ERRORS
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: {{ template "concourse.concourse.fullname" . }}
|
||||
key: worker-fatal-errors
|
||||
resources:
|
||||
{{ toYaml .Values.worker.resources | indent 12 }}
|
||||
securityContext:
|
||||
|
||||
@@ -376,6 +376,25 @@ worker:
|
||||
# values:
|
||||
# - "true"
|
||||
|
||||
## Time to delay after the worker process shuts down. This inserts time between shutdown and startup
|
||||
## to avoid errors caused by a worker restart.
|
||||
postStopDelaySeconds: 60
|
||||
|
||||
## Time to allow the pod to terminate before being forcefully terminated. This should include
|
||||
## postStopDelaySeconds, and should additionally provide time for the worker to retire, e.g.
|
||||
## = postStopDelaySeconds + max time to allow the worker to drain its tasks. See
|
||||
## https://concourse.ci/worker-internals.html for worker lifecycle semantics.
|
||||
terminationGracePeriodSeconds: 120
|
||||
|
||||
## If any of the strings are found in logs, the worker's livenessProbe will fail and trigger a pod restart.
|
||||
## Specify one string per line, exact matching is used.
|
||||
##
|
||||
## "guardian.api.garden-server.create.failed" appears when the worker's filesystem has issues.
|
||||
## "unknown handle" appears if a worker didn't cleanly restart.
|
||||
fatalErrors: |-
|
||||
guardian.api.garden-server.create.failed
|
||||
unknown handle
|
||||
|
||||
## Persistent Volume Storage configuration.
|
||||
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
|
||||
##
|
||||
|
||||
Reference in New Issue
Block a user