[stable/elasticsearch-curator] Make restartPolicy configurable (#17748)

We're currently in a situation where our ES _sometimes_ fails to clear
up indexes, especially under load, but will eventually succeed when the
job is restarted.

There are two approaches to this kind of issues as far as I understand:

- Let the CronJob recreate a new Job resource, which will create a new
  Pod and succeed
- Set the Job's restartPolicy to Always or OnFailure, and let the Job
  finish

The first option is currently doable through this chart, but hits the
default Prometheus alerting rules that will warn you if a Job resource
has failed and is left in failed state for a while. This is of course a
spurious warning as the CronJob has eventually succeeded... But it's
there :(

By giving the user the possibility to change the restartPolicy, we make
that second option available to go around Prometheus' over-sensitivity.

Note: I relaise it's also possible to delete failed Jobs by setting the
`failedJobsHistoryLimit` to zero, but that would hide _actual_ failures
as well as temporary flakiness.

Signed-off-by: Florent Delannoy <florent.delannoy.external@ynap.com>

stable/elasticsearch-curator/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v1
 appVersion: "5.7.6"
 description: A Helm chart for Elasticsearch Curator
 name: elasticsearch-curator
-version: 2.0.3
+version: 2.1.0
 home: https://github.com/elastic/curator
 keywords:
 - curator

stable/elasticsearch-curator/README.md

@@ -50,6 +50,7 @@ their default values.
 | `cronjob.concurrencyPolicy`          | `Allow|Forbid|Replace` concurrent jobs                      | `nil`                                        |
 | `cronjob.failedJobsHistoryLimit`     | Specify the number of failed Jobs to keep                   | `nil`                                        |
 | `cronjob.successfulJobsHistoryLimit` | Specify the number of completed Jobs to keep                | `nil`                                        |
+| `cronjob.jobRestartPolicy`           | Control the Job restartPolicy                               | `Never`                                        |
 | `pod.annotations`                    | Annotations to add to the pod                               | {}                                           |
 | `dryrun`                             | Run Curator in dry-run mode                                 | `false`                                      |
 | `env`                                | Environment variables to add to the cronjob container       | {}                                           |

stable/elasticsearch-curator/templates/cronjob.yaml

@@ -45,7 +45,7 @@ spec:
 {{- if .Values.extraVolumes }}
 {{ toYaml .Values.extraVolumes | indent 12 }}
 {{- end }}
-          restartPolicy: Never
+          restartPolicy: {{ .Values.cronjob.jobRestartPolicy }}
 {{- if .Values.priorityClassName }}
           priorityClassName: "{{ .Values.priorityClassName }}"
 {{- end }}

stable/elasticsearch-curator/values.yaml

@@ -9,6 +9,7 @@ cronjob:
   concurrencyPolicy: ""
   failedJobsHistoryLimit: ""
   successfulJobsHistoryLimit: ""
+  jobRestartPolicy: Never
 
 pod:
   annotations: {}