Helm Chart for Spark Operator
This is the Helm chart for the Kubernetes Operator for Apache Spark.
Prerequisites
The Operator requires Kubernetes version 1.8 and above because it relies on garbage collection of custom resources. If customization of driver and executor pods (through mounting custom ConfigMaps and volumes) is desired, then the Mutating Admission Webhook needs to be enabled and it only became beta in Kubernetes 1.9.
Installing the chart
The chart can be installed by running:
$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
$ helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default
Note that you need to use the --namespace flag during helm install to specify in which namespace you want to install the operator. The namespace can be existing or not. When it's not available, Helm would take care of creating the namespace. Note that this namespace has no relation to the namespace where you would like to deploy Spark jobs (i.e. the setting sparkJobNamespace shown in the table below). They can be the same namespace or different ones.
Configuration
The following table lists the configurable parameters of the Spark operator chart and their default values.
| Parameter | Description | Default |
|---|---|---|
operatorImageName |
The name of the operator image | gcr.io/spark-operator/spark-operator |
operatorVersion |
The version of the operator to install | v1beta2-1.0.1-2.4.4 |
imagePullPolicy |
Docker image pull policy | IfNotPresent |
replicas |
The number of replicas of the operator Deployment | 1 |
sparkJobNamespace |
K8s namespace where Spark jobs are to be deployed | `` |
enableWebhook |
Whether to enable mutating admission webhook | false |
enableMetrics |
Whether to expose metrics to be scraped by Premetheus | true |
controllerThreads |
Number of worker threads used by the SparkApplication controller | 10 |
ingressUrlFormat |
Ingress URL format | "" |
logLevel |
Logging verbosity level | 2 |
installCrds |
Whether to install CRDs | true |
metricsPort |
Port for the metrics endpoint | 10254 |
metricsEndpoint |
Metrics endpoint | "/metrics" |
metricsPrefix |
Prefix for the metrics | "" |
podAnnotations |
annotations to be added to pods | {} |
resyncInterval |
Informer resync interval in seconds | 30 |
webhookPort |
Service port of the webhook server | 8080 |
resources |
Resources needed for the sparkoperator deployment | {} |
enableBatchScheduler |
Whether to enable batch scheduler for pod scheduling | false |
enableResourceQuotaEnforcement |
Whether to enable the ResourceQuota enforcement for SparkApplication resources. Requires the webhook to be enabled by setting enableWebhook to true. | false |
enableLeaderElection |
Whether to enable leader election when the operator Deployment has more than one replica, i.e., when replicas is greater than 1. |
false |
Specify each parameter using the --set key=value[,key=value] argument to helm install.