deprecated-helm-charts/stable/spark

⚠️ Repo Archive Notice

As of Nov 13, 2020, charts in this repo will no longer be updated. For more information, see the Helm Charts Deprecation and Archive Notice, and Update.

Apache Spark Helm Chart

Apache Spark is a fast and general-purpose cluster computing system including Apache Zeppelin.

• http://spark.apache.org/
• https://zeppelin.apache.org/

Inspired from Helm Classic chart https://github.com/helm/charts

DEPRECATION NOTICE

This chart is deprecated and no longer supported.

Chart Details

This chart will do the following:

• 1 x Spark Master with port 8080 exposed on an external LoadBalancer
• 3 x Spark Workers with HorizontalPodAutoscaler to scale to max 10 pods when CPU hits 50% of 100m
• 1 x Zeppelin with port 8080 exposed on an external LoadBalancer
• All using Kubernetes Deployments

Prerequisites

• Assumes that serviceAccount tokens are available under hostname metadata. (Works on GKE by default) URL -- http://metadata/computeMetadata/v1/instance/service-accounts/default/token

Installing the Chart

To install the chart with the release name my-release:

$ helm install --name my-release stable/spark

Configuration

The following table lists the configurable parameters of the Spark chart and their default values.

Spark Master

Parameter	Description	Default
Master.Name	Spark master name	spark-master
Master.Image	Container image name	k8s.gcr.io/spark
Master.ImageTag	Container image tag	1.5.1_v3
Master.Replicas	k8s deployment replicas	1
Master.Component	k8s selector key	spark-master
Master.Cpu	container requested cpu	100m
Master.Memory	container requested memory	512Mi
Master.ServicePort	k8s service port	7077
Master.ContainerPort	Container listening port	7077
Master.DaemonMemory	Master JVM Xms and Xmx option	1g
Master.ServiceType	Kubernetes Service type	LoadBalancer

Spark WebUi

Parameter	Description	Default
WebUi.Name	Spark webui name	spark-webui
WebUi.ServicePort	k8s service port	8080
WebUi.ContainerPort	Container listening port	8080

Spark Worker

Parameter	Description	Default
Worker.Name	Spark worker name	spark-worker
Worker.Image	Container image name	k8s.gcr.io/spark
Worker.ImageTag	Container image tag	1.5.1_v3
Worker.Replicas	k8s hpa and deployment replicas	3
Worker.ReplicasMax	k8s hpa max replicas	10
Worker.Component	k8s selector key	spark-worker
Worker.Cpu	container requested cpu	100m
Worker.Memory	container requested memory	512Mi
Worker.ContainerPort	Container listening port	7077
Worker.CpuTargetPercentage	k8s hpa cpu targetPercentage	50
Worker.DaemonMemory	Worker JVM Xms and Xmx setting	1g
Worker.ExecutorMemory	Worker memory available for executor	1g
Worker.Autoscaling	Enable horizontal pod autoscaling	false

Zeppelin

Parameter	Description	Default
Zeppelin.Name	Zeppelin name	zeppelin-controller
Zeppelin.Image	Container image name	apache/zeppelin
Zeppelin.ImageTag	Container image tag	0.7.3
Zeppelin.Replicas	k8s deployment replicas	1
Zeppelin.Component	k8s selector key	zeppelin
Zeppelin.Cpu	container requested cpu	100m
Zeppelin.ServicePort	k8s service port	8080
Zeppelin.ContainerPort	Container listening port	8080
Zeppelin.Ingress.Enabled	if true, an ingress is created	false
Zeppelin.Ingress.Annotations	annotations for the ingress	{}
Zeppelin.Ingress.Path	the ingress path	/
Zeppelin.Ingress.Hosts	a list of ingress hosts	[zeppelin.example.com]
Zeppelin.Ingress.Tls	a list of IngressTLS items	[]
Zeppelin.ServiceType	Kubernetes Service type	LoadBalancer
Zeppelin.Persistence.Config.Enabled	Enable Persistence for configuration	false
Zeppelin.Persistence.Config.StorageClass	Volume storageClassName	- (no dynamic provisioning)
Zeppelin.Persistence.Config.Size	Configuration Persistence Size	10G
Zeppelin.Persistence.Config.AccessMode	Configuration Persistence AccessMode	ReadWriteOnce
Zeppelin.Persistence.Notebook.Enabled	Enable Persistence for notebook	false
Zeppelin.Persistence.Notebook.StorageClass	Volume storageClassName	- (no dynamic provisioning)
Zeppelin.Persistence.Notebook.Size	Notebook Persistence Size	10G
Zeppelin.Persistence.Notebook.AccessMode	Notebook Persistence AccessMode	ReadWriteOnce

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install --name my-release -f values.yaml stable/spark

Tip

: You can use the default values.yaml