mirror of
https://github.com/helm/charts.git
synced 2026-08-23 22:37:45 +00:00
Spark history server/azure support (#12749)
* Added Azure WASBS support. Updated Dockerfile with Azure Storage dependencies: https://github.com/jahstreet/spark-history-server-docker/blob/feature/azure-wasbs-support/Dockerfile Signed-off-by: Aliaksandr Sasnouskikh <jahstreetlove@gmail.com> * Moved wasbs containerName and storageAccountName to secret. Updated README Signed-off-by: Aliaksandr Sasnouskikh <jahstreetlove@gmail.com> * Ingress Rules basePath support, tls fix. Chart version incremented. Signed-off-by: Aliaksandr Sasnouskikh <jahstreetlove@gmail.com> * Chart version updated. Signed-off-by: Aliaksandr Sasnouskikh <jahstreetlove@gmail.com>
This commit is contained in:
jahstreet
committed by
Kubernetes Prow Robot
Kubernetes Prow Robot
parent
8d31f92cd4
commit
096d011225
@@ -1,5 +1,5 @@
|
||||
name: spark-history-server
|
||||
version: 0.4.1
|
||||
version: 0.5.0
|
||||
appVersion: 2.4.0
|
||||
description: A Helm chart for Spark History Server
|
||||
home: https://spark.apache.org
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Helm Chart for Spark History Server
|
||||
|
||||
[Spark History Server](https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact) provides a web UI for completed and running Spark applications. The supported storage backends are HDFS, Google Cloud Storage (GCS) and PersistentVolumeClaim (PVC). This chart is adapted from the [chart](https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs) from SnappyData Inc.
|
||||
[Spark History Server](https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact) provides a web UI for completed and running Spark applications. The supported storage backends are HDFS, Google Cloud Storage (GCS), Azure Blob Storage (WASBS) and PersistentVolumeClaim (PVC). This chart is adapted from the [chart](https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs) from SnappyData Inc.
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
$ kubectl -n <history-server-namespace> create configmap hdfs-site --from-file=hdfs-site.xml && kubectl -n history-server-namespace create configmap core-site --from-file=core-site.xml
|
||||
```
|
||||
|
||||
* Secret (Only if using GCS or S3 without IAM based authentication)
|
||||
* Secret (Only if using GCS, WASBS or S3 without IAM based authentication)
|
||||
|
||||
If using GCS as storage, follow the preparatory steps below:
|
||||
|
||||
@@ -35,6 +35,25 @@
|
||||
```
|
||||
|
||||
Then install the chart to enable the history server pod to read from the GCS bucket.
|
||||
|
||||
Similarly, if using WASBS as storage, follow the preparatory steps below:
|
||||
```bash
|
||||
# https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-cli
|
||||
# create storage account and blob container
|
||||
# prepare wasbs.storageAccount and wasbs.container values to set
|
||||
# create blob for Spark event logs (default is wasbs:///spark-hs)
|
||||
$ echo "your-storage-account-name" >> azure-storage-account-name
|
||||
$ echo "your-container-name" >> azure-blob-container-name
|
||||
# to auth with sas token (if wasbs.sasKeyMode=true - default)
|
||||
$ echo "your-azure-blob-sas-key" >> azure-blob-sas-key
|
||||
# to auth with storage account key
|
||||
$ echo "your-azure-storage-account-key" >> azure-storage-account-key
|
||||
```
|
||||
|
||||
Then create a secret:
|
||||
```bash
|
||||
$ kubectl create secret generic azure-secrets --from-file=azure-storage-account-name --from-file=azure-blob-container-name --from-file=azure-blob-sas-key --from-file=azure-storage-account-key
|
||||
```
|
||||
|
||||
Similarly, if using S3 as storage, follow the preparatory steps below:
|
||||
```bash
|
||||
@@ -112,6 +131,14 @@ Note that the default image `lightbend/spark-history-server` is built using this
|
||||
| s3.accessKeyName | The file name that contains the AWS access key ID. Omit if using IAM based authentication | aws-access-key |
|
||||
| s3.secretKeyName | The file name that contains the AWS secret access key. Omit if using IAM based authentication | aws-secret-key |
|
||||
| s3.logDirectory | The S3 log directory that starts with "s3a://" | s3a://spark-hs/ |
|
||||
| wasbs.enableWASBS | Whether to use WASBS storage | false |
|
||||
| wasbs.sasKeyMode | Whether to use SAS token based authentication or fall back to using Azure Storage Account access key secret | true |
|
||||
| wasbs.secret | Pre-mounted secret name for WASBS connection | azure-secrets |
|
||||
| wasbs.sasKeyName | The file name that contains the Azure Storage access SAS token | azure-blob-sas-key |
|
||||
| wasbs.storageAccountKeyName | The file name that contains the Azure Storage Account access key | azure-storage-account-key |
|
||||
| wasbs.storageAccountNameKeyName | The file name that contains the Azure Storage account name | azure-storage-account-name |
|
||||
| wasbs.containerKeyName | The file name that contains the Azure Storage container name | azure-blob-container-name |
|
||||
| wasbs.logDirectory | The WASBS log directory that starts with "wasbs:///" | wasbs:///spark-hs |
|
||||
| ingress.enabled | if `true`, an ingress is created | false |
|
||||
| ingress.annotations | annotations for the ingress | {} |
|
||||
| ingress.path | if `true`, an ingress is created | / |
|
||||
@@ -140,6 +167,16 @@ Similarly, only when `s3.enableS3` is `true`, the following settings take effect
|
||||
* s3.secretKeyName
|
||||
* s3.logDirectory
|
||||
|
||||
Similarly, only when `wasbs.enableWASBS` is `true`, the following settings take effect:
|
||||
* wasbs.enableWASBS
|
||||
* wasbs.sasKeyMode
|
||||
* wasbs.secret
|
||||
* wasbs.sasKeyName
|
||||
* wasbs.storageAccountKeyName
|
||||
* wasbs.storageAccountNameKeyName
|
||||
* wasbs.containerKeyName
|
||||
* wasbs.logDirectory
|
||||
|
||||
And only when `pvc.enablePVC` and `gcs.enableGCS` are both `false`, is HDFS used, in which case the settings below are in effect:
|
||||
|
||||
* hdfs.logDirectory
|
||||
@@ -232,3 +269,21 @@ When not using the IAM based authentication, you need to provide additional conf
|
||||
--conf spark.hadoop.fs.s3a.access.key=your-AWS-access-key-ID \
|
||||
--conf spark.hadoop.fs.s3a.secret.key=your-AWS-secret-access-key
|
||||
```
|
||||
|
||||
##### WASBS
|
||||
|
||||
To write event logs to WASBS, you need to provide configs as below:
|
||||
```
|
||||
--conf spark.eventLog.enabled=true \
|
||||
--conf spark.eventLog.dir=wasbs://$container@$storageAccount.blob.core.windows.net/spark-hs
|
||||
```
|
||||
Similar to GCS, note that the image for your Spark job scheme needs to have the necessary dependencies: `hadoop-azure-2.7.3.jar` and `azure-storage-2.0.0.jar`
|
||||
|
||||
You need to provide additional configs for authentication as below:
|
||||
```bash
|
||||
# For SAS token base auth
|
||||
--conf spark.hadoop.fs.azure.local.sas.key.mode=true
|
||||
--conf spark.hadoop.fs.azure.sas.$container.$storageAccount.blob.core.windows.net=your-SAS-token \
|
||||
# For Storage Account key based auth
|
||||
--conf spark.hadoop.fs.azure.account.key.$storageAccount.blob.core.windows.net=your-storage-account-key
|
||||
```
|
||||
@@ -11,6 +11,7 @@ data:
|
||||
enablePVC: {{ .Values.pvc.enablePVC | quote }}
|
||||
enableGCS: {{ .Values.gcs.enableGCS | quote }}
|
||||
enableS3: {{ .Values.s3.enableS3 | quote }}
|
||||
enableWASBS: {{ .Values.wasbs.enableWASBS | quote }}
|
||||
{{- range $key, $val := .Values.environment }}
|
||||
{{ $key }}: {{ $val | quote }}
|
||||
{{- end }}
|
||||
@@ -26,6 +27,10 @@ data:
|
||||
{{- range $key, $val := .Values.s3 }}
|
||||
{{ $key }}: {{ $val | quote }}
|
||||
{{- end }}
|
||||
{{- else if .Values.wasbs.enableWASBS }}
|
||||
{{- range $key, $val := .Values.wasbs }}
|
||||
{{ $key }}: {{ $val | quote }}
|
||||
{{- end }}
|
||||
{{- else }}
|
||||
{{- range $key, $val := .Values.hdfs }}
|
||||
{{ $key }}: {{ $val | quote }}
|
||||
|
||||
@@ -35,7 +35,7 @@ spec:
|
||||
value: "true"
|
||||
ports:
|
||||
- name: historyport
|
||||
containerPort: {{ .Values.service.port }}
|
||||
containerPort: 18080
|
||||
protocol: TCP
|
||||
resources:
|
||||
{{ toYaml .Values.resources | indent 10 }}
|
||||
@@ -52,13 +52,30 @@ spec:
|
||||
-Dspark.history.fs.logDirectory=$logDirectory";
|
||||
elif [ "$enableS3" == "true" ]; then
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.history.fs.logDirectory=$logDirectory
|
||||
-Dspark.history.fs.logDirectory=$logDirectory \
|
||||
-Dspark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem";
|
||||
if [ "$enableIAM" == "false" ]; then
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.hadoop.fs.s3a.access.key=$(cat /etc/secrets/${accessKeyName}) \
|
||||
-Dspark.hadoop.fs.s3a.secret.key=$(cat /etc/secrets/${secretKeyName})";
|
||||
fi;
|
||||
elif [ "$enableWASBS" == "true" ]; then
|
||||
container=$(cat /etc/secrets/${containerKeyName})
|
||||
storageAccount=$(cat /etc/secrets/${storageAccountNameKeyName})
|
||||
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.history.fs.logDirectory=$logDirectory \
|
||||
-Dspark.hadoop.fs.defaultFS=wasbs://$container@$storageAccount.blob.core.windows.net \
|
||||
-Dspark.hadoop.fs.wasbs.impl=org.apache.hadoop.fs.azure.NativeAzureFileSystem \
|
||||
-Dspark.hadoop.fs.AbstractFileSystem.wasbs.impl=org.apache.hadoop.fs.azure.Wasbs";
|
||||
if [ "$sasKeyMode" == "true" ]; then
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.hadoop.fs.azure.local.sas.key.mode=true \
|
||||
-Dspark.hadoop.fs.azure.sas.$container.$storageAccount.blob.core.windows.net=$(cat /etc/secrets/${sasKeyName})";
|
||||
else
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.hadoop.fs.azure.account.key.$storageAccount.blob.core.windows.net=$(cat /etc/secrets/${storageAccountKeyName})";
|
||||
fi;
|
||||
else
|
||||
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
|
||||
-Dspark.history.fs.logDirectory=$logDirectory";
|
||||
@@ -89,6 +106,10 @@ spec:
|
||||
- name: secrets-volume
|
||||
mountPath: /etc/secrets
|
||||
{{- end }}
|
||||
{{- else if .Values.wasbs.enableWASBS }}
|
||||
volumeMounts:
|
||||
- name: secrets-volume
|
||||
mountPath: /etc/secrets
|
||||
{{- else }}
|
||||
volumeMounts:
|
||||
- name: core-site
|
||||
@@ -115,6 +136,11 @@ spec:
|
||||
secret:
|
||||
secretName: {{ .Values.s3.secret }}
|
||||
{{- end }}
|
||||
{{- else if .Values.wasbs.enableWASBS }}
|
||||
volumes:
|
||||
- name: secrets-volume
|
||||
secret:
|
||||
secretName: {{ .Values.wasbs.secret }}
|
||||
{{- else }}
|
||||
volumes:
|
||||
- name: hdfs-site
|
||||
@@ -124,3 +150,19 @@ spec:
|
||||
configMap:
|
||||
name: {{ .Values.hdfs.coreSiteConfigMap }}
|
||||
{{- end }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
|
||||
@@ -19,13 +19,19 @@ spec:
|
||||
- host: {{ $host }}
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
- path: {{ $.Values.ingress.path }}
|
||||
backend:
|
||||
serviceName: {{ $fullname }}
|
||||
servicePort: historyport
|
||||
{{- end -}}
|
||||
{{- if .Values.ingress.tls }}
|
||||
tls:
|
||||
{{ toYaml .Values.ingress.tls | indent 4 }}
|
||||
{{- range .Values.ingress.tls }}
|
||||
- hosts:
|
||||
{{- range .hosts }}
|
||||
- {{ . | quote }}
|
||||
{{- end }}
|
||||
secretName: {{ .secretName }}
|
||||
{{- end }}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
@@ -95,3 +95,21 @@ s3:
|
||||
# secretKey is AWS secret key. Omit for IAM role-based or provider-based authentication.
|
||||
secretKeyName: aws-secret-key
|
||||
logDirectory: s3a://spark-hs/
|
||||
|
||||
wasbs:
|
||||
enableWASBS: false
|
||||
sasKeyMode: true
|
||||
secret: azure-secrets
|
||||
sasKeyName: azure-blob-sas-key
|
||||
storageAccountKeyName: azure-storage-account-key
|
||||
storageAccountNameKeyName: azure-storage-account-name
|
||||
containerKeyName: azure-blob-container-name
|
||||
logDirectory: wasbs:///spark-hs
|
||||
|
||||
imagePullSecrets: []
|
||||
|
||||
nodeSelector: {}
|
||||
|
||||
tolerations: []
|
||||
|
||||
affinity: {}
|
||||
|
||||
Reference in New Issue
Block a user