diff --git a/stable/spark-history-server/Chart.yaml b/stable/spark-history-server/Chart.yaml index baf961d987..3b7f30f532 100644 --- a/stable/spark-history-server/Chart.yaml +++ b/stable/spark-history-server/Chart.yaml @@ -1,5 +1,5 @@ name: spark-history-server -version: 0.4.1 +version: 0.5.0 appVersion: 2.4.0 description: A Helm chart for Spark History Server home: https://spark.apache.org diff --git a/stable/spark-history-server/README.md b/stable/spark-history-server/README.md index 731d2f2893..c453813133 100644 --- a/stable/spark-history-server/README.md +++ b/stable/spark-history-server/README.md @@ -1,6 +1,6 @@ # Helm Chart for Spark History Server -[Spark History Server](https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact) provides a web UI for completed and running Spark applications. The supported storage backends are HDFS, Google Cloud Storage (GCS) and PersistentVolumeClaim (PVC). This chart is adapted from the [chart](https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs) from SnappyData Inc. +[Spark History Server](https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact) provides a web UI for completed and running Spark applications. The supported storage backends are HDFS, Google Cloud Storage (GCS), Azure Blob Storage (WASBS) and PersistentVolumeClaim (PVC). This chart is adapted from the [chart](https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs) from SnappyData Inc. #### Prerequisites @@ -12,7 +12,7 @@ $ kubectl -n create configmap hdfs-site --from-file=hdfs-site.xml && kubectl -n history-server-namespace create configmap core-site --from-file=core-site.xml ``` -* Secret (Only if using GCS or S3 without IAM based authentication) +* Secret (Only if using GCS, WASBS or S3 without IAM based authentication) If using GCS as storage, follow the preparatory steps below: @@ -35,6 +35,25 @@ ``` Then install the chart to enable the history server pod to read from the GCS bucket. + + Similarly, if using WASBS as storage, follow the preparatory steps below: + ```bash + # https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-cli + # create storage account and blob container + # prepare wasbs.storageAccount and wasbs.container values to set + # create blob for Spark event logs (default is wasbs:///spark-hs) + $ echo "your-storage-account-name" >> azure-storage-account-name + $ echo "your-container-name" >> azure-blob-container-name + # to auth with sas token (if wasbs.sasKeyMode=true - default) + $ echo "your-azure-blob-sas-key" >> azure-blob-sas-key + # to auth with storage account key + $ echo "your-azure-storage-account-key" >> azure-storage-account-key + ``` + + Then create a secret: + ```bash + $ kubectl create secret generic azure-secrets --from-file=azure-storage-account-name --from-file=azure-blob-container-name --from-file=azure-blob-sas-key --from-file=azure-storage-account-key + ``` Similarly, if using S3 as storage, follow the preparatory steps below: ```bash @@ -112,6 +131,14 @@ Note that the default image `lightbend/spark-history-server` is built using this | s3.accessKeyName | The file name that contains the AWS access key ID. Omit if using IAM based authentication | aws-access-key | | s3.secretKeyName | The file name that contains the AWS secret access key. Omit if using IAM based authentication | aws-secret-key | | s3.logDirectory | The S3 log directory that starts with "s3a://" | s3a://spark-hs/ | +| wasbs.enableWASBS | Whether to use WASBS storage | false | +| wasbs.sasKeyMode | Whether to use SAS token based authentication or fall back to using Azure Storage Account access key secret | true | +| wasbs.secret | Pre-mounted secret name for WASBS connection | azure-secrets | +| wasbs.sasKeyName | The file name that contains the Azure Storage access SAS token | azure-blob-sas-key | +| wasbs.storageAccountKeyName | The file name that contains the Azure Storage Account access key | azure-storage-account-key | +| wasbs.storageAccountNameKeyName | The file name that contains the Azure Storage account name | azure-storage-account-name | +| wasbs.containerKeyName | The file name that contains the Azure Storage container name | azure-blob-container-name | +| wasbs.logDirectory | The WASBS log directory that starts with "wasbs:///" | wasbs:///spark-hs | | ingress.enabled | if `true`, an ingress is created | false | | ingress.annotations | annotations for the ingress | {} | | ingress.path | if `true`, an ingress is created | / | @@ -140,6 +167,16 @@ Similarly, only when `s3.enableS3` is `true`, the following settings take effect * s3.secretKeyName * s3.logDirectory +Similarly, only when `wasbs.enableWASBS` is `true`, the following settings take effect: +* wasbs.enableWASBS +* wasbs.sasKeyMode +* wasbs.secret +* wasbs.sasKeyName +* wasbs.storageAccountKeyName +* wasbs.storageAccountNameKeyName +* wasbs.containerKeyName +* wasbs.logDirectory + And only when `pvc.enablePVC` and `gcs.enableGCS` are both `false`, is HDFS used, in which case the settings below are in effect: * hdfs.logDirectory @@ -232,3 +269,21 @@ When not using the IAM based authentication, you need to provide additional conf --conf spark.hadoop.fs.s3a.access.key=your-AWS-access-key-ID \ --conf spark.hadoop.fs.s3a.secret.key=your-AWS-secret-access-key ``` + +##### WASBS + +To write event logs to WASBS, you need to provide configs as below: +``` +--conf spark.eventLog.enabled=true \ +--conf spark.eventLog.dir=wasbs://$container@$storageAccount.blob.core.windows.net/spark-hs +``` +Similar to GCS, note that the image for your Spark job scheme needs to have the necessary dependencies: `hadoop-azure-2.7.3.jar` and `azure-storage-2.0.0.jar` + +You need to provide additional configs for authentication as below: +```bash +# For SAS token base auth +--conf spark.hadoop.fs.azure.local.sas.key.mode=true +--conf spark.hadoop.fs.azure.sas.$container.$storageAccount.blob.core.windows.net=your-SAS-token \ +# For Storage Account key based auth +--conf spark.hadoop.fs.azure.account.key.$storageAccount.blob.core.windows.net=your-storage-account-key +``` \ No newline at end of file diff --git a/stable/spark-history-server/templates/configmap.yaml b/stable/spark-history-server/templates/configmap.yaml index d52b5ad67c..2b992f57f1 100644 --- a/stable/spark-history-server/templates/configmap.yaml +++ b/stable/spark-history-server/templates/configmap.yaml @@ -11,6 +11,7 @@ data: enablePVC: {{ .Values.pvc.enablePVC | quote }} enableGCS: {{ .Values.gcs.enableGCS | quote }} enableS3: {{ .Values.s3.enableS3 | quote }} + enableWASBS: {{ .Values.wasbs.enableWASBS | quote }} {{- range $key, $val := .Values.environment }} {{ $key }}: {{ $val | quote }} {{- end }} @@ -26,6 +27,10 @@ data: {{- range $key, $val := .Values.s3 }} {{ $key }}: {{ $val | quote }} {{- end }} + {{- else if .Values.wasbs.enableWASBS }} + {{- range $key, $val := .Values.wasbs }} + {{ $key }}: {{ $val | quote }} + {{- end }} {{- else }} {{- range $key, $val := .Values.hdfs }} {{ $key }}: {{ $val | quote }} diff --git a/stable/spark-history-server/templates/deployment.yaml b/stable/spark-history-server/templates/deployment.yaml index 9b055ab596..64a42e6f12 100644 --- a/stable/spark-history-server/templates/deployment.yaml +++ b/stable/spark-history-server/templates/deployment.yaml @@ -35,7 +35,7 @@ spec: value: "true" ports: - name: historyport - containerPort: {{ .Values.service.port }} + containerPort: 18080 protocol: TCP resources: {{ toYaml .Values.resources | indent 10 }} @@ -52,13 +52,30 @@ spec: -Dspark.history.fs.logDirectory=$logDirectory"; elif [ "$enableS3" == "true" ]; then export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ - -Dspark.history.fs.logDirectory=$logDirectory + -Dspark.history.fs.logDirectory=$logDirectory \ -Dspark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"; if [ "$enableIAM" == "false" ]; then export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ -Dspark.hadoop.fs.s3a.access.key=$(cat /etc/secrets/${accessKeyName}) \ -Dspark.hadoop.fs.s3a.secret.key=$(cat /etc/secrets/${secretKeyName})"; fi; + elif [ "$enableWASBS" == "true" ]; then + container=$(cat /etc/secrets/${containerKeyName}) + storageAccount=$(cat /etc/secrets/${storageAccountNameKeyName}) + + export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ + -Dspark.history.fs.logDirectory=$logDirectory \ + -Dspark.hadoop.fs.defaultFS=wasbs://$container@$storageAccount.blob.core.windows.net \ + -Dspark.hadoop.fs.wasbs.impl=org.apache.hadoop.fs.azure.NativeAzureFileSystem \ + -Dspark.hadoop.fs.AbstractFileSystem.wasbs.impl=org.apache.hadoop.fs.azure.Wasbs"; + if [ "$sasKeyMode" == "true" ]; then + export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ + -Dspark.hadoop.fs.azure.local.sas.key.mode=true \ + -Dspark.hadoop.fs.azure.sas.$container.$storageAccount.blob.core.windows.net=$(cat /etc/secrets/${sasKeyName})"; + else + export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ + -Dspark.hadoop.fs.azure.account.key.$storageAccount.blob.core.windows.net=$(cat /etc/secrets/${storageAccountKeyName})"; + fi; else export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \ -Dspark.history.fs.logDirectory=$logDirectory"; @@ -89,6 +106,10 @@ spec: - name: secrets-volume mountPath: /etc/secrets {{- end }} + {{- else if .Values.wasbs.enableWASBS }} + volumeMounts: + - name: secrets-volume + mountPath: /etc/secrets {{- else }} volumeMounts: - name: core-site @@ -115,6 +136,11 @@ spec: secret: secretName: {{ .Values.s3.secret }} {{- end }} + {{- else if .Values.wasbs.enableWASBS }} + volumes: + - name: secrets-volume + secret: + secretName: {{ .Values.wasbs.secret }} {{- else }} volumes: - name: hdfs-site @@ -124,3 +150,19 @@ spec: configMap: name: {{ .Values.hdfs.coreSiteConfigMap }} {{- end }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/stable/spark-history-server/templates/ingress.yaml b/stable/spark-history-server/templates/ingress.yaml index 042b507583..7c0e966b02 100644 --- a/stable/spark-history-server/templates/ingress.yaml +++ b/stable/spark-history-server/templates/ingress.yaml @@ -19,13 +19,19 @@ spec: - host: {{ $host }} http: paths: - - path: / + - path: {{ $.Values.ingress.path }} backend: serviceName: {{ $fullname }} servicePort: historyport {{- end -}} {{- if .Values.ingress.tls }} tls: - {{ toYaml .Values.ingress.tls | indent 4 }} + {{- range .Values.ingress.tls }} + - hosts: + {{- range .hosts }} + - {{ . | quote }} + {{- end }} + secretName: {{ .secretName }} + {{- end }} {{- end -}} {{- end -}} diff --git a/stable/spark-history-server/values.yaml b/stable/spark-history-server/values.yaml index c603b0ca34..4f368029ec 100644 --- a/stable/spark-history-server/values.yaml +++ b/stable/spark-history-server/values.yaml @@ -95,3 +95,21 @@ s3: # secretKey is AWS secret key. Omit for IAM role-based or provider-based authentication. secretKeyName: aws-secret-key logDirectory: s3a://spark-hs/ + +wasbs: + enableWASBS: false + sasKeyMode: true + secret: azure-secrets + sasKeyName: azure-blob-sas-key + storageAccountKeyName: azure-storage-account-key + storageAccountNameKeyName: azure-storage-account-name + containerKeyName: azure-blob-container-name + logDirectory: wasbs:///spark-hs + +imagePullSecrets: [] + +nodeSelector: {} + +tolerations: [] + +affinity: {}