Added Pachyderm chart (#3114)

* Added Pachyderm chart * Added appVersion field and updated instructions * Added namespace flag to deployment commands * Added OWNERS file * Updated OWNERS file and changed names with github ids
2026-08-23 22:37:45 +00:00 · 2018-01-04 05:12:19 -08:00
parent 4f45dbc891
commit 856f6b5064
14 changed files with 582 additions and 0 deletions
@@ -0,0 +1,21 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
@@ -0,0 +1,24 @@
+---
+apiVersion: v1
+description: >
+  Pachyderm is a large-scale container-based workflow engine that offers complete version control for data and reproducible data processing
+name: pachyderm
+icon: https://avatars2.githubusercontent.com/u/10432478?s=200
+keywords:
+  - pachyderm
+  - workflows
+  - reproducibility
+  - distributed
+  - processing
+version: 0.1.0
+appVersion: 1.6.6
+home: "https://pachyderm.io"
+sources:
+  - "https://github.com/pachyderm/pachyderm"
+maintainers:
+  - name: jonandernovella
+    email: novella.uu@gmail.com
+  - name: dwhitena
+    email: whitenack.daniel@gmail.com
+  - name: jdoliner
+    email: jdoliner@pachyderm.io
@@ -0,0 +1,8 @@
+approvers:
+- jonandernovella
+- dwhitena
+- jdoliner
+reviewers:
+- jonandernovella
+- dwhitena
+- jdoliner
@@ -0,0 +1,142 @@
+Pachyderm Helm Chart
+====================
+
+Pachyderm is a language-agnostic and cloud infrastructure-agnostic large-scale data processing framework based on software containers. This chart can be used to deploy Pachyderm backed by object stores of different Cloud providers.
+
+* https://pachyderm.io
+* https://github.com/pachyderm/pachyderm
+
+
+Prerequisites Details
+---------------------
+
+-	Dynamic provisioning of PVs (for non-local deployments)
+
+General chart settings
+----------------------
+
+The following table lists the configurable parameters of `pachd` and their default values:
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `pachd.image.repository` | Container image name  | `pachyderm/pachd` |
+| `*.image.tag`            | Container image tag   | `<latest version>`|
+| `*.image.pullPolicy`     | Image pull policy     | `Always`          |
+| `*.worker.repository`    | Worker image name     | `pachyderm/worker`|
+| `*.worker.tag`           | Worker image tag      | `<latest version>`|
+| `*.replicaCount`         | Number of pachds      | `1`               |
+| `*.resources.requests`   | Memory and cpu request| `{512M,250m}`     |
+
+
+Next table lists the configurable parameters of `etcd` and their default values:
+
+| Parameter                   | Description           | Default           |
+|-----------------------------|-----------------------|-------------------|
+| `etcd.image.repository`     | Container image name  | `pachyderm/etcd`  |
+| `*.image.tag`               | Container image tag   | `<latest version>`|
+| `*.image.pullPolicy`        | Image pull policy     | `IfNotPresent`    |
+| `*.resources.requests`      | Memory and cpu request| `{250M,250m}`     |
+| `*.persistence.enabled`     | Enable persistence    | `false`           |
+| `*.persistence.size`        | Storage request       | `20G`             |
+| `*.persistence.accessMode`  | Access mode for PV    | `ReadWriteOnce`   |
+| `*.persistence.storageClass`| PVC storage class     | `nil`             |
+
+
+Storage backend settings
+------------------------
+
+In order to set which object store credentials you want to use, please set the flag `credentials` with one of the following values: `local | s3 | google | amazon | microsoft`.
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `credentials`            | Backend credentials   | ""                |
+
+
+Based on the storage credentials used, fill in the corresponding parameters for your object store.
+
+-	The `local` installation will deploy Pachyderm on your local Kubernetes cluster (i.e: minikube) backed by your local storage unit. 
+
+-	With `S3 endpoint` credentials (such as Minio credentials), these are the configurable parameters:
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `s3.accessKey`           | S3 access key         | `""`              |
+| `s3.secretKey`           | S3 secret key         | `""`              |
+| `s3.bucketName`          | S3 bucket name        | `""`              |
+| `s3.endpoint`            | S3 endpoint           | `""`              |
+| `s3.secure`              | S3 secure             | `"0"`             |
+| `s3.signature`           | S3 signature          | `"0"`             |
+
+
+-	With `Google Cloud` credentials, you must define your `GCS bucket name`:
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `google.bucketName`      | GCS bucket name       | `""`              |
+
+
+-	On `Amazon Web Services`, please set the next values:
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `amazon.bucketName`      | Amazon bucket name    | `""`              |
+| `amazon.distribution`    | Amazon distribution   | `""`              |
+| `amazon.id`              | Amazon id             | `""`              |
+| `amazon.region`          | Amazon region         | `""`              |
+| `amazon.secret`          | Amazon secret         | `""`              |
+| `amazon.token`           | Amazon token          | `""`              |
+
+
+-	As for `Microsoft Azure`, you must specify the following parameters:
+
+| Parameter                | Description           | Default           |
+|--------------------------|-----------------------|-------------------|
+| `microsoft.container`    | Container             | `""`              |
+| `microsoft.id`           | Account name          | `""`              |
+| `microsoft.secret`       | Account key           | `""`              |
+
+
+How to install the chart
+------------------------
+
+We strongly suggest that the installation of Pachyderm should be performed in its own namespace. The default installation will deploy Pachyderm on your local Kubernetes cluster:
+
+```console
+$ helm install --namespace pachyderm --name my-release stable/pachyderm
+```
+
+You should install the chart specifying each parameter using the `--set key=value[,key=value]` argument to helm install. Please consult the `values.yaml` file for more information regarding the parameters. For example:
+
+
+```console
+$ helm install --namespace pachyderm --name my-release \
+--set credentials=s3,s3.accessKey=myaccesskey,s3.secretKey=mysecretkey,s3.bucketName=default_bucket,s3.endpoint=domain.subdomain:8080,etcd.persistence.enabled=true,etcd.persistence.accessMode=ReadWriteMany,"signature=\"1\"","secure=\"1\"" \
+stable/pachyderm
+```
+
+Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart:
+
+```console
+$ helm install --namespace pachyderm --name my-release -f values.yaml stable/pachyderm
+```
+
+Accessing the pachd service
+---------------------------
+
+In order to use Pachyderm, please login through ssh to the master node and install the Pachyderm client:
+
+```console
+$ curl -o /tmp/pachctl.deb -L https://github.com/pachyderm/pachyderm/releases/download/v1.6.6/pachctl_1.6.6_amd64.deb && sudo dpkg -i /tmp/pachctl.deb
+```
+
+Please note that the client version should correspond with the pachd service version. For more information please consult: http://pachyderm.readthedocs.io/en/latest/index.html. Also, if you have your kubernetes client properly configured to talk with your remote cluster, you can simply install `pachctl` on your local machine and execute: `pachctl -k '-n=<namespace>' port-forward &`.
+
+Clean-up
+-------
+
+In order to remove the Pachyderm release, you can execute the following commands:
+
+```console
+$ helm list
+$ helm delete --purge <release-name>
+```
@@ -0,0 +1,15 @@
+In order to use Pachyderm, you need to install the Pachyderm command line client: pachctl. Please note that the client version should correspond to the pachd service version. For more information please consult: http://pachyderm.readthedocs.io/en/latest/index.html.
+
+On Linux environments:
+
+$ curl -o /tmp/pachctl.deb -L https://github.com/pachyderm/pachyderm/releases/download/v{{ .Chart.AppVersion }}/pachctl_{{ .Chart.AppVersion }}_amd64.deb && sudo dpkg -i /tmp/pachctl.deb
+
+And on Mac:
+
+$ brew tap pachyderm/tap && brew install pachyderm/tap/pachctl@1.6
+
+To check that installation was successful, you can try running pachctl help, which should return a list of Pachyderm commands.  
+
+If you install the client on your master node, you should already be able to interact with the service. Also, if you have your kubernetes client properly configured to talk with your remote cluster, you can simply install `pachctl` on your local machine and execute: `pachctl -k ‘-n={{ .Release.Namespace }}’ port-forward &`.
+
+To check if port forwarding was successfully completed, try `pachctl version` to make sure everything is working.
@@ -0,0 +1,24 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+*/}}
+{{- define "fullname" -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+*/}}
+{{- define "etcd.fullname" -}}
+{{- printf "%s-%s" .Release.Name "etcd" | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
@@ -0,0 +1,55 @@
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: etcd
+  labels:
+    app: {{ template "etcd.fullname" . }}
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: {{ template "etcd.fullname" . }}
+      suite: pachyderm
+  template:
+    metadata:
+      name: etcd
+      labels:
+        app: {{ template "etcd.fullname" . }}
+        suite: pachyderm
+    spec:
+      volumes:
+        - name: etcdvol
+        {{- if .Values.etcd.persistence.enabled }}
+          persistentVolumeClaim:
+            claimName: {{ template "etcd.fullname" . }}-pvc
+        {{- else }}
+          hostPath:
+           path: "/var/pachyderm/etcd"
+        {{- end }}
+      containers:
+        - name: etcd
+          image: '{{ .Values.etcd.image.repository }}:{{ .Values.etcd.image.tag }}'
+          command:
+          - "/usr/local/bin/etcd"
+          - "--listen-client-urls=http://0.0.0.0:2379"
+          - "--advertise-client-urls=http://0.0.0.0:2379"
+          - "--data-dir=/var/data/etcd"
+          - "--auto-compaction-retention=1"
+          ports:
+            - name: client-port
+              containerPort: 2379
+            - name: peer-port
+              containerPort: 2380
+          resources:
+           requests:
+            cpu: '{{ .Values.etcd.resources.requests.cpu }}'
+            memory: {{ .Values.etcd.resources.requests.memory }}
+          volumeMounts:
+            - name: etcdvol
+              mountPath: "/var/data/etcd"
+          imagePullPolicy: {{ .Values.etcd.image.pullPolicy }}
@@ -0,0 +1,20 @@
+---
+{{- if .Values.etcd.persistence.enabled }}
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: {{ template "etcd.fullname" . }}-pvc
+spec:
+  accessModes:
+    - {{ .Values.etcd.persistence.accessMode | quote }}
+  resources:
+    requests:
+      storage: {{ .Values.etcd.persistence.size | quote }}
+{{- if .Values.etcd.persistence.storageClass }}
+{{- if (eq "-" .Values.etcd.persistence.storageClass) }}
+  storageClassName: ""
+{{- else }}
+  storageClassName: "{{ .Values.etcd.persistence.storageClass }}"
+{{- end }}
+{{- end }}
+{{- end }}
@@ -0,0 +1,19 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: etcd
+  labels:
+    app: {{ template "etcd.fullname" . }}
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
+spec:
+  ports:
+    - name: client-port
+      port: 2379
+      targetPort: 0
+  selector:
+    app: {{ template "etcd.fullname" . }}
+  type: NodePort
@@ -0,0 +1,102 @@
+---
+kind: Deployment
+apiVersion: extensions/v1beta1
+metadata:
+  name: pachd
+  labels:
+    app: pachd
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
+spec:
+  replicas: {{ .Values.pachd.replicaCount }}
+  selector:
+    matchLabels:
+      app: pachd
+      suite: pachyderm
+  template:
+    metadata:
+      name: pachd
+      labels:
+        app: pachd
+        suite: pachyderm
+    spec:
+      volumes:
+      - name: pachdvol
+        {{- if eq .Values.credentials "local" }}
+        hostPath:
+         path: "/var/pachyderm/pachd"
+        {{- end }}
+      - name: pachyderm-storage-secret
+        secret:
+          secretName: pachyderm-storage-secret
+      containers:
+      - name: pachd
+        image: '{{ .Values.pachd.image.repository }}:{{ .Values.pachd.image.tag }}'
+        ports:
+        - name: api-grpc-port
+          containerPort: 650
+          protocol: TCP
+        - name: trace-port
+          containerPort: 651
+        - name: api-http-port
+          containerPort: 652
+        env:
+        - name: PACH_ROOT
+          value: "/pach"
+        - name: NUM_SHARDS
+          value: '16'
+        - name: STORAGE_BACKEND
+          {{- if eq .Values.credentials "s3" }}
+          value: MINIO
+          {{- else if eq .Values.credentials "google" }}
+          value: GOOGLE
+          {{- else if eq .Values.credentials "amazon" }}
+          value: AMAZON
+          {{- else if eq .Values.credentials "microsoft" }}
+          value: MICROSOFT
+          {{- else if eq .Values.credentials "local" }}
+          value: LOCAL
+          {{- end }}
+        - name: STORAGE_HOST_PATH
+          {{- if eq .Values.credentials "local" }}
+          value: "/var/pachyderm/pachd"
+          {{- end }}
+        - name: PACHD_POD_NAMESPACE
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: metadata.namespace
+        - name: WORKER_IMAGE
+          value: '{{ .Values.pachd.worker.repository }}:{{ .Values.pachd.worker.tag }}'
+        - name: WORKER_SIDECAR_IMAGE
+          value: '{{ .Values.pachd.image.repository }}:{{ .Values.pachd.image.tag }}'
+        - name: WORKER_IMAGE_PULL_POLICY
+          value: IfNotPresent
+        - name: PACHD_VERSION
+          value: {{ .Values.pachd.image.tag }}
+        - name: METRICS
+          value: 'true'
+        - name: LOG_LEVEL
+          value: info
+        - name: BLOCK_CACHE_BYTES
+          value: 0G
+        - name: IAM_ROLE
+        - name: PACHYDERM_AUTHENTICATION_DISABLED_FOR_TESTING
+          value: 'false'
+        resources:
+          requests:
+            cpu: '{{ .Values.pachd.resources.requests.cpu }}'
+            memory: {{ .Values.pachd.resources.requests.memory }}
+        volumeMounts:
+        - name: pachdvol
+          mountPath: "/pach"
+        - name: pachyderm-storage-secret
+          mountPath: "/pachyderm-storage-secret"
+        imagePullPolicy: {{ .Values.pachd.image.pullPolicy }}
+        securityContext:
+          privileged: true
+      serviceAccountName: {{ template "fullname" . }}-sa
+  strategy: {}
+status: {}
@@ -0,0 +1,11 @@
+---
+kind: ServiceAccount
+apiVersion: v1
+metadata:
+  name: {{ template "fullname" . }}-sa
+  labels:
+    app: {{ template "fullname" . }}
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
@@ -0,0 +1,33 @@
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pachyderm-storage-secret
+  labels:
+    app: {{ template "fullname" . }}
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
+data:
+  {{- if eq .Values.credentials "s3" }}
+  minio-id: {{ .Values.s3.accessKey | b64enc | quote }}
+  minio-secret: {{ .Values.s3.secretKey | b64enc | quote  }}
+  minio-bucket: {{ .Values.s3.bucketName | b64enc | quote }}
+  minio-endpoint: {{ .Values.s3.endpoint | b64enc | quote }}
+  minio-secure: {{ .Values.s3.secure | b64enc | quote }}
+  minio-signature: {{ .Values.s3.signature | b64enc | quote }}
+  {{- else if eq .Values.credentials "google" }}
+  google-bucket: { .Values.google.bucketName | b64enc | quote }}
+  {{- else if eq .Values.credentials "amazon" }}
+  amazon-bucket: { .Values.amazon.bucketName | b64enc | quote }}
+  amazon-distribution: { .Values.amazon.distribution | b64enc | quote }}
+  amazon-id: { .Values.amazon.id | b64enc | quote }}
+  amazon-region: { .Values.amazon.region | b64enc | quote }}
+  amazon-secret: { .Values.amazon.secret | b64enc | quote }}
+  amazon-token: { .Values.amazon.token | b64enc | quote }}
+  {{- else if eq .Values.credentials "microsoft" }}
+  microsoft-container: { .Values.microsoft.container | b64enc | quote }}
+  microsoft-id: { .Values.microsoft.id | b64enc | quote }}
+  microsoft-secret: { .Values.microsoft.secret | b64enc | quote }}
+  {{- end }}
@@ -0,0 +1,28 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: pachd
+  labels:
+    app: pachd
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: "{{ .Release.Name }}"
+    heritage: "{{ .Release.Service }}"
+    suite: pachyderm
+spec:
+  ports:
+    - name: api-grpc-port
+      port: 650
+      targetPort: 650
+      nodePort: 30650
+    - name: trace-port
+      port: 651
+      targetPort: 651
+      nodePort: 30651
+    - name: api-http-port
+      port: 652
+      targetPort: 652
+      nodePort: 30652
+  selector:
+    app: pachd
+  type: NodePort
@@ -0,0 +1,80 @@
+---
+## Set default credentials for object store
+credentials: "local"
+
+## S3 endpoint (i.e: Minio server) credentials
+s3:
+ accessKey: ""
+ secretKey: ""
+ bucketName: ""
+ ## The endpoint should be something like: <mydomain>.<subdomain>:<port>
+ endpoint: ""
+ ## For 'true' set the value to 1
+ secure: "0"
+ ## For 'S3v2' signature, set the value to 1. For 'S3v4', set this value to 0
+ signature: "0"
+
+## Google Cloud credentials
+google:
+ bucketName: ""
+
+## Amazon Web Services credentials
+amazon:
+ bucketName: ""
+ ## The distribution parameter is often an empty string
+ distribution: ""
+ id: ""
+ region: ""
+ secret: ""
+ token: ""
+
+## Microsoft Azure credentials
+microsoft:
+ container: ""
+ id: ""
+ secret: ""
+
+## Set default image settings, resource requests and number of replicas of pachd
+pachd:
+ replicaCount: 1
+ ## For available images please check: https://hub.docker.com/r/pachyderm/pachd/tags/
+ image:
+  repository: pachyderm/pachd
+  tag: 1.6.6
+  pullPolicy: Always
+ worker:
+  repository: pachyderm/worker
+  tag: 1.6.6
+ resources:
+  ## For non-local deployments, 1 cpu and 2G of memory requests are recommended
+  requests:
+   cpu: 250m
+   memory: 512M
+
+## Set default image settings and persistence settings of etcd
+etcd:
+ ## For available images please check: https://hub.docker.com/r/pachyderm/etcd/tags
+ image:
+  repository: pachyderm/etcd
+  tag: v3.2.7
+  pullPolicy: IfNotPresent
+  ## Enable persistence using Persistent Volume Claims
+  ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
+ persistence:
+  enabled: false
+  ## etcd data Persistent Volume Storage Class
+  ## If defined, storageClassName: <storageClass>
+  ## If set to "-", storageClassName: "", which disables dynamic provisioning
+  ## If undefined (the default) or set to null, no storageClassName spec is
+  ## set, choosing the default provisioner. (gp2 on AWS, standard on
+  ## GKE, AWS & OpenStack)
+  # storageClass: "-"
+  ## Set default PVC size
+  size: 20G
+  ## Set default PVC access mode: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
+  accessMode: ReadWriteOnce
+ resources:
+  ## For non-local deployments, 1 cpu and 2G of memory requests are recommended
+  requests:
+   cpu: 250m
+   memory: 256M