Matthew Rocklin
3ac741bdfd
* Add Dask Chart
This supercedes the previous dask-distributed chart, which will be
deprecated.
This also uses an accurate docker image for the Jupyter server,
and pins docker images generally.
* Comment out resource limits
* Set chart version to 1.0.0, appVersion to 0.17.0
* Update README and chart information
* specify nthreads and memory-limit explicitly in values.yaml
* update notes
* avoid setting --nthreads and --memory-limit if no resource limits
* expand truncation to 63 characters, and trim "."
* bump app version number
* remove change to dask-distributed chart
* remove NOTES.txt
* use explicit repository and image tags
* respond to feedback
* remove specification of containerPorts
* update README
* resolve linting issues in values.yaml
* Clarify extra packages in README
* replace pip/conda/apt packages with generic env solution
* Add option to turn off Jupyter
Also, combine all jupyter documents into one file
* add missing {{ end }}
* split dask-jupyter to multiple files
* Fix Yaml issues
* Fix values.yaml
* Update README with instructions for installing custom packages
* Remove configurable components
Dask Helm Chart
Dask allows distributed computation in Python.
Chart Details
This chart will deploy the following:
- 1 x Dask scheduler with port 8786 (scheduler) and 80 (Web UI) exposed on an external LoadBalancer
- 3 x Dask workers that connect to the scheduler
- 1 x Jupyter notebook (optional) with port 80 exposed on an external LoadBalancer
- All using Kubernetes Deployments
Installing the Chart
To install the chart with the release name my-release:
$ helm install --name my-release stable/dask
Configuration
The following tables lists the configurable parameters of the Dask chart and their default values.
Dask scheduler
| Parameter | Description | Default |
|---|---|---|
scheduler.name |
Dask scheduler name | scheduler |
scheduler.image |
Container image name | daskdev/dask |
scheduler.imageTag |
Container image tag | latest |
scheduler.replicas |
k8s deployment replicas | 1 |
scheduler.resources |
Container resources | {} |
Dask webUI
| Parameter | Description | Default |
|---|---|---|
webUI.name |
Dask webui name | webui |
webUI.servicePort |
k8s service port | 80 |
Dask worker
| Parameter | Description | Default |
|---|---|---|
worker.name |
Dask worker name | worker |
worker.image |
Container image name | daskdev/dask |
worker.imageTag |
Container image tag | 0.17.1 |
worker.replicas |
k8s hpa and deployment replicas | 3 |
worker.resources |
Container resources | {} |
jupyter
| Parameter | Description | Default |
|---|---|---|
jupyter.name |
jupyter name | jupyter |
jupyter.enabled |
Include optional Jupyter server | true |
jupyter.image |
Container image name | daskdev/dask-notebook |
jupyter.imageTag |
Container image tag | 0.17.1 |
jupyter.replicas |
k8s deployment replicas | 1 |
jupyter.servicePort |
k8s service port | 80 |
jupyter.resources |
Container resources | {} |
Specify each parameter using the --set key=value[,key=value] argument to helm install.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
$ helm install --name my-release -f values.yaml stable/dask
Tip
: You can use the default values.yaml
Customizing Python Environment
The default daskdev/dask images have a standard Miniconda installation along
with some common packages like NumPy and Pandas. You can install custom packages
with either Conda or Pip using optional environment variables. This happens
when your container starts up. Consider the following config.yaml file as an
example:
jupyter:
env:
- EXTRA_PIP_PACKAGES: s3fs git+https://github.com/user/repo.git --upgrade
- EXTRA_CONDA_PACKAGES: scipy matplotlib -c conda-forge
worker:
env:
- EXTRA_PIP_PACKAGES: s3fs git+https://github.com/user/repo.git --upgrade
- EXTRA_CONDA_PACKAGES: scipy -c conda-forge
Note that the Jupyter and Dask worker environments should have matching software environments, at least where a user is likely to distribute that functionality.