Dask Kubernetes

Dask Kubernetes

PyPI Conda Forge Python Support Kubernetes Support

Welcome to the documentation for dask-kubernetes.

Note

If you are looking for general documentation on deploying Dask on Kubernetes new users should head to the Dask documentation page on Kubernetes.

The package dask-kubernetes provides cluster managers for Kubernetes.

KubeCluster

KubeCluster deploys Dask clusters on Kubernetes clusters using custom Kubernetes resources. It is designed to dynamically launch ad-hoc deployments.

$ # Install operator CRDs and controller, needs to be done once on your Kubernetes cluster
$ helm repo add dask https://helm.dask.org && helm repo update
$ kubectl create ns dask-operator
$ helm install --namespace dask-operator dask-operator dask/dask-kubernetes-operator
from dask_kubernetes.operator import KubeCluster
cluster = KubeCluster(name="my_dask_cluster", image='ghcr.io/dask/dask:latest')
cluster.scale(10)

HelmCluster

HelmCluster is for managing an existing Dask cluster which has been deployed using Helm. You must have already installed the Dask Helm chart and have the cluster running. You can then use it to manage scaling and retrieve logs.

from dask_kubernetes import HelmCluster

cluster = HelmCluster(release_name="myrelease")
cluster.scale(10)