This is the Flyte SDK (v2) version of this plugin. The Flytekit version is available as flytekitplugins-dask.
Dask
flyteplugins-dask
Flyte can execute dask jobs natively on a Kubernetes Cluster, which manages the virtual dask cluster's lifecycle
pip install flyteplugins-daskQuick Start(example, may need adjustment)
See full examplespip install flyteplugins-dask
from flytekit import task, workflow
from flyteplugins.dask import Dask, Scheduler, WorkerGroup
config = Dask(...)
@task
def my_task() -> None:
...
@workflow
def my_workflow() -> None:
my_task()Available Imports (3)
Configuration for the dask task.
extends dataclass — configuration or data structure for plugin setup
from flyteplugins.dask import Dask
Configuration for the scheduler pod defaults to None.
extends dataclass — configuration or data structure for plugin setup
from flyteplugins.dask import Scheduler
Configuration for a group of dask worker pods defaults to None.
extends dataclass — configuration or data structure for plugin setup
from flyteplugins.dask import WorkerGroup
Related Plugins
Dask
Flyte can execute dask jobs natively on a Kubernetes Cluster, which manages the virtual dask cluster's lifecycle
Ray
Union can execute Ray jobs natively on a Kubernetes Cluster,
Kubeflow MPI
This plugin uses the Kubeflow MPI Operator and provides an extremely simplified interface for executing distributed training.
pytorch
Union can execute PyTorch distributed training jobs natively on a Kubernetes Cluster, which manages the lifecycle of worker pods, rendezvous coordination, spin-up, and tear down. It leverages the open-sourced TorchElastic (torch.distributed.elastic) launcher and the Kubeflow PyTorch Operator, enabling fault-tolerant and elastic training across multiple nodes.
