A Flyte SDK (v2) version of this plugin is available as flyteplugins-dask.
Dask
flytekitplugins-dask
Flyte can execute dask jobs natively on a Kubernetes Cluster, which manages the virtual dask cluster's lifecycle
pip install flytekitplugins-daskQuick Start(example, may need adjustment)
See full examplespip install flytekitplugins-dask
from flytekit import task, workflow
from flytekitplugins.dask import Dask, Scheduler, WorkerGroup
config = Dask(...)
@task
def my_task() -> None:
...
@workflow
def my_workflow() -> None:
my_task()Available Imports (3)
Configuration for the custom dask job to run.
extends dataclass — configuration or data structure for plugin setup
from flytekitplugins.dask import Dask
Configuration for the scheduler pod.
extends dataclass — configuration or data structure for plugin setup
from flytekitplugins.dask import Scheduler
Configuration for a dask worker group.
extends dataclass — configuration or data structure for plugin setup
from flytekitplugins.dask import WorkerGroup
Related Plugins
Dask
Flyte can execute dask jobs natively on a Kubernetes Cluster, which manages the virtual dask cluster's lifecycle
Ray
Flyte backend can be connected with Ray. Once enabled, it allows you to run flyte task on Ray cluster
Ray
Union can execute Ray jobs natively on a Kubernetes Cluster,
Kubeflow MPI
This plugin uses the Kubeflow MPI Operator and provides an extremely simplified interface for executing distributed training.
