Slurm

flytekitplugins-slurm

FlytekitCloud & Infrastructureslurmhpcclusterbatch

The Slurm connector is designed to integrate Flyte workflows with Slurm-managed high-performance computing (HPC) clusters, enabling users to leverage Slurm's capability of compute resource allocation, scheduling, and monitoring.

Install

pip install flytekitplugins-slurm

Quick Start(example, may need adjustment)

See full examples

pip install flytekitplugins-slurm

from flytekit import task, workflow
from flytekitplugins.slurm import SlurmFunctionConnector, SlurmFunctionConfig, SlurmFunctionTask, SlurmScriptConnector

@task(task_config=SlurmFunctionTask(...))
def my_task() -> None:
    ...

@workflow
def my_workflow() -> None:
    my_task()

Available Imports (8)

connectorSlurmFunctionConnector

Backend connector for Slurm.

from flytekitplugins.slurm import SlurmFunctionConnector

configSlurmFunctionConfig

Configure Slurm settings.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.slurm import SlurmFunctionConfig

taskSlurmFunctionTask

Actual Plugin that transforms the local python code for execution within a slurm context...

extends PythonFunctionTask — a flyte task that can be used in workflows

from flytekitplugins.slurm import SlurmFunctionTask

connectorSlurmScriptConnector

Backend connector for Slurm.

from flytekitplugins.slurm import SlurmScriptConnector

configSlurmConfig

Configure Slurm settings.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.slurm import SlurmConfig

configSlurmScriptConfig

Encounter collision if Slurm is shared btw SlurmTask and SlurmShellTask.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.slurm import SlurmScriptConfig

taskSlurmShellTask

Task for Slurm.

extends PythonTask — a flyte task that can be used in workflows

from flytekitplugins.slurm import SlurmShellTask

taskSlurmTask

Task for Slurm.

extends PythonTask — a flyte task that can be used in workflows

from flytekitplugins.slurm import SlurmTask

Dependencies

asyncssh

Related Plugins

AWS Batch

Cloud & Infrastructure

Flyte backend can be connected with AWS batch. Once enabled, it allows you to run flyte task on AWS batch service

AWS Athena

Cloud & Infrastructure

Flyte backend can be connected with Athena. Once enabled, it allows you to query AWS Athena service (Presto + ANSI SQL Support) and retrieve typed schema (optionally).