Kubernetes Stateful

flytekitplugins-k8sdataservice

FlytekitCloud & Infrastructurekubernetesk8sdata-service

This plugin provides support for Kubernetes StatefulSet and Service integration, enabling seamless provisioning and coordination with any Kubernetes services or Flyte tasks. It is especially suited for deep learning use cases at scale, where distributed and parallelized data loading and caching across nodes are required.

Install

pip install flytekitplugins-k8sdataservice

Quick Start(example, may need adjustment)

pip install flytekitplugins-k8sdataservice

from flytekit import task, workflow
from flytekitplugins.k8sdataservice import DataServiceConnector, CleanupSensor, DataServiceConfig, DataServiceTask

@task(task_config=DataServiceTask(...))
def my_task() -> None:
    ...

@workflow
def my_workflow() -> None:
    my_task()

Available Imports (4)

connectorDataServiceConnector

Backend connector for Kubernetes Stateful.

from flytekitplugins.k8sdataservice import DataServiceConnector

sensorCleanupSensor

Sensor for Kubernetes Stateful.

extends BaseSensor — a sensor that waits for external conditions

from flytekitplugins.k8sdataservice import CleanupSensor

configDataServiceConfig

DataServiceConfig should be used to configure a DataServiceTask.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.k8sdataservice import DataServiceConfig

taskDataServiceTask

Task for Kubernetes Stateful.

extends PythonTask — a flyte task that can be used in workflows

from flytekitplugins.k8sdataservice import DataServiceTask

Dependencies

kubernetes

Related Plugins

Kubernetes Pod

Cloud & Infrastructure

By default, Flyte tasks decorated with @task are essentially single functions that are loaded in one container. But often, there is a need to run a job with more than one container.

AWS Athena

Cloud & Infrastructure

Flyte backend can be connected with Athena. Once enabled, it allows you to query AWS Athena service (Presto + ANSI SQL Support) and retrieve typed schema (optionally).