Databricks

flyteplugins-databricks

Flyte SDK (v2)Cloud & Infrastructuredatabrickssparkcloudconnector

This plugin provides Databricks integration for Flyte, enabling you to run Spark jobs on Databricks as Flyte tasks.

Install

pip install flyteplugins-databricks

Quick Start(example, may need adjustment)

pip install flyteplugins-databricks

from flytekit import task, workflow
from flyteplugins.databricks import Databricks, DatabricksConnector

config = Databricks(...)

@task
def my_task() -> None:
    ...

@workflow
def my_workflow() -> None:
    my_task()

Available Imports (2)

configDatabricks

Configuration for a Databricks task.

extends dataclass — configuration or data structure for plugin setup

from flyteplugins.databricks import Databricks

connectorDatabricksConnector

Backend connector for Databricks.

from flyteplugins.databricks import DatabricksConnector

Related Plugins

AWS Batch

Cloud & Infrastructure

Flyte backend can be connected with AWS batch. Once enabled, it allows you to run flyte task on AWS batch service

Memory Machine Cloud

Cloud & Infrastructure

Flyte Connector plugin to allow executing Flyte tasks using MemVerge Memory Machine Cloud.

AWS Athena

Cloud & Infrastructure

Flyte backend can be connected with Athena. Once enabled, it allows you to query AWS Athena service (Presto + ANSI SQL Support) and retrieve typed schema (optionally).