A Flyte SDK (v2) version of this plugin is available as flyteplugins-polars.
Polars
flytekitplugins-polars
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.
pip install flytekitplugins-polarsQuick Start(example, may need adjustment)
pip install flytekitplugins-polars
from flytekit import task, workflow
from flytekitplugins.polars import ParquetToPolarsDataFrameDecodingHandler, PolarsDataFrameToParquetEncodingHandler
config = ParquetToPolarsDataFrameDecodingHandler(...)
@task
def my_task() -> None:
...
@workflow
def my_workflow() -> None:
my_task()Available Imports (2)
Configuration type for Polars.
from flytekitplugins.polars import ParquetToPolarsDataFrameDecodingHandler
Configuration type for Polars.
from flytekitplugins.polars import PolarsDataFrameToParquetEncodingHandler
Dependencies
Related Plugins
polars
This plugin provides native support for Polars DataFrames and LazyFrames in Flyte, enabling efficient data processing with Polars' high-performance DataFrame library.
Modin
Modin is a pandas-accelerator that helps handle large datasets. It is a light-weight extension that is similar to the pandas API. It uses the concept of parallelism to reduce overhead, and improve the performance of pandas operations by leveraging the compute resources available.
Vaex
Vaex is a high-performance Python library for lazy out-of-core DataFrames
Async FSSpec
The Flyte async fsspec plugin is a powerful addition to the Flyte ecosystem designed to optimize the performance of object transmission. This plugin focuses on overriding key methods of the file systems in fsspec to introduce efficiency improvements, resulting in accelerated data transfers between Flyte workflows and object storage.
