Hugging Face
flytekitplugins-huggingface
Hugging Face is a community and data science platform that provides: Tools that enable users to build, train and deploy ML models based on open source (OS) code and technologies
pip install flytekitplugins-huggingfaceQuick Start(example, may need adjustment)
pip install flytekitplugins-huggingface
from flytekit import task, workflow
from flytekitplugins.huggingface import HuggingFaceDatasetToParquetEncodingHandler, ParquetToHuggingFaceDatasetDecodingHandler
config = HuggingFaceDatasetToParquetEncodingHandler(...)
@task
def my_task() -> None:
...
@workflow
def my_workflow() -> None:
my_task()Available Imports (2)
Configuration type for Hugging Face.
from flytekitplugins.huggingface import HuggingFaceDatasetToParquetEncodingHandler
Configuration type for Hugging Face.
from flytekitplugins.huggingface import ParquetToHuggingFaceDatasetDecodingHandler
Dependencies
Related Plugins
Async FSSpec
The Flyte async fsspec plugin is a powerful addition to the Flyte ecosystem designed to optimize the performance of object transmission. This plugin focuses on overriding key methods of the file systems in fsspec to introduce efficiency improvements, resulting in accelerated data transfers between Flyte workflows and object storage.
FSSpec
This plugin provides an implementation of the data persistence layer in Flytekit that uses fsspec. Once this plugin
Geopandas
GeoPandas GeoPandas is an open source project to make working with geospatial data in python easier.
Modin
Modin is a pandas-accelerator that helps handle large datasets. It is a light-weight extension that is similar to the pandas API. It uses the concept of parallelism to reduce overhead, and improve the performance of pandas operations by leveraging the compute resources available.
