Papermill
flytekitplugins-papermill
It is possible to run a Jupyter notebook as a Flyte task using Papermill. Papermill executes the notebook as a whole, so before using this plugin, it is essential to construct your notebook as recommended by Papermill.
pip install flytekitplugins-papermillQuick Start(example, may need adjustment)
See full examplespip install flytekitplugins-papermill
from flytekit import task, workflow
from flytekitplugins.papermill import NotebookTask, load_flytedirectory, load_flytefile, load_structureddataset
@task(task_config=NotebookTask(...))
def my_task() -> None:
...
@workflow
def my_workflow() -> None:
my_task()Available Imports (5)
Simple Papermill based input output handling for a Python Jupyter notebook.
from flytekitplugins.papermill import NotebookTask
Task for Papermill.
from flytekitplugins.papermill import load_flytedirectory
Task for Papermill.
from flytekitplugins.papermill import load_flytefile
Task for Papermill.
from flytekitplugins.papermill import load_structureddataset
Use this method to record outputs from a notebook.
from flytekitplugins.papermill import record_outputs
Dependencies
Related Plugins
Apache Airflow
Airflow plugin allows you to seamlessly run Airflow tasks in the Flyte workflow without changing any code.
DBT
Flytekit plugin for performing DBT tasks. Currently it supports dbt run , dbt test, dbt source freshness tasks.
Flyte Interactive
FlyteInteractive plugin provides users' favorite interface to develop and debug a flyte task interactively. We support vscode, jupyter (WIP), and neovim (WIP).
