Great Expectations

flytekitplugins-great_expectations

FlytekitData Validationgreat-expectationsdata-qualityvalidation

Great Expectations helps enforce data quality. The plugin supports the usage of Great Expectations as task and type.

Install

pip install flytekitplugins-great_expectations

Quick Start(example, may need adjustment)

pip install flytekitplugins-great_expectations

from flytekit import task, workflow
from flytekitplugins.great_expectations import GreatExpectationsFlyteConfig, GreatExpectationsType, BatchRequestConfig, GreatExpectationsTask

@task(task_config=GreatExpectationsTask(...))
def my_task() -> None:
    ...

@workflow
def my_workflow() -> None:
    my_task()

Available Imports (4)

configGreatExpectationsFlyteConfig

Use this configuration to configure GreatExpectations Plugin.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.great_expectations import GreatExpectationsFlyteConfig

typeGreatExpectationsType

Use this class to send the GreatExpectationsFlyteConfig.

from flytekitplugins.great_expectations import GreatExpectationsType

configBatchRequestConfig

Use this configuration to configure Batch Request.

extends dataclass — configuration or data structure for plugin setup

from flytekitplugins.great_expectations import BatchRequestConfig

taskGreatExpectationsTask

This task can be used to validate your data.

from flytekitplugins.great_expectations import GreatExpectationsTask

Dependencies

great-expectationssqlalchemypysparks3fs

Related Plugins

Pandera

Data Validation

Flytekit python natively supports many data types, including a FlyteSchema type for type-annotating pandas DataFrames. The Flytekit Pandera plugin provides an alternative for defining DataFrame schemas by integrating with Pandera, a runtime data validation tool for pandas DataFrames.