Skip to main content
Version: 0.18.21

Collect OpenLineage metadata using an Action

OpenLineage is an open framework for collection and analysis of data lineage. It tracks the movement of data over time, tracing relationships between datasets. Data engineers can use data lineage metadata to determine the root cause of failures, identify performance bottlenecks, and simulate the effects of planned changes.

Enhancing the metadata in OpenLineage with results from an Expectation SuiteA collection of verifiable assertions about data. makes it possible to answer questions like:

  • have there been failed assertions in any upstream datasets?
  • what jobs are currently consuming data that is known to be of poor quality?
  • is there something in common among failed assertions that seem otherwise unrelated?

This guide will explain how to use an ActionA Python class with a run method that takes a Validation Result and does something with it to emit results to an OpenLineage backend, where their effect on related datasets can be studied.

Prerequisites​

Ensure that the openlineage-integration-common package has been installed in your Python environment.​

% pip3 install openlineage-integration-common

Update the action_list key in your Validation Operator config.​

Add the OpenLineageValidationAction action to the action_list key your Checkpoint configuration.

action_list:
- name: openlineage
  action:
    class_name: OpenLineageValidationAction
    module_name: openlineage.common.provider.great_expectations
    openlineage_host: ${OPENLINEAGE_URL}
    openlineage_apiKey: ${OPENLINEAGE_API_KEY}
    job_name: ge_validation # This is user-definable
    openlineage_namespace: ge_namespace # This is user-definable

The openlineage_host and openlineage_apiKey values can be set via the environment, as shown above, or can be implemented as variables in uncommitted/config_variables.yml. The openlineage_apiKey value is optional, and is not required by all OpenLineage backends.

A Great Expecations CheckpointThe primary means for validating data in a production deployment of Great Expectations. is recorded as a Job in OpenLineage, and will be named according to the job_name value. Similarly, the openlineage_namespace value can be optionally set. For more information on job naming, consult the Naming section of the OpenLineage spec.

Test your Action by Validating a Batch of data.​

Run the following command to retrieve and run a Checkpoint to Validate a BatchA selection of records from a Data Asset. of data and then emit lineage events to the OpenLineage backend:

Python
checkpoint = context.get_checkpoint("taxi_checkpoint")
checkpoint.run()
Reminder

Our guide on how to Validate data by running a Checkpoint has more detailed instructions for this step, including instructions on how to run a checkpoint from a Python script.