Skip to main content

How to create and edit Expectations based on domain knowledge, without inspecting data directly

This guide shows how to create an Expectation Suite without a sample Batch.

Here are some reasons why you may wish to do this but please contact us if there are other use cases we have not yet considered:

  1. You don't have a sample.

  2. You don't currently have access to the data to make a sample.

  3. You know exactly how you want your Expectations to be configured.

  4. You want to create Expectations parametrically (you can also do this in interactive mode).

  5. You don't want to spend the time to validate against a sample.

Prerequisites: This how-to guide assumes you have:

Steps#

  1. Use the CLI to generate a helper notebook

    From the command line, run:

    great_expectations --v3-api suite new
  2. Create Expectation Configurations in the helper notebook

    You are adding Expectation configurations to the suite. Since there is no sample Batch of data, no validation happens during this process. To illustrate how to do this, consider a hypothetical example. Suppose that you have a table with the columns account_id, user_id, transaction_id, transaction_type, and transaction_amt_usd. Then the following code snipped adds an expectation that the columns of the actual table will appear in the order specified above:

    # Create an Expectationexpectation_configuration = ExpectationConfiguration(   # Name of expectation type being added   expectation_type="expect_table_columns_to_match_ordered_list",   # These are the arguments of the expectation   # The keys allowed in the dictionary are Parameters and   # Keyword Arguments of this Expectation Type   kwargs={      "column_list": [         "account_id", "user_id", "transaction_id", "transaction_type", "transaction_amt_usd"      ]   },   # This is how you can optionally add a comment about this expectation.   # It will be rendered in Data Docs.   # See this guide for details:   # `How to add comments to Expectations and display them in Data Docs`.   meta={      "notes": {         "format": "markdown",         "content": "Some clever comment about this expectation. **Markdown** `Supported`"      }   })# Add the Expectation to the suitesuite.add_expectation(expectation_configuration=expectation_configuration)

    Here are a few more example expectations for this dataset:

    expectation_configuration = ExpectationConfiguration(   expectation_type="expect_column_values_to_be_in_set",   kwargs={      "column": "transaction_type",      "value_set": ["purchase", "refund", "upgrade"]   },   # Note optional comments omitted)suite.add_expectation(expectation_configuration=expectation_configuration)
    expectation_configuration = ExpectationConfiguration(   expectation_type="expect_column_values_to_not_be_null",   kwargs={      "column": "account_id",      "mostly": 1.0,   },   meta={      "notes": {         "format": "markdown",         "content": "Some clever comment about this expectation. **Markdown** `Supported`"      }   })suite.add_expectation(expectation_configuration=expectation_configuration)
    expectation_configuration = ExpectationConfiguration(   expectation_type="expect_column_values_to_not_be_null",   kwargs={      "column": "user_id",      "mostly": 0.75,   },   meta={      "notes": {         "format": "markdown",         "content": "Some clever comment about this expectation. **Markdown** `Supported`"      }   })suite.add_expectation(expectation_configuration=expectation_configuration)

    You can see all the available Expectations in the :ref:expectation_glossary.

  3. Save your Expectation Suite

    Run the final cell in the helper notebook to save your Expectation Suite.

    This will create a JSON file with your Expectation Suite in the Store you have configured, which you can then load and use for Validation.