Skip to main content
Version: 1.5.10

Manage Expectations

You can manually create Expecatations and use several different GX Cloud features to generate Expecations.

This page provides instructions for working with Expectations. To learn about Expectation types and options, see the Expectations overview.

Prerequisites​

Add an Expectation​

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Click New Expectation.

  4. Select a data quality issue to test for or an option for writing your own test.

    Options for accelerating test coverage

    If you selected the Schema, Volume, or Completeness data quality issue, you will have the Automatic option to generate Expectations for Anomaly Detection. If you instead want to create your own rules, click Manual.

    If you selected custom SQL, you will have the option to Generate SQL with ExpectAI (BETA). You can write your own SQL if you prefer.

  5. Select an Expectation type.

  6. Complete the mandatory and optional fields for the Expectation. A recurring validation schedule will be applied automatically to your Expectation.

  7. Click Save or click Save & Add More and then repeat steps 4 through 7 to add additional Expectations.

  8. Optional. Run a Validation. See Run a Validation.

Save time with ExpectAI Beta​

ExpectAI is an analytical AI tool that you can use to generate tests.

Generate Expectations​

To accelerate test coverage, you can use ExpectAI to generate recommended Expectations for a Data Asset. These will be personalized based on an analysis of a sample of your data.

Keep the following requirements in mind when working with ExpectAI:

  • Your organization must be using a fully-hosted deployment.
  • The Data Asset's Data Source must be AlloyDB, Amazon Aurora PostgreSQL, Citus, Databricks SQL, Neon, PostgreSQL, Redshift, or Snowflake.

To add AI-recommended Expectations:

  1. In GX Cloud, click Data Assets.
  2. In the Data Assets list, click the Data Asset name.
  3. Click Generate Expectations.
    This might take a few minutes

    ExpectAI may take a few minutes to analyze your data and recommend personalized Expectations. You can navigate away from the page while ExpectAI works in the background. GX will send an email alert when your recommended Expectations are ready for review.

  4. Review the recommended Expectations and Approve (✓) or Reject (✗) them within 48 hours. After 48 hours, any remaining recommendations will be discarded.
  5. Optional. Run a Validation. See Run a Validation.
  6. Optional. Edit AI-generated Expectations based on the insights you get from running a Validation and your data quality needs.

Generate SQL​

To simplify working with custom SQL Expectations, you can use ExpectAI to generate a SQL query based on a natural language prompt you provide and a data profile GX Cloud automatically provides.

For example, imagine you have a New York City taxi trip dataset with columns named pickup_borough, vehicle_type, and passenger_count. If you add a custom SQL Expectation with a Prompt for SQL generation like sedan rides in Manhattan shouldn't have more than 4 passengers then ExpectAI would generate a SQL query similar to the following:

SQL query
SELECT
*
FROM
{batch}
WHERE
pickup_borough = 'Manhattan'
AND vehicle_type = 'Sedan'
AND passenger_count > 4

Keep the following requirements in mind when working with ExpectAI:

  • Your organization must be using a fully-hosted deployment.
  • The Data Asset's Data Source must be AlloyDB, Amazon Aurora PostgreSQL, Citus, Databricks SQL, Neon, PostgreSQL, Redshift, or Snowflake.

Optional. Define a Batch​

If your Data Asset has at least one DATE or DATETIME column, you can define a Batch to validate your data incrementally.

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Next to the current batch configuration, click pencil icon Edit batch.

  4. Choose how to Validate by. Select the Entire Asset tab to provide all Data Asset records to your Expectations and validations, or select one of the Year/Month/Day tabs to use subsets of Data Asset records for your Expectations and validations. Year partitions Data Asset records by year, Month partitions Data Asset records by year and month, Day partitions Data Asset records by year, month, and day.

  5. Select the Batch column that contains the DATE or DATETIME data to partition on.

Edit an Expectation​

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Find the Expectation that you want to edit.

  4. Click pencil icon Edit Expectation for the Expectation that you want to edit.

  5. Edit the Expectation configuration.

  6. Click Save.

Delete an Expectation​

  1. In GX Cloud, click Data Assets.

  2. In the Data Assets list, click the Data Asset name.

  3. Find the Expectation that you want to delete.

  4. Click trash icon Delete Expectation for the Expectation that you want to delete.

    You can delete Expectations in bulk

    If you want to delete all Expectations that test for a certain data quality issue, you can instead click trash icon Bulk-delete Expectations for the relevant category.

  5. Click Delete.