Deploying Great Expectations in a hosted environment without file system or CLI¶
If you follow the steps of the Getting started tutorial, you create a standard deployment of Great Expectations. By default, this relies on two components:
The Great Expectations command-line interface (CLI) to initialize a Data Context, create Expectation Suites, add Datasources, etc.
great_expectations.ymlfile to configure your Data Context, e.g. to point at different Stores for validation results, etc.
However, you might not have these components available in hosted environments, such as Databricks, AWS EMR, Google Cloud Composer, and others. This workflow guide will outline the main steps required to successfully use Great Expectations in a hosted environment.
Step 1: Configure your Data Context¶
Instead of using the Great Expectations CLI, you can create a Data Context directly in code. Your Data Context also manages the following components described in this guide:
Datasources to connect to data
Stores to save Expectations and validation results
Data Docs hosting
The following guide gives an overview of creating an in-code Data Context including defaults to help you more quickly set one up for common configurations:
The following guides will contain examples for each environment we have tested out:
Step 2: Create Expectation Suites and add Expectations¶
If you want to create an Expectation Suite in your environment without using the CLI, you can simply follow this guide: How to create a new Expectation Suite without the CLI.
In order to store your Expectation Suites so you can load them for validation at a later point, you will need to ensure that you have an Expectation Store configured: Configuring metadata stores.
Step 3: Run validation¶
In order to use an Expectation Suite you’ve created to validate data, follow this guide: How to validate data without a Checkpoint
Step 4: Use Data Docs¶
Finally, if you would like to build and view Data Docs in your environment, please follow the guides for configuring Data Docs: Configuring Data Docs.