By default, Validation results are stored in the
uncommitted/validations/ directory. Since Validations may include examples of data (which could be sensitive or regulated) they should not be committed to a source control system. This guide will help you configure a new storage location for Validations on your filesystem.
This guide will explain how to use a Validation Action to update Data Docs sites with new Validation Results from Validation Operator runs.
Prerequisites: This how-to guide assumes you have:
- Completed the Getting Started Tutorial
- Have a working installation of Great Expectations
- Configured a Data Context.
- Configured an Expectation Suite .
- Configured a Checkpoint.
- Determined a new storage location where you would like to store Validations. This can either be a local path, or a path to a secure network filesystem.
Configure a new folder on your filesystem where Validation results will be stored.
Create a new folder where you would like to store your Validation results, and move your existing Validation results over to the new location. In our case, the name of the Validation result is
npi_validationsand the path to our new storage location is
# in the great_expectations/ foldermkdir shared_validationsmv uncommitted/validations/npi_validations/ uncommitted/shared_validations/
Identify your Data Context Validations Store
great_expectations.yml, look for the following lines. The configuration tells Great Expectations to look for Validations in a store called
validations_storeis set to
validations_store_name: validations_store stores: validations_store: class_name: ValidationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: uncommitted/validations/
Update your configuration file to include a new store for Validation results on your filesystem
In the example below, Validations Store is being set to
shared_validations_filesystem_store, but it can be any name you like. Also, the
base_directoryis being set to
uncommitted/shared_validations/, but it can be set to any path accessible by Great Expectations.
validations_store_name: shared_validations_filesystem_store stores: shared_validations_filesystem_store: class_name: ValidationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: uncommitted/shared_validations/
Confirm that the location has been updated by running
great_expectations --v3-api store list.
Notice the output contains two Validation stores: the original
shared_validations_filesystem_storewe just configured. This is ok, since Great Expectations will look for Validations in the
uncommitted/shared_validations/folder as long as we set the
shared_validations_filesystem_store. The config for
validations_storecan be removed if you would like.
great_expectations --v3-api store list - name: validations_store class_name: ValidationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: uncommitted/validations/ - name: shared_validations_filesystem_store class_name: ValidationsStore store_backend: class_name: TupleFilesystemStoreBackend base_directory: uncommitted/shared_validations/
Confirm that the Validations store has been correctly configured
If it would be useful to you, please comment with a +1 and feel free to add any suggestions or questions below. Also, please reach out to us on Slack if you would like to learn more, or have any questions.