great_expectations.data_context.store

Package Contents

Classes

DatabaseStoreBackend(credentials, table_name, key_columns, fixed_length_key=True)

A store backend acts as a key-value store that can accept tuples as keys, to abstract away

ExpectationsStore(store_backend=None, runtime_environment=None)

An Expectations Store provides a way to store Expectation Suites accessible to a Data Context.

HtmlSiteStore(store_backend=None, runtime_environment=None)

A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results.

EvaluationParameterStore(store_backend=None)

A MetricStore stores ValidationMetric information to be used between runs.

MetricStore(store_backend=None)

A MetricStore stores ValidationMetric information to be used between runs.

SqlAlchemyQueryStore(credentials, queries=None, store_backend=None, runtime_environment=None)

SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query

Store(store_backend=None, runtime_environment=None)

A store is responsible for reading and writing Great Expectations objects

InMemoryStoreBackend(runtime_environment=None, fixed_length_key=False)

Uses an in-memory dictionary as a store backend.

StoreBackend(fixed_length_key=False)

A store backend acts as a key-value store that can accept tuples as keys, to abstract away

TupleFilesystemStoreBackend(base_directory, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, root_directory=None, fixed_length_key=False, base_public_path=None)

Uses a local filepath as a store.

TupleGCSStoreBackend(bucket, project, prefix=’’, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, public_urls=True, base_public_path=None)

Uses a GCS bucket as a store.

TupleS3StoreBackend(bucket, prefix=’’, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, base_public_path=None, endpoint_url=None)

Uses an S3 bucket as a store.

TupleStoreBackend(filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, fixed_length_key=False, base_public_path=None)

If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with

ValidationsStore(store_backend=None, runtime_environment=None)

A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs.

class great_expectations.data_context.store.DatabaseStoreBackend(credentials, table_name, key_columns, fixed_length_key=True)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.

In general a StoreBackend implementation must provide implementations of:
  • _get

  • _set

  • list_keys

  • _has_key

_get(self, key)
_set(self, key, value, allow_update=True)
abstract _move(self)
get_url_for_key(self, key)
_convert_engine_and_key_to_url(self, key)
_has_key(self, key)
list_keys(self, prefix=())
remove_key(self, key)
class great_expectations.data_context.store.ExpectationsStore(store_backend=None, runtime_environment=None)

Bases: great_expectations.data_context.store.store.Store

An Expectations Store provides a way to store Expectation Suites accessible to a Data Context.

Feature Maturity

icon-08800558347111eb94ce0242ac110002 Expectation Store - Git - How-to Guide
Use a git repository to store expectation suites.
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-08800710347111eb94ce0242ac110002 Expectation Store - Filesystem - How-to Guide
Filesystem-based Expectations Store
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-088008be347111eb94ce0242ac110002 Expectation Store - S3 - How-to Guide
Use an Amazon Web Services S3 bucket to store expectations.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Low
icon-088009c2347111eb94ce0242ac110002 Expectation Store - GCS - How-to Guide
Use a Google Cloud Platform Cloud Storage bucket to store expectations.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial
Bug Risk: Low
icon-08800a94347111eb94ce0242ac110002 Expectation Store - Azure - How-to Guide
Use Microsoft Azure Blob Storage to store expectations.
Maturity: N/A
Details:
API Stability: Stable
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
remove_key(self, key)
serialize(self, key, value)
deserialize(self, key, value)
self_check(self, pretty_print)
class great_expectations.data_context.store.HtmlSiteStore(store_backend=None, runtime_environment=None)

A HtmlSiteStore facilitates publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results.

Feature Maturity

icon-0880409a347111eb94ce0242ac110002 HTML Site Store - Filesystem - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on the Filesystem
Maturity: Production
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Partial
Bug Risk: Low
icon-0880420c347111eb94ce0242ac110002 HTML Site Store - S3 - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on S3
Maturity: Beta
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Moderate
icon-088042fc347111eb94ce0242ac110002 HTMLSiteStore - GCS - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on GCS
Maturity: Beta
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial (needs auth)
Bug Risk: Moderate (resource URL may have bugs)
icon-088043c4347111eb94ce0242ac110002 HTMLSiteStore - Azure - How-to Guide
For publishing rendered documentation built from Expectation Suites, Profiling Results, and Validation Results on Azure Blob Storage
Maturity: N/A
Details:
API Stability: Mostly Stable (profiling)
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
get(self, key)
set(self, key, serialized_value)
get_url_for_resource(self, resource_identifier=None, only_if_exists=True)

Return the URL of the HTML document that renders a resource (e.g., an expectation suite or a validation result).

Parameters

resource_identifier – ExpectationSuiteIdentifier, ValidationResultIdentifier or any other type’s identifier. The argument is optional - when not supplied, the method returns the URL of the index page.

Returns

URL (string)

_validate_key(self, key)
list_keys(self)
write_index_page(self, page)

This third param_store has a special method, which uses a zero-length tuple as a key.

clean_site(self)
copy_static_assets(self, static_assets_source_dir=None)

Copies static assets, using a special “static_assets” backend store that accepts variable-length tuples as keys, with no filepath_template.

class great_expectations.data_context.store.EvaluationParameterStore(store_backend=None)

Bases: great_expectations.data_context.store.metric_store.MetricStore

A MetricStore stores ValidationMetric information to be used between runs.

get_bind_params(self, run_id)
class great_expectations.data_context.store.MetricStore(store_backend=None)

Bases: great_expectations.data_context.store.store.Store

A MetricStore stores ValidationMetric information to be used between runs.

_key_class
_validate_value(self, value)
serialize(self, key, value)
deserialize(self, key, value)
class great_expectations.data_context.store.SqlAlchemyQueryStore(credentials, queries=None, store_backend=None, runtime_environment=None)

Bases: great_expectations.data_context.store.store.Store

SqlAlchemyQueryStore stores queries by name, and makes it possible to retrieve the resulting value by query name.

_key_class
_convert_key(self, key)
get(self, key)
set(self, key, value)
get_query_result(self, key, query_parameters=None)
class great_expectations.data_context.store.Store(store_backend=None, runtime_environment=None)

A store is responsible for reading and writing Great Expectations objects to appropriate backends. It provides a generic API that the DataContext can use independently of any particular ORM and backend.

An implementation of a store will generally need to define the following:
  • serialize

  • deserialize

  • _key_class (class of expected key type)

All keys must have a to_tuple() method.

_key_class
_validate_key(self, key)
property store_backend(self)
serialize(self, key, value)
key_to_tuple(self, key)
tuple_to_key(self, tuple_)
deserialize(self, key, value)
get(self, key)
set(self, key, value)
list_keys(self)
has_key(self, key)
self_check(self, pretty_print)
class great_expectations.data_context.store.InMemoryStoreBackend(runtime_environment=None, fixed_length_key=False)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

Uses an in-memory dictionary as a store backend.

_get(self, key)
_set(self, key, value, **kwargs)
_move(self, source_key, dest_key, **kwargs)
list_keys(self, prefix=())
_has_key(self, key)
remove_key(self, key)
class great_expectations.data_context.store.StoreBackend(fixed_length_key=False)

A store backend acts as a key-value store that can accept tuples as keys, to abstract away reading and writing to a persistence layer.

In general a StoreBackend implementation must provide implementations of:
  • _get

  • _set

  • list_keys

  • _has_key

IGNORED_FILES = ['.ipynb_checkpoints']
property fixed_length_key(self)
get(self, key, **kwargs)
set(self, key, value, **kwargs)
move(self, source_key, dest_key, **kwargs)
has_key(self, key)
get_url_for_key(self, key, protocol=None)
_validate_key(self, key)
_validate_value(self, value)
abstract _get(self, key)
abstract _set(self, key, value, **kwargs)
abstract _move(self, source_key, dest_key, **kwargs)
abstract list_keys(self, prefix=())
abstract remove_key(self, key)
abstract _has_key(self, key)
is_ignored_key(self, key)
class great_expectations.data_context.store.TupleFilesystemStoreBackend(base_directory, filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, root_directory=None, fixed_length_key=False, base_public_path=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses a local filepath as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.

_get(self, key)
_set(self, key, value, **kwargs)
_move(self, source_key, dest_key, **kwargs)
list_keys(self, prefix=())
rrmdir(self, mroot, curpath)

recursively removes empty dirs between curpath and mroot inclusive

remove_key(self, key)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
_has_key(self, key)
class great_expectations.data_context.store.TupleGCSStoreBackend(bucket, project, prefix='', filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, public_urls=True, base_public_path=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses a GCS bucket as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added.

The filepath_template is a string template used to convert the key to a filepath.

_build_gcs_object_key(self, key)
_move(self, source_key, dest_key, **kwargs)
_get(self, key)
_set(self, key, value, content_encoding='utf-8', content_type='application/json')
list_keys(self)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
_get_path_url(self, path)
remove_key(self, key)
_has_key(self, key)
class great_expectations.data_context.store.TupleS3StoreBackend(bucket, prefix='', filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=False, fixed_length_key=False, base_public_path=None, endpoint_url=None)

Bases: great_expectations.data_context.store.tuple_store_backend.TupleStoreBackend

Uses an S3 bucket as a store.

The key to this StoreBackend must be a tuple with fixed length based on the filepath_template, or a variable-length tuple may be used and returned with an optional filepath_suffix (to be) added. The filepath_template is a string template used to convert the key to a filepath.

_build_s3_object_key(self, key)
_get(self, key)
_set(self, key, value, content_encoding='utf-8', content_type='application/json')
_move(self, source_key, dest_key, **kwargs)
list_keys(self)
get_url_for_key(self, key, protocol=None)
get_public_url_for_key(self, key, protocol=None)
remove_key(self, key)
_has_key(self, key)
class great_expectations.data_context.store.TupleStoreBackend(filepath_template=None, filepath_prefix=None, filepath_suffix=None, forbidden_substrings=None, platform_specific_separator=True, fixed_length_key=False, base_public_path=None)

Bases: great_expectations.data_context.store.store_backend.StoreBackend

If filepath_template is provided, the key to this StoreBackend abstract class must be a tuple with fixed length equal to the number of unique components matching the regex r”{d+}”

For example, in the following template path: expectations/{0}/{1}/{2}/prefix-{2}.json, keys must have three components.

_validate_key(self, key)
_validate_value(self, value)
_convert_key_to_filepath(self, key)
_convert_filepath_to_key(self, filepath)
verify_that_key_to_filepath_operation_is_reversible(self)
class great_expectations.data_context.store.ValidationsStore(store_backend=None, runtime_environment=None)

Bases: great_expectations.data_context.store.store.Store

A ValidationsStore manages Validation Results to ensure they are accessible via a Data Context for review and rendering into Data Docs.

Feature Maturity

icon-08816f2e347111eb94ce0242ac110002 Validations Store - Filesystem - How-to Guide
Use a locally-mounted filesystem to store validation results.
Maturity: Production
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: N/A
Documentation Completeness: Complete
Bug Risk: Low
icon-088170a0347111eb94ce0242ac110002 Validations Store - S3 - How-to Guide
Use an Amazon Web Services S3 bucket to store validation results.
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Complete
Bug Risk: Low
icon-08817190347111eb94ce0242ac110002 Validations Store - GCS - How-to Guide
# What it does <br /> Store validation results in a Google Cloud Storage bucket. You may optionally specify a key to use. <br /> <br /> See the GCS Store backend [module docs](https://docs.greatexpectations.io/en/latest/autoapi/great_expectations/data_context/store/tuple_store_backend/index.html#great_expectations.data_context.store.tuple_store_backend.TupleGCSStoreBackend) for more information.”
Maturity: Beta
Details:
API Stability: Stable
Implementation Completeness: Complete
Unit Test Coverage: Complete
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Partial
Bug Risk: Low
icon-08817262347111eb94ce0242ac110002 Validations Store - Azure - How-to Guide
Use Microsoft Azure Blob Storage to store validation results.
Maturity: N/A
Details:
API Stability: Stable
Implementation Completeness: Minimal
Unit Test Coverage: Minimal
Integration Infrastructure/Test Coverage: Minimal
Documentation Completeness: Minimal
Bug Risk: Moderate
_key_class
serialize(self, key, value)
deserialize(self, key, value)
self_check(self, pretty_print)