Skip to main content

How to configure credentials using a secrets store

Choose which Secret Manager you are using:

This guide will explain how to configure your great_expectations.yml project config to substitute variables from AWS Secrets Manager.

Prerequisites: This how-to guide assumes you have:
warning

Secrets store substitution uses the configurations from your great_expectations.yml project config after all other types of substitution are applied (from environment variables or from the config_variables.yml config file)

The secrets store substitution works based on keywords. It tries to retrieve secrets from the secrets store for the following values :

  • AWS: values starting with secret|arn:aws:secretsmanager if the values you provide don't match with the keywords above, the values won't be substituted.

Setup#

To use AWS Secrets Manager, you may need to install the great_expectations package with its aws_secrets extra requirement:

pip install great_expectations[aws_secrets]

In order to substitute your value by a secret in AWS Secrets Manager, you need to provide an arn of the secret like this one: secret|arn:aws:secretsmanager:123456789012:secret:my_secret-1zAyu6

note

The last 7 characters of the arn are automatically generated by AWS and are not mandatory to retrieve the secret, thus secret|arn:aws:secretsmanager:region-name-1:123456789012:secret:my_secret will retrieve the same secret.

You will get the latest version of the secret by default.

You can get a specific version of the secret you want to retrieve by specifying its version UUID like this: secret|arn:aws:secretsmanager:region-name-1:123456789012:secret:my_secret:00000000-0000-0000-0000-000000000000

If your secret value is a JSON string, you can retrieve a specific value like this: secret|arn:aws:secretsmanager:region-name-1:123456789012:secret:my_secret|key

Or like this: secret|arn:aws:secretsmanager:region-name-1:123456789012:secret:my_secret:00000000-0000-0000-0000-000000000000|key

Example:

# great_expectations/great_expectations.yml
datasources:  dev_postgres_db:    class_name: SqlAlchemyDatasource    data_asset_type:      class_name: SqlAlchemyDataset      module_name: great_expectations.dataset    module_name: great_expectations.datasource    credentials:      drivername: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|drivername      host: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|host      port: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|port      username: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|username      password: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|password      database: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|database  prod_postgres_db:    class_name: SqlAlchemyDatasource    data_asset_type:      class_name: SqlAlchemyDataset      module_name: great_expectations.dataset    module_name: great_expectations.datasource    credentials:      drivername: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_DRIVERNAME      host: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_HOST      port: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_PORT      username: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_USERNAME      password: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_PASSWORD      database: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:PROD_DB_CREDENTIALS_DATABASE