Configure credentials, tokens, and connection strings

Credentials, whether they are tokens for accessing third party apps such as Slack or connection strings for accessing your data, should be stored securely outside of version control. GX Core allows you to securely store credentials of all types as environment variables on a local system, or as entries in an uncommitted config file. These credentials are then referenced by variable name in your version controlled code, and implemented by GX through string substitution.

Prerequisites

• The ability to set environment variables or a File Data Context.

GX Core also supports referencing credentials that have been stored in the AWS Secrets Manager, Google Cloud Secret Manager, and Azure Key Vault secrets managers. To set up GX Core to access one of these secrets managers you will additionally require:

• The ability to install Python modules with pip.

Procedure

1. Assign the credentials to a reference variable.

   GX supports the following methods of securely storing credentials. Chose one to implement for your connection string:

   Environment Variables

   Environment variables provide the quickest way to securely set up your credentials.

   You can set environment variables by replacing the values in <> with your information and entering export <VARIABLE_NAME>=<VALUE> commands in the terminal or adding the commands to your ~/.bashrc file. If you use the export command from the terminal, the environment variables will not persist beyond the current session. However, if you add them to your shell config file (~/.bashrc for Bash, ~./zshrc for Z Shell), the variables will be exported each time you log in.

   You can export credentials as individual components, or as an entire connection string (or token). For example:

   Terminal, ~/.bashrc, or ~/.zshrc

   export MY_POSTGRES_USERNAME=<USERNAME>
   export MY_POSTGRES_PASSWORD=<PASSWORD>

   or:

   Terminal or ~/.bashrc

   export POSTGRES_CONNECTION_STRING=postgresql+psycopg2://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>

   You can also reference your stored credentials within a stored connection string by wrapping their corresponding variable in ${ and }. For example:

   Terminal or ~/.bashrc

   export MY_POSTGRES_USERNAME=<USERNAME>
   export MY_POSTGRES_PASSWORD=<PASSWORD>
   export POSTGRES_CONNECTION_STRING=postgresql+psycopg2://${MY_POSTGRES_USERNAME}:${MY_POSTGRES_PASSWORD}@<HOST>:<PORT>/<DATABASE>

   Because the dollar sign character $ is used to indicate the start of a string substitution they should be escaped using a backslash \ if they are part of your credentials. For example, if your password is pa$$word then in the previous examples you would use the command:

   Terminal or ~/.bashrc

   export MY_POSTGRES_PASSWORD=pa\$\$word

   config.yml

   YAML files make variables more visible, are easier to edit, and allow for modularization. For example, you can create a YAML file for development and testing and another for production.

   A File Data Context is required before you can configure credentials in a YAML file. By default, the credentials file in a File Data Context is located at /great_expectations/uncommitted/config_variables.yml. The uncommitted/ directory is included in a default .gitignore and will be excluded from version control.

   These examples demonstrate how to save credentials in the form of a connection string for a database. However, the same process can be used for things such as web app tokens or any other credential that can be stored in string format.

   Each entry in config_variables.yml should consist of two parts. The first is a variable which you will reference in the place of the credential. The second is the value that should be substituted for that variable when it is referenced. For example:

   config_variables.yml

   MY_POSTGRES_USERNAME: <USERNAME>
   MY_POSTGRES_PASSWORD: <PASSWORD>

   or:

   config_variables.yml

   POSTGRES_CONNECTION_STRING: postgresql+psycopg2://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>

   You can also reference your stored credentials within a stored connection string by wrapping their corresponding variable in ${ and }. For example:

   config_variables.yml

   MY_POSTGRES_USERNAME: <USERNAME>
   MY_POSTGRES_PASSWORD: <PASSWORD>
   POSTGRES_CONNECTION_STRING: postgresql+psycopg2://${MY_POSTGRES_USERNAME}:${MY_POSTGRES_PASSWORD}@<HOST>:<PORT>/<DATABASE>

   Because the dollar sign character $ is used to indicate the start of a string substitution they should be escaped using a backslash \ if they are part of your credentials. For example, if your password is pa$$word then in the previous examples you would use the command:

   Terminal

   export MY_POSTGRES_PASSWORD=pa\$\$word

2. Access your credentials in Python strings.

   Securely stored credentials are implemented via string substitution. You can reference your credentials in a Python string by wrapping the variable name they are assigned to in ${ and }. Using individual credentials for a connection string would look like:

   Python

   connection_string="postgresql+psycopg2://${MY_POSTGRES_USERNAME}:${MY_POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DATABASE}",

   Or you could reference a configured variable that contains the full connection string by providing a Python string that contains just a reference to that variable:

   Python

   my_connection_string = "${POSTGRESQL_CONNECTION_STRING}"

   When you pass a string that references your stored credentials to a GX Core method that requires string formatted credentials as a parameter the referenced variable in your Python string will be substituted for the corresponding stored value.