Install additional dependencies

Some environments and Data Sources require additional Python libraries or third party utilities that are not included in the base installation of GX Core. Use the information provided here to install the necessary dependencies for Amazon S3, Azure Blob Storage, Google Cloud Storage, and SQL databases.

Amazon S3

GX Core uses the Python library boto3 to access objects stored in Amazon S3 buckets, but you must configure your Amazon S3 account and credentials through AWS and the AWS command line interface (CLI).

Prerequisites

• The AWS CLI. See Installing or updating the latest version of the AWS CLI.
• AWS credentials. See Configuring the AWS CLI.
• Python version 3.8 to 3.11
• Recommended. A Python virtual environment.

Installation

Python interacts with AWS through the boto3 library. GX Core uses the library in the background when working with AWS. Although you won't use boto3 directly, must install it in your Python environment.

To set up boto3 with AWS, and use boto3 within Python, see the Boto3 documentation.

1. Run the following code to verify the AWS CLI version:

   Terminal input

   aws --version

   If this command does not return AWS CLI version information, reinstall or update the AWS CLI. See Install or update to the latest version of the AWS CLI.

2. Run the following terminal command to install boto3 in your Python environment:

   Terminal input

   python -m pip install boto3
   tip

   If the python -m pip install boto3 does not work, try:

   Terminal input

   python3 -m pip install boto3

   If these pip commands work, verify that Python is installed correctly.

3. Run the following terminal command to verify your AWS credentials are properly configured:

   Terminal input

   aws sts get-caller-identity

   When your credentials are properly configured, your UserId, Account, and Arn are returned. If your credentials are not configured correctly, an error message appears. If you received an error message, or you couldn't verify your credentials, see Amazon's documentation for Configure the AWS CLI.

4. Install the Python dependencies for AWS S3 support.

   Run the following terminal command to install the optional dependencies required by GX Core to work with AWS S3:

   info

   If you installed GX in a virtual environment, your virtual environment should be active when you install these dependencies.
   Terminal input

   python -m pip install 'great_expectations[s3]'

   GX Core and the requirements for the boto3 Python library are installed.

Microsoft Azure Blob Storage

Azure Blob Storage stores unstructured data on the Microsoft cloud data storage platform. To validate Azure Blob Storage data with GX Core you install additional Python libraries and define a connection string.

Prerequisites

• An Azure Storage account.
• Azure storage account access keys.
• Python version 3.8 to 3.11
• Recommended. A Python virtual environment.

Installation

1. Install the Python dependencies for Azure Blob Storage support.

   Run the following code to install GX Core with the additional Python libraries needed to work with Azure Blob Storage:

   info

   If you installed GX in a virtual environment, your virtual environment should be active when you install these dependencies.
   Terminal input

   python -m pip install 'great_expectations[azure]'

2. Configure your Azure Blob Storage credentials.

   To store your Azure Blob Storage credentials as an environment variable, replace <YOUR-STORAGE-ACCOUNT-NAME> and <YOUR-STORAGE-ACCOUNT-KEY> in the following terminal command with your Azure Blob Storage account values:

   export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=<YOUR-STORAGE-ACCOUNT-NAME>;AccountKey=<YOUR-STORAGE-ACCOUNT-KEY>"
   info

   You can manage your credentials for all environments and Data Sources by storing them as environment variables. To do this, enter export ENV_VARIABLE_NAME=env_var_value in the terminal or add the equivalent command to your ~/.bashrc file.

   As an alternative to environment variables, you can also store credentials in the file config_variables.yml after you have created a File Data Context.

Google Cloud Storage

To validate Google Cloud Platform (GCP) data with GX Core, you create your GX Python environment, configure your GCP credentials, and install GX Core locally with the additional dependencies to support GCP.

Prerequisites

• A GCP service account with permissions to access GCP resources and storage Objects.
• The GOOGLE_APPLICATION_CREDENTIALS environment variable is set. See the Google documentation Set up Application Default Credentials.
• Google Cloud API authentication is set up. See the Google documentation Set up authentication.
• Python version 3.8 to 3.11
• Recommended. A Python virtual environment.

Installation

1. Ensure your GCP credentials are correctly configured. This process includes:

   • Creating a Google Cloud Platform (GCP) service account.
   • Setting the GOOGLE_APPLICATION_CREDENTIALS environment variable,
   • Verifying authentication by running a Google Cloud Storage client library script.

   For more information, see the GCP documentation on how to verify authentication for the Google Cloud API.

2. Install the Python dependencies for GCP support.

   Run the following terminal command to install GX Core with the additional dependencies for GCP support:

   info

   If you installed GX in a virtual environment, your virtual environment should be active when you install these dependencies.
   Terminal input

   python -m pip install 'great_expectations[gcp]'

SQL databases

To validate data stored on SQL databases with GX Core, you create your GX Python environment, install GX Core locally, and then configure the necessary dependencies.

Prerequisites

• Python version 3.8 to 3.11
• Recommended. A Python virtual environment.

Installation

1. Run the pip command to install the dependencies for your data's SQL dialect.

   The following table lists the installation commands used to install GX Core dependencies for specific SQL dialects.

   SQL Dialect	Command
   AWS Athena	pip install 'great_expectations[athena]'
   BigQuery	pip install 'great_expectations[bigquery]'
   MSSQL	pip install 'great_expectations[mssql]'
   PostgreSQL	pip install 'great_expectations[postgresql]'
   Redshift	pip install 'great_expectations[redshift]'
   Snowflake	pip install 'great_expectations[snowflake]'
   Trino	pip install 'great_expectations[trino]'

   To install dependencies for a specific SQL dialect, use the corresponding command from the table above.

   If you are not using one of the listed dialects, you can install the dependencies for SQLAlchemy with the command:

   Terminal input

   python -m pip install 'great_expectations[sqlalchemy]'

2. Configure your SQL database credentials.

   You can store your SQL database password by replacing <MY_PASSWORD> with your password in the following command:

   Terminal input

   export MY_DB_PW=<MY_PASSWORD>

   Or you can store your entire SQL database connection string by replacing <MY_CONNECTION_STRING> with it and running:

   Terminal input

   export MY_DB_CONNECTION_STRING=<MY_CONNECTION_STRING>
   info

   You can manage your credentials for all environments and Data Sources by storing them as environment variables. To do this, enter export ENV_VARIABLE_NAME=env_var_value in the terminal or add the equivalent command to your ~/.bashrc file.

   You can reference environment variables in GX Core by including them in strings using the format ${ENV_VARIABLE_NAME}. For instance, to insert the password stored as MY_DB_PASSWORD into a PostgreSql connection string you would provide the string:

   Example PostgreSql Connection String

   "postgresql+psycopg2://<username>:${MY_DB_PW}@<host>:<port>/<database>"

   As an alternative to environment variables, you can also store credentials in the file config_variables.yml after you have created a File Data Context.

Spark

To validate data while using Spark to read from dataframes or file formats such as .csv and .parquet with GX Core, you create your GX Python environment, install GX Core locally, and then configure the necessary dependencies.

Prerequisites

• Python version 3.8 to 3.11
• Recommended. A Python virtual environment.

Installation

1. Optional. Activate your virtual environment.

   If you installed GX in a virtual environment, your virtual environment should be active when you install these dependencies.

2. Run the pip command to install the dependencies for Spark:

   Terminal input

   python -m pip install 'great_expectations[spark]'