Data Docs Reference

Data Docs make it simple to visualize data quality in your project. These include Expectations, Validations & Profiles. They are built for all Datasources from JSON artifacts in the local repo including validations & profiles from the uncommitted directory.

Users have full control over configuring Data Documentation for their project - they can modify the pre-configured site (or remove it altogether) and add new sites with a configuration that meets the project’s needs. The easiest way to add a new site to the configuration is to copy the “local_site” configuration block in great_expectations.yml, give the copy a new name and modify the details as needed.

Data Docs Site Configuration

The default Data Docs site configuration looks like this:

data_docs_sites:
  local_site:
    class_name: SiteBuilder
    store_backend:
      class_name: FixedLengthTupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/

Here is an example of a site configuration from great_expectations.yml with defaults defined explicitly:

data_docs_sites:
  local_site: # site name
    datasource_whitelist: '*' # used to restrict the Datasources
    module_name: great_expectations.render.renderer.site_builder
    class_name: SiteBuilder
    store_backend:
      class_name: FixedLengthTupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
      class_name: DefaultSiteIndexBuilder
    site_section_builders:
      expectations: # if not present, expectation suites are not rendered
        class_name: DefaultSiteSectionBuilder
        source_store_name: expectations_store
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ExpectationSuitePageRenderer

      validations: # if not present, validation results are not rendered
        class_name: DefaultSiteSectionBuilder
        source_store_name: validations_store
        run_id_filter:
          ne: profiling # exclude validations with run id "profiling" - reserved for profiling results
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ValidationResultsPageRenderer

      profiling: # if not present, profiling results are not rendered
        class_name: DefaultSiteSectionBuilder
        source_store_name: validations_store
        run_id_filter:
          eq: profiling
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ProfilingResultsPageRenderer

validations_store in the example above specifies the name of a store configured in the stores section. Validation and profiling results from that store will be included in the documentation. The optional run_id_filter attribute allows to include (eq for exact match) or exclude (ne) validation results with a particular run id.

Automatically Publishing Data Docs

It is possible to directly publish (continuously updating!) data docs sites to a shared location such as a static site hosted in S3 by simply updating the store_backend configuration in the site configuration. If we modify the configuration in the example above to adjust the store backend to an S3 bucket of our choosing, our SiteBuilder will automatically save the resulting site to that bucket.

store_backend:
  class_name: FixedLengthTupleS3StoreBackend
  bucket: data-docs.my_org.org
  prefix:

See the tutorial on publishing data docs to S3 for more information.

More advanced configuration

It is possible to extend renderers and views and customize the particular class used to render any of the objects in your documentation. In this more advanced configuration, a “CustomTableContentBlockRenderer” is used only for the validations renderer, and no profiling results are rendered at all.

data_docs_sites:
  # Data Docs make it simple to visualize data quality in your project. These
  # include Expectations, Validations & Profiles. The are built for all
  # Datasources from JSON artifacts in the local repo including validations &
  # profiles from the uncommitted directory. Read more at
  # https://docs.greatexpectations.io/en/latestfeatures/data_docs.html
  local_site:
    class_name: SiteBuilder
    store_backend:
      class_name: FixedLengthTupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_section_builders:
      expectations:
        class_name: DefaultSiteSectionBuilder
        source_store_name: expectations_store
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ExpectationSuitePageRenderer

      validations:
        class_name: DefaultSiteSectionBuilder
        source_store_name: validations_store
        run_id_filter:
          ne: profiling
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ValidationResultsPageRenderer
          column_section_renderer:
            class_name: ValidationResultsColumnSectionRenderer
            table_renderer:
              module_name: custom_renderers.custom_table_content_block
              class_name: CustomTableContentBlockRenderer

To support that custom renderer, we need to ensure the implementation is available in our plugins/ directory. Note that we can use a subdirectory and standard python submodule notation, but that we need to include an __init__.py file in our custom_renderers package.

plugins/
├── custom_renderers
│   ├── __init__.py
│   └── custom_table_content_block.py
└── additional_ge_plugin.py

Building Data Docs

Using the CLI

The great_expectations CLI can build comprehensive Data Docs from expectation suites available to the configured context and validations available in the great_expectations/uncommitted directory.

great_expectations build-docs

When called without additional arguments, this command will render all the Data Docs sites specified in great_expectations.yml configuration file into HTML and open them in a web browser.

The command will print out the locations of index.html file for each site.

To disable the web browser opening behavior use –no-view option.

To render just one site, use –site_name SITE_NAME option.

Here is when the build-docs command should be called:

  • when you want to fully rebuild a Data Docs site

  • after a new expectation suite is added or an existing one is edited

  • after new data is profiled (only if you declined the prompt to build data docs when running the profiling command)

When a new validation result is generated after running a Validation Operator, the Data Docs sites will add this result automatically if the operator has the UpdateDataDocsAction action configured (read Actions).

Using the raw API

The underlying python API for rendering documentation is still new and evolving. Use the following snippet as a guide for how to profile a single batch of data and build documentation from the validation_result.

from great_expectations.profile.basic_dataset_profiler import BasicDatasetProfiler
from great_expectations.render.renderer import ProfilingResultsPageRenderer, ExpectationSuitePageRenderer
from great_expectations.data_context.util import safe_mmkdir
from great_expectations.render.view import DefaultJinjaPageView

profiling_html_filepath = '/path/into/which/to/save/results'

# obtain the DataContext object
context = ge.data_context.DataContext()

# load a batch from the data asset
data_asset_name = context.normalize_data_asset_name('ratings')
context.creat_expectation_suite(data_asset_name, 'default'),
batch = context.get_batch(
  data_asset_name=data_asset_name,
  expectation_suite_name='default',
  context.yield_batch_kwargs(data_asset_name))

# run the profiler on the batch - this returns an expectation suite and validation results for this suite
expectation_suite, validation_result = BasicDatasetProfiler().profile(batch)

# use a renderer to produce a document model from the validation results
document_model = ProfilingResultsPageRenderer().render(validation_result)

# use a view to render the document model (produced by the renderer) into a HTML document
safe_mmkdir(os.path.dirname(profiling_html_filepath))
with open(profiling_html_filepath, 'w') as writer:
    writer.write(DefaultJinjaPageView.render(document_model))

Customizing Data Docs

Introduction

Data Docs uses the Jinja template engine to generate HTML pages. The built-in Jinja templates used to compile Data Docs pages are implemented in the great_expectations.render.view module and are tied to View classes. Views determine how page content is displayed. The content data that Views specify and consume is generated by Renderer classes and are implemented in the great_expectations.render.renderer module. Renderers take Great Expectations objects as input and return typed dictionaries - Views take these dictionaries as input and output rendered HTML.

Built-In Views and Renderers

Out of the box, Data Docs supports two top-level Views (i.e. pages), great_expectations.render.view.DefaultJinjaIndexPageView, for a site index page, and great_expectations.render.view.DefaultJinjaPageView for all other pages. Pages are broken into sections - great_expectations.render.view.DefaultJinjaSectionView - which are composed of UI components - great_expectations.render.view.DefaultJinjaComponentView. Each of these Views references a single base Jinja template, which can incorporate any number of other templates through inheritance.

Data Docs comes with the following built-in site page Renderers:

  • great_expectations.render.renderer.SiteIndexPageRenderer (index page)

  • great_expectations.render.renderer.ProfilingResultsPageRenderer (Profiling Results pages)

  • great_expectations.render.renderer.ExpectationSuitePageRenderer (Expectation Suite pages)

  • great_expectations.render.renderer.ValidationResultsPageRenderer (Validation Results pages)

Analogous to the base View templates referenced above, these Renderers can be thought of as base Renderers for the primary Data Docs pages, and may call on many other ancillary Renderers.

It is possible to extend Renderers and Views and customize the particular class used to render any of the objects in your documentation.

Other Tools

In addition to Jinja, Data Docs draws on the following libraries to compile HTML documentation, which you can use to further customize HTML documentation:

  • Bootstrap

  • Font Awesome

  • jQuery

  • Altair

Getting Started

Making Minor Adjustments

Many of the HTML elements in the default Data Docs pages have pre-configured classes that you may use to make minor adjustments using your own custom CSS. By default, when you run great_expectations init, Great Expectations creates a scaffold within the plugins directory for customizing Data Docs. Within this scaffold is a file called data_docs_custom_styles.css - this CSS file contains all the pre-configured classes you may use to customize the look and feel of the default Data Docs pages. All custom CSS, applied to these pre-configured classes or any other HTML elements, must be placed in this file.

Scaffolded directory tree:

plugins
└── custom_data_docs
    ├── renderers
    ├── styles
    │   └── data_docs_custom_styles.css
    └── views

Using Custom Views and Renderers

Suppose you start a new Great Expectations project by running great_expectations init and compile your first Data Docs site. After looking over the local site, you decide you want to implement the following changes:

  1. A completely new Expectation Suite page, requiring a new View and Renderer

  2. A smaller modification to the default Validation page, swapping out a child renderer for a custom version

  3. Remove Profiling Results pages from the documentation

To make these changes, you must first implement the custom View and Renderers and ensure they are available in the plugins directory specified in your project configuration (plugins/ by default). Note that you can use a subdirectory and standard python submodule notation, but you must include an __init__.py file in your custom package. By default, when you run great_expectations init, Great Expectations creates placeholder directories for your custom views, renderers, and CSS stylesheets within the plugins directory. If you wish, you may save your custom views and renderers in an alternate location however, any CSS stylesheets must be saved to plugins/custom_data_docs/styles.

Scaffolded directory tree:

plugins
└── custom_data_docs
    ├── renderers
    ├── styles
    │   └── data_docs_custom_styles.css
    └── views

When you are done with your implementations, your plugins/ directory has the following structure:

plugins
└── custom_data_docs
    ├── renderers
        ├── __init__.py
        ├── custom_table_renderer.py
        └── custom_expectation_suite_page_renderer.py
    ├── styles
    │   └── data_docs_custom_styles.css
    └── views
        ├── __init__.py
        └── custom_expectation_suite_view.py

For Data Docs to use your custom Views and Renderers when compiling your local Data Docs site, you must specify where to use them in the data_docs_sites section of your project configuration.

Before modifying your project configuration, the relevant section looks like this:

data_docs_sites:
  local_site:
    class_name: SiteBuilder
    store_backend:
      class_name: FixedLengthTupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/

This is what it looks like after your changes are added:

data_docs_sites:
  local_site:
    class_name: SiteBuilder
    store_backend:
      class_name: FixedLengthTupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_section_builders:
      expectations:
        class_name: DefaultSiteSectionBuilder
        source_store_name: expectations_store
        renderer:
          module_name: custom_data_docs.renderers.custom_expectation_suite_page_renderer
          class_name: CustomExpectationSuitePageRenderer
        view:
          module_name: custom_data_docs.views.custom_expectation_suite_view
          class_name: CustomExpectationSuiteView
      validations:
        class_name: DefaultSiteSectionBuilder
        source_store_name: validations_store
        run_id_filter:
          ne: profiling
        renderer:
          module_name: great_expectations.render.renderer
          class_name: ValidationResultsPageRenderer
          column_section_renderer:
            class_name: ValidationResultsColumnSectionRenderer
            table_renderer:
              module_name: custom_data_docs.renderers.custom_table_renderer
              class_name: CustomTableRenderer

Note that if your data_docs_sites configuration contains a site_section_builders key, you must now explicitly provide defaults for anything you would like rendered. By omitting the profiling key within site_section_builders, your third goal is achieved and Data Docs will no longer render Profiling Results pages.

Lastly, to compile your newly-customized Data Docs local site, you run great_expectations build-docs from the command line.

site_section_builders defaults:

site_section_builders:
  expectations: # if not present, expectation suites are not rendered
    class_name: DefaultSiteSectionBuilder
    source_store_name: expectations_store
    renderer:
      module_name: great_expectations.render.renderer
      class_name: ExpectationSuitePageRenderer

  validations: # if not present, validation results are not rendered
    class_name: DefaultSiteSectionBuilder
    source_store_name: validations_store
    run_id_filter:
      ne: profiling # exclude validations with run id "profiling" - reserved for profiling results
    renderer:
      module_name: great_expectations.render.renderer
      class_name: ValidationResultsPageRenderer

  profiling: # if not present, profiling results are not rendered
    class_name: DefaultSiteSectionBuilder
    source_store_name: validations_store
    run_id_filter:
      eq: profiling
    renderer:
      module_name: great_expectations.render.renderer
      class_name: ProfilingResultsPageRenderer

Re-Purposing Built-In Views

If you would like to re-purpose a built-in View, you may do so by implementing a custom renderer that outputs an appropriately typed and structured dictionary for that View.

Built-in Views and corresponding input types:

  • great_expectations.render.view.DefaultJinjaPageView: great_expectations.render.types.RenderedDocumentContent

  • great_expectations.render.view.DefaultJinjaSectionView: great_expectations.render.types.RenderedSectionContent

  • great_expectations.render.view.DefaultJinjaComponentView: great_expectations.render.types.RenderedComponentContent

An example of a custom page Renderer, using all built-in UI elements is provided below.

Custom Page Renderer Example

import altair as alt
import pandas as pd

from great_expectations.render.types import (
    RenderedDocumentContent,
    RenderedSectionContent,
    RenderedComponentContent,
)
from great_expectations.render.renderer import Renderer


class CustomPageRenderer(Renderer):
    @classmethod
    def _get_header_content_block(cls, header="", subheader="", highlight=True):
        return RenderedComponentContent(**{
            "content_block_type": "header",
            "header": {
                "template": header
            },
            "subheader": subheader,
            "styling": {
                "classes": ["col-12"],
                "header": {
                    "classes": ["alert", "alert-secondary"] if highlight else []
                }
            }
        })
    
    @classmethod
    def _get_bullet_list_content_block(cls, header="", subheader="", col=12):
        return RenderedComponentContent(**{
            "content_block_type": "bullet_list",
            "header": header,
            "subheader": subheader,
            "bullet_list": [
                "Aenean porttitor turpis.",
                "Curabitur ligula urna.",
                cls._get_header_content_block(header="nested header content block", subheader="subheader",
                                              highlight=False)
            ],
            "styling": {
                "classes": ["col-{}".format(col)],
                "styles": {
                    "margin-top": "20px"
                },
            },
        })
    
    @classmethod
    def _get_table_content_block(cls, header="", subheader="", col=12):
        return RenderedComponentContent(**{
            "content_block_type": "table",
            "header": header,
            "subheader": subheader,
            "table": [
                ["", "column_1", "column_2"],
                [
                    "row_1",
                    cls._get_bullet_list_content_block(subheader="Nested Bullet List Content Block"),
                    "buffalo"
                ],
                ["row_2", "crayon", "derby"],
            ],
            "styling": {
                "classes": ["col-{}".format(col), "table-responsive"],
                "styles": {
                    "margin-top": "20px"
                },
                "body": {
                    "classes": ["table", "table-sm"]
                }
            },
        })
    
    @classmethod
    def _get_graph_content_block(cls, header="", subheader="", col=12):
        df = pd.DataFrame({
            "value": [1, 2, 3, 4, 5, 6],
            "count": [123, 232, 543, 234, 332, 888]
        })
        bars = alt.Chart(df).mark_bar(size=20).encode(
            y='count:Q',
            x="value:O"
        ).properties(height=200, width=200, autosize="fit")
        chart = bars.to_json()
        
        return RenderedComponentContent(**{
            "content_block_type": "graph",
            "header": header,
            "subheader": subheader,
            "graph": chart,
            "styling": {
                "classes": ["col-{}".format(col)],
                "styles": {
                    "margin-top": "20px"
                },
            },
        })
    
    @classmethod
    def _get_tooltip_string_template_content_block(cls):
        return RenderedComponentContent(**{
            "content_block_type": "string_template",
            "string_template": {
                "template": "This is a string template with tooltip, using a top-level custom tag.",
                "tag": "code",
                "tooltip": {
                    "content": "This is the tooltip content."
                }
            },
            "styling": {
                "classes": ["col-12"],
                "styles": {
                    "margin-top": "20px"
                },
            },
        })
    
    @classmethod
    def _get_string_template_content_block(cls):
        return RenderedComponentContent(**{
            "content_block_type": "string_template",
            "string_template": {
                "template": "$icon This is a Font Awesome Icon, using a param-level custom tag\n$red_text\n$bold_serif",
                "params": {
                    "icon": "",
                    "red_text": "And this is red text!",
                    "bold_serif": "And this is big, bold serif text using style attribute..."
                },
                "styling": {
                    "params": {
                        "icon": {
                            "classes": ["fas", "fa-check-circle", "text-success"],
                            "tag": "i"
                        },
                        "red_text": {
                            "classes": ["text-danger"]
                        },
                        "bold_serif": {
                            "styles": {
                                "font-size": "22px",
                                "font-weight": "bold",
                                "font-family": "serif"
                            }
                        }
                    }
                }
            },
            "styling": {
                "classes": ["col-12"],
                "styles": {
                    "margin-top": "20px"
                },
            },
        })
    
    @classmethod
    def _get_value_list_content_block(cls, header="", subheader="", col=12):
        return RenderedComponentContent(**{
            "content_block_type": "value_list",
            "header": header,
            "subheader": subheader,
            "value_list": [{
                "content_block_type": "string_template",
                "string_template": {
                    "template": "$value",
                    "params": {
                        "value": value
                    },
                    "styling": {
                        "default": {
                            "classes": ["badge", "badge-info"],
                        },
                    }
                }
            } for value in [
                "Andrew",
                "Elijah",
                "Matthew",
                "Cindy",
                "Pam"
            ]],
            "styling": {
                "classes": ["col-{}".format(col)]
            },
        })
    
    @classmethod
    def render(cls, ge_dict={}):
        return RenderedDocumentContent(**{
            "renderer_type": "CustomValidationResultsPageRenderer",
            "data_asset_name": "my_data_asset_name",
            "full_data_asset_identifier": "my_datasource/my_generator/my_generator_asset",
            "page_title": "My Page Title",
            "sections": [
                RenderedSectionContent(**{
                    "section_name": "Header Content Block",
                    "content_blocks": [
                        cls._get_header_content_block(header="Header Content Block", subheader="subheader")]
                }),
                RenderedSectionContent(**{
                    "section_name": "Bullet List Content Block",
                    "content_blocks": [
                        cls._get_header_content_block(header="Bullet List Content Block"),
                        cls._get_bullet_list_content_block(header="My Important List",
                                                           subheader="Unremarkable Subheader")
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "Table Content Block",
                    "content_blocks": [
                        cls._get_header_content_block(header="Table Content Block"),
                        cls._get_table_content_block(header="My Big Data Table"),
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "Value List Content Block",
                    "content_blocks": [
                        cls._get_header_content_block(header="Value List Content Block"),
                        cls._get_value_list_content_block(header="My Name Value List"),
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "Graph Content Block",
                    "content_blocks": [
                        cls._get_header_content_block(header="Graph Content Block"),
                        cls._get_graph_content_block(header="My Big Data Graph"),
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "String Template Content Block With Icon",
                    "content_blocks": [
                        cls._get_header_content_block(header="String Template Content Block With Icon"),
                        cls._get_string_template_content_block()
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "String Template Content Block With Tooltip",
                    "content_blocks": [
                        cls._get_header_content_block(header="String Template Content Block With Tooltip"),
                        cls._get_tooltip_string_template_content_block()
                    ]
                }),
                RenderedSectionContent(**{
                    "section_name": "Multiple Content Block Section",
                    "content_blocks": [
                        cls._get_header_content_block(header="Multiple Content Block Section"),
                        cls._get_graph_content_block(header="My col-4 Graph", col=4),
                        cls._get_graph_content_block(header="My col-4 Graph", col=4),
                        cls._get_graph_content_block(header="My col-4 Graph", col=4),
                        cls._get_table_content_block(header="My col-6 Table", col=6),
                        cls._get_bullet_list_content_block(header="My col-6 List", subheader="subheader", col=6)
                    ]
                }),
            ]
        })
../_images/customizing_data_docs.png

Dependencies

  • Font Awesome 5.10.1

  • Bootstrap 4.3.1

  • jQuery 3.2.1

  • altair 3.1.0

  • Vega 5.3.5

  • Vega-Lite 3.2.1

  • Vega-Embed 4.0.0