Quality Data


Quality assurance (QA), in the context of data, is the process of ensuring that the data used in analysis is of a high enough quality that it gives decision makers confidence in the findings.

  • aka data cleaning

Quality data is data that has integrity and is accurate, complete, and consistent.

Attributes of Quality Data

  • data completeness
    • data is complete and all expected and required fields are entered
  • data accuracy
    • data is correct and accurate
  • data consistency
    • data is consistently entered the same way and as intended based on business rules
  • timeliness
    • data is up to date and relevant in regards to the event that has triggered it
    • available when required
    • important metric for proper decision-making
  • uniqueness
    • value is distinct and not duplicated
    • Each record should be a unique event, with no redundant entries
    • key metric for data governance

Rules and Metrics for Data Quality

  • Data governance team will determine the list of rules and metrics for quality standards
  • Rules and metrics serve as key performance indicators (KPIs) to evaluate data quality and monitor progress
  • identify metrics that can be used to set measurable goals for data quality based on the needs of an organization
  • Creating rules about the data is a key part of ensuring quality
    • can be internal to the system design or through a process
  • adding constraints to the way data is captured
    • helps increase data quality
  • use rules and constraints
  • evaluating data processes is also important
    • processes may be incomplete or not followed, leading to quality issues
  • data validation tools
    • dbt
      • open source framework
    • Great Expectations
      • Python library
    • have built-in functions and metrics that enable you to validate quality by establishing ways to measure it
    • designed to support data quality that regulated companies must adhere to maintain compliance