Quality Data
Quality assurance (QA), in the context of data, is the process of ensuring that the data used in analysis is of a high enough quality that it gives decision makers confidence in the findings.
- aka data cleaning
Quality data is data that has integrity and is accurate, complete, and consistent.
Attributes of Quality Data
- data completeness
- data is complete and all expected and required fields are entered
- data accuracy
- data is correct and accurate
- data consistency
- data is consistently entered the same way and as intended based on business rules
- timeliness
- data is up to date and relevant in regards to the event that has triggered it
- available when required
- important metric for proper decision-making
- uniqueness
- value is distinct and not duplicated
- Each record should be a unique event, with no redundant entries
- key metric for data governance
Rules and Metrics for Data Quality
- Data governance team will determine the list of rules and metrics for quality standards
- Rules and metrics serve as key performance indicators (KPIs) to evaluate data quality and monitor progress
- identify metrics that can be used to set measurable goals for data quality based on the needs of an organization
- Creating rules about the data is a key part of ensuring quality
- can be internal to the system design or through a process
- adding constraints to the way data is captured
- helps increase data quality
- use rules and constraints
- evaluating data processes is also important
- processes may be incomplete or not followed, leading to quality issues
- data validation tools
dbt- open source framework
- Great Expectations
- Python library
- have built-in functions and metrics that enable you to validate quality by establishing ways to measure it
- designed to support data quality that regulated companies must adhere to maintain compliance