Data Profiling
Data profiling is determining the volume of data, the types of data and quality of the data.
- begin profiling your data prior to working with a data set
- helps you understand the data you are working with
- provides reasonable expectations about the data and valuable information for checking your work
- key way to validate and verify your information across the lifecycle of your project
- help you to identify any data quality issues so that you can correct them
- help inform what data cleansing needs to occur in order to improve the data for reporting and analysis
- involves understanding:
- the source of the data
- keys of the data
- relationships within the data
- record counts
Steps
- Identify and document the source of the data and its integrity
- Identify the field names and data types, and determine whether they are appropriate for what you are reporting
- Determine the main fields identified for reports and what their column profile entails
- Check whether a key field (primary, natural, or foreign) is represented where expected
- Recognize the total of all the data in the data set
- Identify the number of observations or record counts
Tools and Techniques
- can use manual techniques or advanced software to profile data
- most popular data tools have built-in tools to profile data
- can profile entire data set or a subset
- should profile a column of data during initial exploration
- gives basic insight into just that column of data
Data Profiling with Python
- Modern versions of Excel include the ability to execute Python code directly in the workbook
- can leverage Python to profile large data sets directly in Excel