Data Profiling


Data profiling is determining the volume of data, the types of data and quality of the data.

  • begin profiling your data prior to working with a data set
  • helps you understand the data you are working with
  • provides reasonable expectations about the data and valuable information for checking your work
  • key way to validate and verify your information across the lifecycle of your project
  • help you to identify any data quality issues so that you can correct them
  • help inform what data cleansing needs to occur in order to improve the data for reporting and analysis
  • involves understanding:
    • the source of the data
    • keys of the data
    • relationships within the data
    • record counts

Steps

  1. Identify and document the source of the data and its integrity
  2. Identify the field names and data types, and determine whether they are appropriate for what you are reporting
  3. Determine the main fields identified for reports and what their column profile entails
  4. Check whether a key field (primary, natural, or foreign) is represented where expected
  5. Recognize the total of all the data in the data set
  6. Identify the number of observations or record counts

Tools and Techniques

  • can use manual techniques or advanced software to profile data
  • most popular data tools have built-in tools to profile data
  • can profile entire data set or a subset
  • should profile a column of data during initial exploration
    • gives basic insight into just that column of data

Data Profiling with Python

  • Modern versions of Excel include the ability to execute Python code directly in the workbook
  • can leverage Python to profile large data sets directly in Excel