Profiling and Cleansing Data


Profile Data

Address Redundant, Duplicated, and Unnecessary Data

Address Missing and Null Values

Address Invalid Data

  • Correcting or Removing Invalid Data
  • Identifying Invalid Data
    • Reasons it occurs:
      • data can become invalid over time
      • survey data may be invalid if a question is deemed invalid
      • outliers or extreme values can sometimes indicate invalid data
      • data that cannot possible be correct is invalid
      • invisible characters
        • non-printable characters
        • leading spaces
        • trailing spaces
    • validate outliers
      • identify with visualizations and z-scores
  • Removing and Replacing Invalid Data
    • remove or correct it

Convert Data to Meet Specifications

  • Data Does Not Meet Specifications
  • Converting Data Types