Profiling and Cleansing Data
Profile Data
Address Redundant, Duplicated, and Unnecessary Data
Address Missing and Null Values
Address Invalid Data
- Correcting or Removing Invalid Data
- Identifying Invalid Data
- Reasons it occurs:
- data can become invalid over time
- survey data may be invalid if a question is deemed invalid
- outliers or extreme values can sometimes indicate invalid data
- data that cannot possible be correct is invalid
- invisible characters
- non-printable characters
- leading spaces
- trailing spaces
- validate outliers
- identify with visualizations and z-scores
- Removing and Replacing Invalid Data
Convert Data to Meet Specifications
- Data Does Not Meet Specifications
- Converting Data Types