Missing values are one of the most common data quality problems. A blank cell, a null value, or a placeholder like "N/A" or "999" all represent missing data — and they need to be handled carefully.
Types of Missingness
Missing Completely at Random (MCAR) — the missing values have no pattern. A sensor that randomly fails to record occasionally produces MCAR data.
Missing at Random (MAR) — the missingness is related to other variables but not to the missing value itself. Survey respondents who skip income questions may do so based on age or education, not income level.
Missing Not at Random (MNAR) — the missingness is related to the value itself. High earners may be more likely to skip income questions precisely because their income is high. This is the hardest type to handle.
How to Identify Missing Values
Start by profiling your dataset: count the number of missing values in each column, calculate the percentage missing, and look for patterns. Are certain columns always missing together? Are missing values concentrated in certain time periods or geographic areas?
Options for Handling Missing Values
- Leave them — if the missingness is acceptable for your use case and you can handle nulls in your analysis
- Remove rows — if the missing values are few and random, dropping incomplete records may be acceptable
- Impute — fill in missing values using a rule (mean, median, mode) or a model
- Flag — add a separate column indicating that a value was missing, preserving the information about missingness
Imputing missing values introduces assumptions. Replacing missing ages with the average age changes the distribution of your data. Always document what you imputed and why, and consider whether imputation is appropriate for your use case.
Learn about Duplicate Records.