The process of identifying and removing duplicate records from a dataset to ensure each entity is represented only once.
Data deduplication (also called deduping) is a specific data quality task focused on eliminating redundant records. It is more targeted than general data cleansing. Deduplication can be exact (removing records that are identical in all fields) or fuzzy (identifying records that represent the same entity despite minor differences in spelling, formatting, or data entry).
Fuzzy deduplication uses techniques like string similarity matching, phonetic matching (Soundex, Metaphone), and machine learning to identify probable duplicates that are not exact copies.
Example: A national address registry compiled from provincial sources contains 2.3 million records. After exact deduplication, 180,000 exact duplicates are removed. Fuzzy deduplication then identifies an additional 45,000 probable duplicates — the same address appearing with slight variations ("Ave" vs "Avenue", "St" vs "Street"). Human review confirms and merges the fuzzy matches, resulting in a cleaner, more reliable registry.