Fuzzy matching and deduplicating data: Techniques for advanced data prep

Description

Machine learning transforms can enable you to identify duplicate or linked records in your dataset, even when the records don’t have a common unique identifier and no fields match exactly. Machine learning transforms can help you with the following problems:

• Linking customer records across different customer databases, even when many customer fields don’t match exactly across the databases (e.g. different name spelling, address differences)
• Matching external product lists against your product catalog, such as lists of hazardous goods or lists of goods that can’t be transported by air
• Deduplicating customer accounts, when the same person makes multiple registrations

Nikki Rouda and Janisha Anand explain how they used machine learning transforms to find matching records from Abt.com’s and Buy.com’s lists of consumer electronic products from the Abt-Buy dataset. These lists include significant overlap in products, but neither IDs nor most names or descriptions match exactly.