Ancestry.com: Managing Big Data Reaching Back to the 11th Century with Hadoop

In 2013 it will be possible to map a person’s entire genome for less than $1,000, which represents a million-fold reduction in cost from the first such process only a few years ago. Consumers attracted by this affordable price and by the growing potential of gene sequencing in areas including personalized medicine, pharmacogenomic testing and family history research will generate vast amounts of searchable, highly personal data.

Much of this new data will be sent to Ancestry.com, a Big Data company that already manages more than 11 billion records (4 petabytes) of searchable structured and unstructured data consisting of birth, death, census, military, immigration and other records. In the past 17 years, Ancestry users have created more than 47 million family trees containing more than 5 billion profiles of relatives. Added to the current mass archive, the new flood of gene-sequencing data generated by Ancestry’s recently-introduced DNA testing product will present Big Data challenges and opportunities of interest to many other companies whose business models are predicated on similar massive data troves.

In this session, Scott Sorensen, Ancestry.com CTO, will present an informative explanation of how the company is leveraging its Big Data capabilities by using Hadoop.

Specifically, Sorensen will provide two broad category overviews of how the company uses Hadoop: 1) with analytics and 2) product features. Details include:

Analytics:

• Marketing—Marketing team uses predictive modeling to customize and target messages for specific customers (e.g., if we know who is expected to cancel and we understand their behavior on the site we can target these customers with assistance specific to their needs). We also mine customer behavior data to customize our marketing messages (customers that upload a high volume of old photos will receive information about Ancestry’s photo scanning features, messages about new content collections will be tailored to specific customers based on their family trees, etc.).

• Content Investment—Content usage data is stored in Hadoop and then data models are created to help the Company make content (record) investment decisions to increase ROI.

• Product—Customer behavior data is captured in Hadoop during controlled experiments and that data is minded to provide direction for the product roadmap.

Product Features: Improving the customer experience

• Search—Ancestry.com has an unmatched genealogy search algorithm that is able to help users distinguish their “John Smith’s” records from the 70 million other “John Smith” records in the database.

• Hinting—Similar to how Amazon provides suggestions for what books you might be interested in, Ancestry.com uses record linking to serve up historical record and family tree hints to help users further family history discoveries. Machine Learning that leverages Hadoop is used to create the record linking algorithms that generate these discoveries.

• Entity Extraction— Uses natural language processing to accurately find and classify key events and relationships in unstructured text and serve it up to customers through searches and hinting.

• AncestryDNA—Science and engineering teams use Hadoop to scale genealogy algorithm GERMLINE for ethnicity prediction and cousin matching for 1700% increase in performance.