Engineering the Future of Software
29–31 Oct 2018: Tutorials & Conference
31 Oct–1 Nov 2018: Training
London, UK

Redesigning a data platform while avoiding the pipeline jungle

Leemay Nassery (Comcast)
15:5016:40 Monday, 29 October 2018
Location: Blenheim Room - Palace Suite
Secondary topics:  Best Practice, Case Study
Average rating: ***..
(3.80, 10 ratings)

Who is this presentation for?

  • Developers and architects

Prerequisite knowledge

  • A basic understanding of data processing frameworks (e.g., Spark and Flink) and lambda functions (useful but not required)

What you'll learn

  • Explore considerations for developing and building data pipelines that have upstream dependencies such as recommendations engines


You cannot have a machine learning platform that drives content recommendations without data. You also cannot properly measure the performance of recommendations without data. Data is key to monitoring, improving, and building complex recommendations systems that impact UX experiences. However, architecting a system that properly manages this data is important to allow for downstream clients to properly use, extract, and interact with said datasets.

Leemay Nassery explains the importance of data collection pipelines and walks you through efficiently storing various datasets. Along the way, Leemay shares how Comcast migrated its recommendations platform from bare-metal Hadoop infrastructure to an event-streaming cloud platform, comparing the legacy platform to the current cloud-based system and detailing how these changes improved the reliability, stability, and results generated by the respective downstream consumers like the company’s machine learning tier. Join in to learn how to avoid building a data pipeline jungle and ensure that the overall architecture meets the needs of your downstream data consumers.

Photo of Leemay Nassery

Leemay Nassery


Leemay Nassery is a senior engineer leading the recommendations and targeting engineering efforts at Comcast. She also sets the strategic direction for content personalization for Comcast’s Xfinity consumer-facing video products and leads efforts with A/B testing, testing and targeting, and producing the metrics to measure successful customer outcomes.