Engineering the Future of Software
29–31 Oct 2018: Tutorials & Conference
31 Oct–1 Nov 2018: Training
London, UK

Redesigning a data platform while avoiding the pipeline jungle

Leemay Nassery (Comcast)
15:5016:40 Monday, 29 October 2018
Fundamentals
Location: Blenheim Room - Palace Suite Level: Beginner
Secondary topics:  Best Practice, Case Study

Who is this presentation for?

  • Developers and architects

Prerequisite knowledge

  • A basic understanding of data processing frameworks (e.g., Spark and Flink) and lambda functions (useful but not required)

What you'll learn

  • Explore considerations for developing and building data pipelines that have upstream dependencies such as recommendations engines

Description

You cannot have a machine learning platform that drives content recommendations without data. You also cannot properly measure the performance of recommendations without data. Data is key to monitoring, improving, and building complex recommendations systems that impact UX experiences. However, architecting a system that properly manages this data is important to allow for downstream clients to properly use, extract, and interact with said datasets.

Leemay Nassery explains the importance of data collection pipelines and walks you through efficiently storing various datasets with the intention of avoiding the “pipeline jungle” construct by thinking holistically about the data and the tiers that follow the initial consumption of these events. Along the way, Leemay shares how Comcast migrated its recommendations platform from bare-metal Hadoop infrastructure to an event-streaming cloud platform, comparing the legacy platform to the current cloud-based system and detailing how these changes improved the reliability, stability, and results generated by the respective downstream consumers like the company’s machine learning tier. Join in to learn how to avoid building a data pipeline jungle and ensure that the overall architecture meets the needs of your downstream data consumers.

Photo of Leemay Nassery

Leemay Nassery

Comcast

Leemay leads the Recommendations and Targeting engineering efforts at Comcast, and sets the strategic direction for Content Personalization for Comcast’s Xfinity consumer facing video products. Leemay also leads efforts with A/B testing, Testing and Targeting, and producing the metrics to measure successful customer outcomes.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)