Engineering the Future of Software
29–31 Oct 2018: Tutorials & Conference
31 Oct–1 Nov 2018: Training
London, UK

Redesigning a Data Platform While Avoiding The Pipeline Jungle

Leemay Nassery (Comcast)

Who is this presentation for?

developers or architects who have an interest in building extensible data platform

Prerequisite knowledge

Very basic understanding of data processing frameworks like spark, flink, lambda functions; although that's not necessarily required but it would help

What you'll learn

Better understanding of what should be considered when developing or building data pipelines that have upstream dependencies such as recommendations engine,

Description

You cannot have a machine learning platform that drives content recommendations without data. You also cannot properly measure the performance of recommendations without data.

Data is key to monitoring, improving and building complex recommendations systems that impact UX experiences. However, architecting a system that properly manages this data is important to allow for downstream clients to properly use, extract and interact with said datasets. In this talk we’ll walk through the importance of data collection pipelines and the importance of efficiently storing various datasets by describing how we migrated our recommendations platform from the bare metal Hadoop infrastructure to an event streaming cloud platform.

We’ll discuss in detail comparing the legacy platform to the current cloud base system and how these changes increased the reliability, stability and in general improved the of the results generated by the respective downstream consumers like our machine learning tier. Key takeaways are how to avoid building a data pipeline jungle and went to take a step back to reevaluate the overall architecture meets the needs of the downstream data consumers.

Photo of Leemay Nassery

Leemay Nassery

Comcast

Leemay leads the Recommendations and Targeting engineering efforts at Comcast, and sets the strategic direction for Content Personalization for Comcast’s Xfinity consumer facing video products. Leemay also leads efforts with A/B testing, Testing and Targeting, and producing the metrics to measure successful customer outcomes.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)