Presented By O'Reilly and Cloudera
Make Data Work
Sept 29–Oct 1, 2015 • New York, NY

How Spark is working out at Comcast scale

Sridhar Alla (Comcast), Jan Neumann (Comcast)
4:35pm–5:15pm Thursday, 10/01/2015
Spark & Beyond
Location: 1 E20 / 1 E21 Level: Intermediate
Tags: media
Average rating: ***..
(3.67, 12 ratings)

Spark is redefining the big Data ecosystem and opening doors to capabilities not available before. Comcast is moving in the direction of adopting Spark for several projects, ranging from real-time processing, data science, and large scale analytics, to the Xfinity personalization platform.

This talk will showcase some of the use cases and how we use Spark to solve many of the tough problems at Comcast scale.

1. Real time processing is a challenging topic, and historically we had to use a mixture of home-grown and off-the-shelf technologies to get a working model. We will talk about how we use Kafka to get events from over 40 million STBs, and hand over to Spark streaming which processes the events.

2. Data science is a very challenging field and more requirements pop up all the time. We are looking at Spark to build a reliable flexible framework that integrates with Hadoop, HBase, and Solr as well as provide ease of migration for data scientists who work in R and SAS.

3. Comcast provides personalized recommendations to its customers on the X1 platform. Our initial implementation was built on the Hadoop map-reduce framework using a batch computation model. When we wanted to explore how we can offer real-time recommendations, we looked to Spark because of its increased computational efficiency, and its ease in developing both streaming and batch processing solutions using the same code base.

In short, there are many fields that can profit from the plethora of capabilities provided by Spark.

Photo of Sridhar Alla

Sridhar Alla

Comcast

Sridhar Alla is a Big Data expert helping small & big companies solve complex problems such as data warehousing, governance, security, real time processing, high-frequency trading and establishing large scale data science practice. Sridhar Alla is also an agile practitioner as well as a certified agile Devops practitioner and implementer. Sridhar started his career as a Storage Software Engineer in Network Appliance, Sunnyvale then worked as the Chief Technology Officer at a cyber security firm eIQNetworks, Boston. His most recent job is that of a Director of Data Science & Engineering at Comcast, Philadelphia. Sridhar is an avid presenter at numerous Strata, Hadoop World, Spark Summit and other conferences. Sridhar also provides onsite/online training on several technologies. He also has several patents filed with the US PTO on large scale computing and distributed systems. He holds a Bachelors degree in Computer science from JNTU, Hyderabad, India and lives with his wife in New Jersey.

Photo of Jan Neumann

Jan Neumann

Comcast

Jan Neumann leads Comcast’s Applied Artificial Intelligence Research Group, which combines large-scale machine learning, deep learning, NLP, and computer vision to develop novel algorithms and product concepts that improve the experience of Comcast’s customers such as the voice interfaces, virtual assistants, and video and IoT analytics. Previously, Jan worked for Siemens Corporate Research on various computer vision-related projects, such as driver assistance systems and video surveillance. He has published over 20 papers in scientific conferences and journals and is a frequent speaker on machine learning and data science. He holds a PhD in computer science from the University of Maryland, College Park.