There are many considerations when building a distributed real-time architecture supporting Comcast scale. It is important that every part of the system is fault tolerant, scales horizontally, has low latency, and supports high throughput. Moreover, it needs to be designed for high impact writes that are produced from stream processing.
Comcast IP Video players report health and quality metrics to a Flume NG data collection tier. The flexibility that Flume provides through its plugin-based architecture, combined with the dynamic flow management, are key components for efficiently managing the complexities of data transformation and decoration. On the other hand, the Storm framework allows the stateless streams to participate in complex stream processing; with the ability for state awareness via Storm’s built-in field grouping. Bridging Flume to Storm is achieved with custom Flume sinks and Storm spouts. As the locations of Storm workers within the cluster are determined dynamically at runtime, intelligent discovery of Flume nodes is required by the spouts.
Although this architecture can be adapted to fit any context, it has brought Comcast a powerful real-time analytical platform. It also opens the door to many new concepts, such as the optimization of the CDN’s caching algorithm, real-time anomaly detection, and guide enhancements.
Chris Lintz is currently a Principal Engineer and team lead for the Comcast VIPER (Video IP Engineering and Research) Big Data team. His 18 years in software development and architecture span many different industries including Entertainment, Telecommunication and Defense. Prior to Comcast he was an Application team lead at OneRiot (acquired by Walmart) helping build a mobile real-time social ad platform. He holds a degree in Computer Science from the University of Colorado at Colorado Springs.
Gabriel Commeau is a principal software engineer at Comcast, working on Big Data and real-time analytics for the next generation of video players. He holds a Master’s degree in computer engineering from UTC, France. Since then, he has been working in the telecom and defense industries, both in France and in the US where he moved a decade ago. Throughout his career, he has had the opportunity to put his analytical mind in action, working on a wide variety of complex systems ranging from web conferencing to JTAG. Gabriel is a Flume contributor.
For exhibition and sponsorship opportunities, contact Susan Stewart at firstname.lastname@example.org
For information on trade opportunities with O'Reilly conferences email mediapartners
For media-related inquiries, contact Maureen Jennings at email@example.com
View a complete list of Strata + Hadoop World 2013 contacts