Presented By O'Reilly and Cloudera
Make Data Work
31 May–1 June 2016: Training
1 June–3 June 2016: Conference
London, UK

Processing billions of events in real time with Heron

Karthik Ramasamy (Twitter)
14:55–15:35 Friday, 3/06/2016
IoT & real-time
Location: Capital Suite 13 Level: Intermediate
Average rating: ***..
(3.00, 3 ratings)

Prerequisite knowledge

Attendees should have an understanding of real-time processing.

Description

Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge, so Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly two years and is widely used by several teams for diverse use cases. Karthik Ramasamy describes Heron in detail, covering a few use cases in-depth and sharing the operating experiences and challenges of running Heron at scale.

Photo of Karthik Ramasamy

Karthik Ramasamy

Twitter

Karthik Ramasamy is the engineering manager and technical lead for real-time analytics at Twitter. Karthik is the cocreator of Heron and has more than two decades of experience working in parallel databases, big data infrastructure, and networking. He cofounded Locomatix, a company that specializes in real-time stream processing on Hadoop and Cassandra using SQL, which was acquired by Twitter. Before Locomatix, he had a brief stint with Greenplum, where he worked on parallel query scheduling. Greenplum was eventually acquired by EMC for more than $300M. Prior to Greenplum, Karthik was at Juniper Networks, where he designed and delivered platforms, protocols, databases, and high-availability solutions for network routers that are widely deployed in the Internet. He is the author of several patents, publications, and one best-selling book, Network Routing: Algorithms, Protocols, and Architectures. Karthik has a PhD in computer science from UW Madison with a focus on databases, where he worked extensively in parallel database systems, query processing, scale-out technologies, storage engines, and online analytical systems. Several of these research projects were spun out as a company later acquired by Teradata.