Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

Twitter Heron at scale

Karthik Ramasamy (Twitter)
1:50pm–2:30pm Thursday, 03/31/2016
IoT and Real-time

Location: 230 C
Tags: real-time, iot
Average rating: *****
(5.00, 1 rating)

Prerequisite knowledge

Attendees should be familiar with stream processing.

Description

Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge, so Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. Karthik Ramasamy discusses Twitter’s operating experiences and shares the challenges of running Heron at scale as well as the approaches that Twitter took to solve them.

Photo of Karthik Ramasamy

Karthik Ramasamy

Twitter

Karthik Ramasamy is the engineering manager and technical lead for real-time analytics at Twitter. Karthik is the cocreator of Heron and has more than two decades of experience working in parallel databases, big data infrastructure, and networking. He cofounded Locomatix, a company that specializes in real-time stream processing on Hadoop and Cassandra using SQL, which was acquired by Twitter. Before Locomatix, he had a brief stint with Greenplum, where he worked on parallel query scheduling. Greenplum was eventually acquired by EMC for more than $300M. Prior to Greenplum, Karthik was at Juniper Networks, where he designed and delivered platforms, protocols, databases, and high-availability solutions for network routers that are widely deployed in the Internet. He is the author of several patents, publications, and one best-selling book, Network Routing: Algorithms, Protocols, and Architectures. Karthik has a PhD in computer science from UW Madison with a focus on databases, where he worked extensively in parallel database systems, query processing, scale-out technologies, storage engines, and online analytical systems. Several of these research projects were spun out as a company later acquired by Teradata.