Twitter is all about real-time data at scale. Twitter’s data centers continuously process billions of events per day the instant the data is generated. To achieve real-time performance, Twitter has developed and deployed Heron, a next-generation cloud streaming engine. Heron provides unparalleled performance at large scale and has been successfully meeting Twitter’s strict performance requirements for various streaming applications.
Heron, in production at Twitter for more than two and a half years, has proven to be scalable and reliable and is now an open source project with contributors from various institutions. However, until now, Heron has not been optimized from a performance perspective. (The performance numbers reported in the Twitter Heron paper, published in SIGMOD 2015, were without any optimizations.)
Sanjeev Kulkarni and Maosong Fu share several optimizations implemented in Heron to improve throughput by 5x and reduce latency by 50–60%, describing in detail how they identified optimization opportunities with detailed profiling that indicated several issues, including multiple serializations/deserialization, eager serialization/deserialization, and immutable design. Based on these observations, Sanjeev and Maosong came up with several techniques to mitigate these costs. Along the way, Sanjeev and Maosong show how certain parameters, such as max spout pending and cache drain frequency, affect throughput and latency, and how a careful choice of these parameters can achieve latencies as low as 12 ms.
Sanjeev Kulkarni is the cofounder of Streamlio, a company focused on building a next-generation real-time stack. Previously, he was the technical lead for real-time analytics at Twitter, where he cocreated Twitter Heron; worked at Locomatix handling the company’s engineering stack; and led several initiatives for the AdSense team at Google. Sanjeev holds an MS in computer science from the University of Wisconsin-Madison.
Maosong Fu is the technical lead for Heron and real-time analytics at Twitter and the author of few publications in the distributed area. Maosong holds a master’s degree from Carnegie Mellon University and bachelor’s from Huazhong University of Science and Technology.
©2017, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org