Twitter users start and join multiple conversations every day, so monitoring and mitigating any potential network issue in real time is the key to a smooth user experience. Monitoring and analyzing a network at Twitter scale is extremely hard—and almost impossible to achieve manually. As a result, the company needed to create dedicated data pipelines to monitor network activity, detect potential issues, and generate network usage reports.
Julien Delange and Neng Lu explain how Twitter uses the Heron stream processing engine to monitor and analyze its network infrastructure. Within two months, engineers without intensive data science knowledge implemented a new data pipeline that ingests large-scale network events and reports networking issue and network equipment utilization. This pipeline ingests multiple data sources and processes about 1 billion tuples per day to detect network issues and generate usage statistics. As of today, this data pipeline is deployed in production and helps network engineers to detect issues, troubleshoot network usage, and manage network capacity.
Join Julien and Neng for an overview of the Heron project. Along the way, you’ll learn the challenges Twitter faced when building it and the key technologies it used to overcome those challenges and achieve its scalability, low-maintenance, and low-latency goals.
Julien Delange is a staff software engineer at Twitter working on infrastructure services. Previously, he was a senior software engineer at Amazon Web Services, a senior member of the technical staff at Carnegie Mellon University, and a software engineer at the European Space Agency. Julien holds a PhD in computer science from Télécom ParisTec and a master’s degree in computer science from Université Pierre-et-Marie-Curie.
Neng Lu is a software engineer at Twitter, where he is the core member of Twitter’s real-time compute team and the core committer to the Apache Heron project (incubating). He has a broad interest in distributed systems and real-time analytics and has worked on Twitter’s key-value storage system, Manhattan; its monitoring system, Cuckoo; and its real-time processing system, Heron. He holds an MS in CS from UCLA and a bachelor’s degree in CS from Zhejiang University.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org