Real-time monitoring of Twitter's network infrastructure with Heron

Description

Twitter users start and join multiple conversations every day, so monitoring and mitigating any potential network issue in real time is the key to a smooth user experience. Monitoring and analyzing a network at Twitter scale is extremely hard—and almost impossible to achieve manually. As a result, the company needed to create dedicated data pipelines to monitor network activity, detect potential issues, and generate network usage reports.

Julien Delange and Neng Lu explain how Twitter uses the Heron stream processing engine to monitor and analyze its network infrastructure. Within two months, engineers without intensive data science knowledge implemented a new data pipeline that ingests large-scale network events and reports networking issue and network equipment utilization. This pipeline ingests multiple data sources and processes about 1 billion tuples per day to detect network issues and generate usage statistics. As of today, this data pipeline is deployed in production and helps network engineers to detect issues, troubleshoot network usage, and manage network capacity.

Join Julien and Neng for an overview of the Heron project. Along the way, you’ll learn the challenges Twitter faced when building it and the key technologies it used to overcome those challenges and achieve its scalability, low-maintenance, and low-latency goals.