Build Systems that Drive Business
30–31 Oct 2018: Training
31 Oct–2 Nov 2018: Tutorials & Conference
London, UK

Building a distributed real-time stream processing system

A boyle (New Relic)
14:10–14:50 Friday, 2 November 2018
Distributed Systems, Systems Engineering and Architecture
 Location: Buckingham Room - Palace Suite
Secondary topics:  Systems Monitoring & Orchestration
Average rating: ***..
(3.57, 7 ratings)
Download slides (PDF)

Who is this presentation for?

  • Engineers and architects new to stream processing systems

Prerequisite knowledge

  • Basic knowledge about how to build and deploy software
  • Familiarity with containers

What you'll learn

  • Learn how to build, scale, and monitor a stream processing pipeline

Description

The future of software is distributed. If you run a backend service of consequence, you’re probably dealing with some sort of distributed system. Stream processing applications form the backbone of New Relic’s data pipeline processing billions of data points a minute. As a result, the company has learned a few useful things about building scalable distributed stream processing systems.

While there are many great tools such as Kafka and Docker orchestration upon which to build feature-rich systems, you still need to understand how these building blocks work and how to apply them effectively and reliably at scale. Amy Boyle walks you through building, scaling, and monitoring a stream processing pipeline, drawing on examples from New Relic’s data pipeline.

Topics include:

  • Data partitioning
  • Event-driven versus time-based processing
  • Dealing with node failure
  • Trade-offs on reliability, correctness, throughput, and complexity
  • Snapshotting state
  • Monitoring
Photo of A boyle

A boyle

New Relic

Amy Boyle is a senior software engineer at New Relic focusing on the core data platform. She works in distributed systems, stream processing, and lots of data.