Build resilient systems at scale
28–30 October 2015 • Amsterdam, The Netherlands

Building real-time metrics pipelines

9:30–11:00 Wednesday, 28/10/2015
Tutorial
Location: G102/103
Average rating: ***..
(3.68, 34 ratings)

Prerequisite Knowledge

Programming experience, preferably with Node.js or Java. In order to work with example code and tool, some knowledge of git is needed.

Materials or downloads needed in advance

Attendees will need a laptop with a recent version of Node.js installed.

Description

This talk explores how the real-time data architecture developed by AOL’s Media Platform group manages the collection and distribution of data collected from consumer devices, internal and external systems.

The first challenge is understanding what to measure, and why a data-driven approach to performance management is so important. We will examine the different types of data that can be assembled to create a meaningful picture of application health.

With the scope of the problem defined, we’ll dive into the technologies used to collect, stream, store, aggregate, and visualize application data in real time. We’ll discuss how to use tools like RabbitMQ as a high-performance data broker, and explore how to use Node.js to create lightweight, highly-focused micro-services that allow for massive distribution of aggregation and data distribution load. We’ll explore ElasticSearch’s powerful data aggregation tools and learn how to find meaningful associations in data, how to construct efficient indexes, and build effective queries.

Photo of Samantha Quiñones

Samantha Quiñones

AOL

Samantha Quiñones is a polyglot hacker and systems architecture expert. Over the course of her 17-year career, she has built software and led teams for some of the largest names in technology, and she is currently a principal software engineer at AOL. Samantha is a frequent speaker at technology conferences and participated in the White House Data Jam on STEM Workforce Quality, Flow, and Diversity. She has been recognized by the Huffington Post as one of the top Latin@s in American Media, and is a recipient of the DCFT Powerful Female Programmers Award.

Comments on this page are now closed.

Comments

Damian Jóźwiak
28/10/2015 11:19 CET

How did you measure throughput of each version of your solution?
Was it:

  • sample of productions traffic
  • real production traffic
  • synthetic test
  • something else
    ?

Did you use other ways to make sure your system won’t crash when deployed to production?