February 23–26, 2020
Please log in
Please log in

Building a real-time metrics database for trillions of points per day

Joel Barciauskas (Datadog)
3:50pm4:40pm Tuesday, February 25, 2020
Location: Beekman Parlor
Secondary topics:  Case Study
Average rating: **...
(2.00, 3 ratings)

Who is this presentation for?

  • Senior and staff engineers, software architects, team leads, and tech leads




Datadog—a monitoring and observability platform used by thousands of companies to understand how their systems are behaving and performing—has seen engineering organizations adopt technologies such as containers and serverless functions that shorten the lifecycle of their compute infrastructure from months down to minutes—if not seconds. These organizations want to understand their applications by querying across an increasing number of dimensions, all the way down to tracking performance on an individual customer level. This has led to an explosion of metrics.

As Datadog has grown, it’s had to decide how to tier its architecture to store data in multiple formats to answer different questions related to metrics, from high-level dashboards to granular alerting queries. To determine the right price and performance trade-off for each category of query, it uses a blend of in-house, open source, and cloud services.

Joel Barciauskas details how the company uses Apache Kafka, Cassandra, object stores like S3, and in-memory databases to handle its workload. Join in to discover how Datadog’s challenges of scale, performance, cost, and data accuracy have influenced the way it structures data, and the impact this has had on the company’s architecture.

Prerequisite knowledge

  • A basic understanding of microservice patterns and data distribution through messaging systems

What you'll learn

  • Discover approaches for mixing and tiering your storage, database, and data structures
Photo of Joel Barciauskas

Joel Barciauskas


Joel Barciauskas is the director of engineering at Datadog, where he leads its distribution metrics team, providing accurate, low-latency percentile measures for customers across their infrastructure. He’s an experienced engineering lead and technical manager with an extensive engineering and technical consulting background.

  • IBM
  • LaunchDarkly
  • LightStep
  • Red Hat
  • ThoughtWorks
  • Auth0
  • Check Point Software
  • Contentful
  • Contrast Security
  • Datadog
  • Diamanti
  • Octobot.io
  • Optimizely
  • Perforce
  • Robin.io
  • SmartBear
  • Tidelift
  • WhiteSource
  • Synopsys
  • AxonIQ
  • Codefresh
  • CodeStream
  • Hello2morrow
  • LogRocket
  • Rookout
  • Solo.io
  • CNN
  • Boundless Notions, LLC

Contact us


For conference registration information and customer service


For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference


For media/analyst press inquires