Building a real-time metrics database for trillions of points per day





Who is this presentation for?
- Senior and staff engineers, software architects, team leads, and tech leads
Level
Description
Datadog—a monitoring and observability platform used by thousands of companies to understand how their systems are behaving and performing—has seen engineering organizations adopt technologies such as containers and serverless functions that shorten the lifecycle of their compute infrastructure from months down to minutes—if not seconds. These organizations want to understand their applications by querying across an increasing number of dimensions, all the way down to tracking performance on an individual customer level. This has led to an explosion of metrics.
As Datadog has grown, it’s had to decide how to tier its architecture to store data in multiple formats to answer different questions related to metrics, from high-level dashboards to granular alerting queries. To determine the right price and performance trade-off for each category of query, it uses a blend of in-house, open source, and cloud services.
Joel Barciauskas details how the company uses Apache Kafka, Cassandra, object stores like S3, and in-memory databases to handle its workload. Join in to discover how Datadog’s challenges of scale, performance, cost, and data accuracy have influenced the way it structures data, and the impact this has had on the company’s architecture.
Prerequisite knowledge
- A basic understanding of microservice patterns and data distribution through messaging systems
What you'll learn
- Discover approaches for mixing and tiering your storage, database, and data structures

Joel Barciauskas
Datadog
Joel Barciauskas is the director of engineering at Datadog, where he leads its distribution metrics team, providing accurate, low-latency percentile measures for customers across their infrastructure. He’s an experienced engineering lead and technical manager with an extensive engineering and technical consulting background.
Platinum Sponsor
Gold Sponsors
Silver Sponsors
Exhibitor
Innovators
Supporting
Community Partner
Contact us
confreg@oreilly.com
For conference registration information and customer service
partners@oreilly.com
For more information on community discounts and trade opportunities with O’Reilly conferences
Become a sponsor
For information on exhibiting or sponsoring a conference
pr@oreilly.com
For media/analyst press inquires