Building a real-time metrics database for trillions of points per day
Who is this presentation for?
- Senior and staff engineers, software architects, team leads, and tech leads
Datadog—a monitoring and observability platform used by thousands of companies to understand how their systems are behaving and performing—has seen engineering organizations adopt technologies such as containers and serverless functions that shorten the lifecycle of their compute infrastructure from months down to minutes—if not seconds. These organizations want to understand their applications by querying across an increasing number of dimensions, all the way down to tracking performance on an individual customer level. This has led to an explosion of metrics.
As Datadog has grown, it’s had to decide how to tier its architecture to store data in multiple formats to answer different questions related to metrics, from high-level dashboards to granular alerting queries. To determine the right price and performance trade-off for each category of query, it uses a blend of in-house, open source, and cloud services.
Joel Barciauskas details how the company uses Apache Kafka, Cassandra, object stores like S3, and in-memory databases to handle its workload. Join in to discover how Datadog’s challenges of scale, performance, cost, and data accuracy have influenced the way it structures data, and the impact this has had on the company’s architecture.
- A basic understanding of microservice patterns and data distribution through messaging systems
What you'll learn
- Discover approaches for mixing and tiering your storage, database, and data structures
Joel Barciauskas is the director of engineering at Datadog, where he leads its distribution metrics team, providing accurate, low-latency percentile measures for customers across their infrastructure. He’s an experienced engineering lead and technical manager with an extensive engineering and technical consulting background.
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires