Skip to main content

Moving VividCortex to the Log-Backed Infrastructure

John Berryman (Eventbrite), Baron Schwartz (VividCortex)
Average rating: ****.
(4.29, 7 ratings)

Big Data is all the buzz and it will continue to be so for quite some time. But when we talk about Big Data, the conversation is usually about how to store the data (e.g. HDFS, Amazon S3, Cassandra) or how to ask interesting questions of the data (e.g. MapReduce, Hive, Cascading). In these conversations, Big Data is considered merely a component of business. But for many companies these days, Big Data is the business. And managing the crushing volume and velocity of data requires Big Data planning from the very foundations of the infrastructure.

As VividCortex approached our April 2nd debut release we were already feeling mounting data and infrastructure pressures. At about this same time, the deep-reaching implications of Jay Krep’s Unifying Log article ( was permeating through the IT community. In this article, Kreps outlines a radically new approach to data infrastructure in which the data log serves as the backbone of the IT infrastructure. In this new mindset, search engines, and even databases are thought of as merely indexes and aggregators over the data that stream though an infrastructure.

The true system of record is the Log itself. As a company embraces this understanding, infrastructure can be simplified from a web of APIs, and queues, to a clean set of publishers and subscribers that all connect the singular Log — the backbone of the infrastructure. Almost as a byproduct of adopting the Log structure, the addition and removal of functionality become straightforward, catastrophe recovery becomes straightforward, and the infrastructure becomes robust to unpredictable, spikey loading.

In this fast paced talk we will cover the details or our interpretation of the Log-centered infrastructure. We’ll explain where VividCortex was prior to adopting the Log and we will discuss the strategy we took to move our production systems to the Log. Attendees will leave with sufficient knowledge of the Log infrastructure to determine whether or not they can apply it to the infrastructure at their own company.

Photo of John Berryman

John Berryman


John’s first career was as an aerospace engineer, but after several years in aerospace he found that he most loved his job when he was either programming or working on a good math problem. Eventually John cut out the aircraft and satellites and started working full-time with software development, infrastructure architecture, and data science. These days John works at VividCortex building best-of-breed server performance monitoring tools.

Photo of Baron Schwartz

Baron Schwartz


Baron Schwartz is the founder and CTO of VividCortex, the best way to see what your production database servers are doing. Baron has written a lot of open source software and several books, including High Performance MySQL. He’s focused his career on learning and teaching about performance and observability of systems generally, including the view that teams are systems and culture influences their performance, and databases specifically.

Comments on this page are now closed.


Picture of John Berryman
John Berryman
07/14/2014 9:40am EDT

The talk has yet to be written. What are you guys interested in hearing about?