Build resilient systems at scale
October 12–14, 2015 • New York, NY

Challenging fundamental assumptions of datacenters: Decoupling infrastructure from hardware

Thorvald Natvig (Medallia, Inc.)
3:40pm–4:20pm Wednesday, 10/14/2015
Location: Regent Parlor
Average rating: ***..
(3.92, 12 ratings)
Slides:   1-PDF 

Prerequisite Knowledge

Basic understanding of networks (basic IP routing), storage (block devices), and containers (Docker).


When we were switching to a microservice architecture for our application, we needed a larger and more automated datacenter to run it on. When we looked at existing datacenter designs that would give us the availability and redundancy we needed, costs rose quickly and performance dropped. Why is it we couldn’t take our little three-server test cluster and get linear performance and capacity increase without an exponential increase in cost?

Asking this question led us down the path of revisiting a number of assumptions:

  • You need a flat network to do IP mobility, or you need to tie yourself to a specific vendor’s overlay technology.
  • You need very specialized knowledge to configure networking.
  • You must have an expensive storage appliance to get resilient storage at the cost of performance.
  • You still need a large number of “special” boxes; your jump-host, your management box, your DNS servers, your DHCP/TFTP servers; and all of these boxes are maintained separately from your “regular” boxes.

It turns out none of these assumptions are true anymore. This enabled us to build a datacenter that is very different from our historical ones:

  • Switches run Cumulus — they are Linux servers with 32 network cards.
  • Our storage “cluster” consists of Linux servers running CEPH; each server has SSDs for data with NVDIMM journals.
  • Our compute nodes are Linux servers with Docker, using Mesos and Aurora for orchestration.
  • The entire network is pure IP; every network link is a separate /30 subnet.
  • Every switch and every server runs OSPF.

By realizing that switches and servers are the same, and work better together than they work apart, you can with minimal modification design a datacenter architecture that leverages the full programmability of modern hardware — bringing the principles of software, with rapid and agile development and change, to the hardware world. In fact, the only modification we have done is to Docker; the ability to have a dedicated IP address for every container, and having it store data volumes on CEPH.

This gives us some amazing benefits:

  • You can run containers with high performance networking and storage on commodity hardware and software, and not worry about infrastructure failures.
  • The overall design is simple, with few components, making it easy to understand and therefore easy to operate and maintain.
  • The services can relocate, both in case of hardware failure or in the case of hardware maintenance, and when relocated, maintain both their dedicated IP address and their bindings to storage volumes.
  • On a 40GbE network, performance for storage is on par with local all-SSD storage.
  • With a non-blocking leaf/spine design, there is no network locality, and there is no need to worry about where in the datacenter the service runs; performance is identical from all locations.
  • This allows running services like ZooKeeper, DNS servers, databases, and Mesos masters in Docker containers. We even run our load balancers as a relocatable service; as the IP address follows the service, we do not need to update the external-facing firewall when the service relocates.
  • This means we have no special servers; every service runs on standardized infrastructure.

This session will share everything we’ve learned, and cover everything that you need to replicate what we’ve done. We can no longer imagine a world without this, and we want more people to enjoy the amazing benefits we now take for granted.

Photo of Thorvald Natvig

Thorvald Natvig

Medallia, Inc.

Thorvald Natvig is an architect at Medallia, working on a wide range of infrastructure and core application components, from network and server design to writing in-memory databases. He evangelizes synergistic designs, where the whole is greater than the sum of its parts. Thorvald is an open source enthusiast, most known for creating the VoIP tool Mumble. Before joining Medallia, he got a PhD in high performance computing from NTNU in Norway.

Comments on this page are now closed.


10/15/2015 11:36am EDT

In my opinion, by far the best breakout of the day! Identified his problems, presented possible solutions, and ultimately gave reasons for why he chose what he did from an objective standpoint. One of the very few presenters who did not show outright bias for a specific product or technology.

Stay Connected

Follow Velocity on Twitter Facebook Group Google+ LinkedIn Group


More Videos »

O’Reilly Media

Tech insight, analysis, and research