Within a service oriented architecture, different services will expect varied guarantees from their datastores. Some services need proven technology that offers reliable transactions. Others need performant real time search, and yet others need extremely high-write throughput with tunable consistency. Which datastore a service chooses is one of the most important factors determining application flexibility, iteration speed, performance, and reliability in production.
Our session will chronicle a technical journey as Yelp experienced it, describing the problems we solved as we iterated on our SOA architecture, from zero services to hundreds and from primarily MySQL to polyglot, including Cassandra, Elasticsearch, and Zookeeper.
We will consider three major technical topics:
Complementing the technical aspects, we will discuss the organizational and cultural challenges of putting control over datastores in the hands of developers. We will first discuss how we structure our configuration management to enable developers to provision datastores without ever writing a line of Puppet, and we will cover how we isolate datastores and ensure that one service’s datastore cannot impact another.
We will then show how we do discovery and request routing with a novel combination of discovery coprocesses, service registries, and query proxies. Our infrastructure is designed so that datastore backends can appear and disappear, all without impacting user traffic or requiring application developers to change code.
Finally we will show how we customize off-the-shelf monitoring tools like Graphite and Sensu to provide easy-to-use monitoring systems for developers. These systems strive to not only notify developers when their datastore is completely unavailable, but also to provide early warning so that self-healing systems or operators can remedy issues before they become sitewide problems.
Joseph Lynch is a software engineer for Yelp who focuses on building data store and service infrastructure. Joey is a core contributor to Yelp’s data store platform, which has allowed Yelp to go from a primarily MySQL data tier to a polyglot data tier including Elasticsearch, Cassandra, and Zookeeper. He loves pushing the edge of how Yelp uses DevOps tools to automate infrastructure and never met a problem he didn’t want to automate away. When not wrangling clusters of data stores, Joey enjoys building service discovery, reliable communication, fast deployment, and monitoring into Yelp’s SOA.
Joshua Snyder is an SRE with a penchant for dealing with data. He likes designing web infrastructure that serves its functions reliably and silently. To that end, Josh is always rounding off sharp corners of infrastructure, and crafting new functionality to be as maintainable as possible. He works mostly in relational stores like MySQL and Postgres, but occasionally finds himself fiddling with Cassandra. Josh has previously spoken at MySQL Connect and Percona Live.
©2015, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org