Engineering the Future of Software
Feb 25–26, 2018: Training
Feb 26–28, 2018: Tutorials & Conference
New York, NY

Why concurrency management is better than rate limiting

Jonathan Moore (Comcast Cable)
1:15pm–2:05pm Tuesday, February 27, 2018
Secondary topics:  Best Practice, Theoretical
Average rating: ****.
(4.15, 13 ratings)

Who is this presentation for?

  • Architects, tech leads, and engineers

Prerequisite knowledge

  • A general idea of what an API gateway does and why you need one

What you'll learn

  • Understand Little's law, a useful bit of queueing theory, and rate limiting versus concurrency management in an API gateway context
  • Learn a novel algorithm for dynamic capacity management


In an era of cloud computing and microservices, it’s imperative to be able to isolate clients from one another in terms of the resources they can use. Otherwise, one misconfigured or malicious client can make a service unusable for everyone else. (Steve Yegge mentions quotas and throttling as key practices in his notorious “platform rant.”)

Rate limiting—making sure a particular client sends no more than a certain number of requests per unit time—is the most common capacity management approach for API gateways, but as it turns out, this is exactly the wrong abstraction. Concurrency management is a fundamentally better concept for a variety of reasons. Jon Moore outlines the basic queuing theory behind concurrency management and shares an extension to the basic implementation that permits decentralized enforcement and an adaptive, optimistic algorithm that works well with a constantly changing mix of elastic origin capacity, population of clients, and fluctuating usage.

Photo of Jonathan Moore

Jonathan Moore

Comcast Cable

Jon Moore is the chief software architect at Comcast Cable, where he focuses on delivering a core set of scalable, performant, robust software components for the company’s varied software product development groups. Jon specializes in the “art of the possible,” finding ways to coordinate working solutions for complex problems and deliver them on time. He is equally comfortable leading and managing teams and personally writing production-ready code and has a passion for software engineering, continuously learning, and teaching colleagues new ways to deliver working, maintainable software with ever-higher quality and ever-shorter delivery times. His interests include distributed systems, fault tolerance, building healthy and engaging engineering cultures, and Texas Hold’em. Jon holds a PhD in computer and information science from the University of Pennsylvania. He resides in West Philadelphia, although he was neither born nor raised there and does not spend most of his days on playgrounds.

Comments on this page are now closed.


03/13/2018 6:30pm EDT

Hi Jon, I very much enjoyed your session and am planning on implementing the concurrency idea.
would you please share the slide-deck?