In an era of cloud computing and microservices, it’s imperative to be able to isolate clients from one another in terms of the resources they can use. Otherwise, one misconfigured or malicious client can make a service unusable for everyone else. (Steve Yegge mentions quotas and throttling as key practices in his notorious “platform rant.”)
Rate limiting—making sure a particular client sends no more than a certain number of requests per unit time—is the most common capacity management approach for API gateways, but as it turns out, this is exactly the wrong abstraction. Concurrency management is a fundamentally better concept for a variety of reasons. Jon Moore outlines the basic queuing theory behind concurrency management and shares an extension to the basic implementation that permits decentralized enforcement and an adaptive, optimistic algorithm that works well with a constantly changing mix of elastic origin capacity, population of clients, and fluctuating usage.
Jon Moore is the chief software architect at Comcast Cable, where he focuses on delivering a core set of scalable, performant, robust software components for the company’s varied software product development groups. Jon specializes in the “art of the possible,” finding ways to coordinate working solutions for complex problems and deliver them on time. He is equally comfortable leading and managing teams and personally writing production-ready code and has a passion for software engineering, continuously learning, and teaching colleagues new ways to deliver working, maintainable software with ever-higher quality and ever-shorter delivery times. His interests include distributed systems, fault tolerance, building healthy and engaging engineering cultures, and Texas Hold’em. Jon holds a PhD in computer and information science from the University of Pennsylvania. He resides in West Philadelphia, although he was neither born nor raised there and does not spend most of his days on playgrounds.
Comments on this page are now closed.
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com