Automated bot squashing: How to build your own bot fighting infrastructure

Description

In a world with ever-growing DDoS attacks, L7 attacks give even the most experienced engineers headaches. Now imagine if instead of following easy-to-detect patterns, bots mimicked customer behavior. That’s exactly what Shopify sees every day during flash sales.

For small stores that release tiny numbers of sought-after products, those products are often resold for a huge profit, creating a situation where, for bad actors, it’s advantageous to buy as many products as quickly as possible. During flash sales, when milliseconds matter, bots buy faster than humans. These bots’ constant search for new products created a constant load on Shopify’s infrastructure and SREs—until the company decided to create an automated system to detect and block nearly all bot traffic on its load balancers.

Felix Glaser offers an overview of this system and shares the challenges Shopify faced differentiating between bots and humans. Bots act as headless browsers or browser extensions, rotate their user agent to appear as NATed users, and mimic human browsing as best they can. When the stakes are real customers unable to complete their checkouts, misclassification isn’t an option.

Join in to learn how Shopify used simple statistics, heuristics, and some reasonable thresholds to block bots on its online stores and cut down up to 50% of its traffic and developed a bot detection software that doubles as a robust DDoS protection system.