Engineering the Future of Software
April 10–11, 2016: Training
April 11–13, 2016: Conference
New York, NY

It probably works: Deploying probabilistic algorithms on a large scale

Tyler McMullen (Fastly)
3:50pm–4:40pm Tuesday, 04/12/2016
Distributed systems
Location: Grand Ballroom West Level: Intermediate
Average rating: ****.
(4.50, 4 ratings)

Probabilistic algorithms exist to solve problems that are either impossible or unrealistic (too expensive, too time consuming, etc.) to solve precisely. In an ideal world, you would never actually need to use probabilistic algorithms. For programmers who are not familiar with them, the concept can be positively nerve-racking: “How do I know it will actually work? What if it is inexplicably wrong? How can I debug it? Maybe we should just punt on this problem or buy a whole lot more servers. . .”

However, for those who either deeply understand probability theory or at least have used and observed the behavior of probabilistic algorithms in large-scale production environments, these algorithms are not only acceptable but also worth using at any opportunity. This is because they can help solve problems, create systems that are less expensive and more predictable, and do things that could not be done otherwise.

Tyler McMullen presents three straightforward probabilistic algorithms that have seen successful deployments at massive scale:

  • Bimodal multicast—an algorithm used for reliably sending messages to a widely-distributed network of servers
  • Locality-sensitive hashing—an algorithm used for calculating the similarity of vectors, which can be used in nearest-neighbor search, recommendation systems, and plagiarism detectors
  • HyperLogLog—an algorithm for the count-distinct problem, which can be used for estimating the number of unique items in an unbounded stream

OFFICE HOURS
Tyler will take part in Office Hours on Wednesday, April 13 at 12:15 PM in the Mercury Complex. Come by and meet him!

Photo of Tyler McMullen

Tyler McMullen

Fastly

Tyler McMullen is CTO of Fastly, where he is responsible for the system architecture and leads the company’s technology vision. As part of the founding team, Tyler built the first versions of Fastly’s instant purging system, API, and real-time analytics. Previously, Tyler worked on text analysis and recommendations at Scribd. A self-described technology curmudgeon, Tyler has experience in everything from web design to kernel development and loathes all of it. Especially distributed systems.

Comments on this page are now closed.

Comments

James Pestridge
04/18/2016 12:22pm EDT

Hi Tyler, I saw your talk and thought it was great. Could I please make arrangements with you to get the slides for review? (Sorry, I don’t have Twitter.)