Sep 23–26, 2019

Trill: The Crown Jewel of Microsoft’s Streaming Pipeline Explained

James Terwilliger (Microsoft Corporation), Badrish Chandramouli (Microsoft Research), Jonathan Goldstein (Microsoft Research)
4:35pm5:15pm Wednesday, September 25, 2019
Location: 1A 15/16
Secondary topics:  Cloud Platforms and SaaS, Data Integration and Data Processing, Media and Advertising, Streaming and IoT

Who is this presentation for?

Developers, Data Engineers, Enterprise Systems Engineers

Level

Intermediate

Description

The Trill data engine is the power behind many of Microsoft’s offerings, from products like Azure Stream Analytics to billion-dollar services like Bing Ads. It has now been open-sourced and is available to everyone. But it has been a long path to get here.

We begin with a history of decades of streaming data processing at Microsoft: a beginning in research, a first product in StreamInsight, the transition to the cloud, and all the pain points along the way. A key result of that lineage and learning has been the Trill engine, which has three key properties:
- A single standalone data processing engine for all temporal data, no matter if the data is streamed or stored
- A simple API that integrates seamlessly with the programming language
- Performance without ego, a willingness to use every lesson learned to improve throughput in every way possible

We then dive deep into why each of those properties is important through examples.
- A simple application to demonstrate the basics of Trill: joins, aggregation, windowing
- A more complicated application to demonstrate the power of Trill’s API: progressive windowing, regular expressions and pattern detection, data-dependent windows
- An overview of the kind of query used by Bing Ads, a query to run a multi-billion-dollar business
- Lastly, a performance showcase: running the previous examples to demonstrate how Trill got its name – processing a trillion events per day on a single node

Prerequisite knowledge

A working knowledge of streaming data systems is helpful.

What you'll learn

Temporal data versus temporal queries Data-dependent and custom temporal windowing
Photo of James Terwilliger

James Terwilliger

Microsoft Corporation

James is a 10-year veteran at Microsoft, having spent time on both product and research teams. He began as an intern during the last year of his PhD research at Portland State University. His background is in innovative data query and exploration interfaces and streaming data processing.

Badrish Chandramouli

Microsoft Research

Badrish Chandramouli is a principal researcher in the database group at Microsoft Research. He is interested in creating technologies to perform real-time and offline big and raw data processing, as well as resilient state management and computation for cloud and edge applications. His research work first shipped in 2010 as part of the Microsoft SQL Server StreamInsight engine. Starting 2012, Badrish has been working on Trill, a streaming analytics engine that is widely used at Microsoft, for example, in the Bing ads platform and in the Azure Stream Analytics cloud service. More recently, Badrish built FASTER, a high-performance embedded, resilient, and concurrent state store and cache that supports larger-than-memory data and is optimized for streaming analytics. He has also worked on simplifying distributed computing via frameworks such as Ambrosia and CRA.

Jonathan Goldstein

Microsoft Research

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

strataconf@oreilly.com

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of Strata Data Conference contacts