Trill: The Crown Jewel of Microsoft’s Streaming Pipeline Explained
Who is this presentation for?Developers, Data Engineers, Enterprise Systems Engineers
The Trill data engine is the power behind many of Microsoft’s offerings, from products like Azure Stream Analytics to billion-dollar services like Bing Ads. It has now been open-sourced and is available to everyone. But it has been a long path to get here.
We begin with a history of decades of streaming data processing at Microsoft: a beginning in research, a first product in StreamInsight, the transition to the cloud, and all the pain points along the way. A key result of that lineage and learning has been the Trill engine, which has three key properties:
- A single standalone data processing engine for all temporal data, no matter if the data is streamed or stored
- A simple API that integrates seamlessly with the programming language
- Performance without ego, a willingness to use every lesson learned to improve throughput in every way possible
We then dive deep into why each of those properties is important through examples.
- A simple application to demonstrate the basics of Trill: joins, aggregation, windowing
- A more complicated application to demonstrate the power of Trill’s API: progressive windowing, regular expressions and pattern detection, data-dependent windows
- An overview of the kind of query used by Bing Ads, a query to run a multi-billion-dollar business
- Lastly, a performance showcase: running the previous examples to demonstrate how Trill got its name – processing a trillion events per day on a single node
Prerequisite knowledgeA working knowledge of streaming data systems is helpful.
What you'll learn
James is a 10-year veteran at Microsoft, having spent time on both product and research teams. He began as an intern during the last year of his PhD research at Portland State University. His background is in innovative data query and exploration interfaces and streaming data processing.
Badrish Chandramouli is a principal researcher in the database group at Microsoft Research. He is interested in creating technologies to perform real-time and offline big and raw data processing, as well as resilient state management and computation for cloud and edge applications. His research work first shipped in 2010 as part of the Microsoft SQL Server StreamInsight engine. Starting 2012, Badrish has been working on Trill, a streaming analytics engine that is widely used at Microsoft, for example, in the Bing ads platform and in the Azure Stream Analytics cloud service. More recently, Badrish built FASTER, a high-performance embedded, resilient, and concurrent state store and cache that supports larger-than-memory data and is optimized for streaming analytics. He has also worked on simplifying distributed computing via frameworks such as Ambrosia and CRA.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
View a complete list of Strata Data Conference contacts