Luigi - Batch Data Processing at Large Scale

Erik Bernhardsson (Spotify)
Sponsored Sessions
Location: E143
Average rating: ****.
(4.43, 7 ratings)
Slides:   1-PDF 

Erik will be talking about Luigi, a recently open-sourced Python framework that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows.

It also comes with Hadoop support built in (and that’s where really where its strength becomes clear). Luigi provides an infrastructure that powers several Spotify features including recommendations, top lists, A/B test analysis, external reports, internal dashboards, and many more.

Read more about it on Github.

This session is sponsored by Spotify

Erik Bernhardsson


Erik is the Engineering Manager of the Discovery team at Spotify. He is building a music recommender system using large scale machine learning algorithms, mainly matrix factorization of big matrices using Hadoop.

Previously head of the Business Intelligence team in the Stockholm office where he was responsible for collecting, aggregating and making sense out of all the data.


Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at (707) 827-7065 or

Contact Us

View a complete list of OSCON contacts