Building a data ecosystem at Sweden's Television: Lessons and pitfalls
Who is this presentation for?
- Data engineers and architects building a scalable ecosystem for collecting and analyzing data and technical leads and product owners responsible for delivering data products within their organizations
If you had to determine which users read a news article and how many actually enjoyed it, you’ll most likely realize that these questions are very loosely defined.
At Sweden’s Television (SVT), the data team started as an effort to provide support in answering those and similar questions. The data team takes care of the whole of data infrastructure from collection to analysis. The team supports SVT in creating content that engages, entertains, and educates through its online products. Through collecting and analyzing data, it provides support for making informed decisions on both an operative and strategic level. This comes with a number of challenges.
Each of SVTs online products has a different goal, handles different types of content, and is interested in serving different users (e.g., one product delivers sports news and another delivers movies and TV shows online). These products exist across multiple platforms (web, Android, iOS, etc.), each requiring different programming languages and a different way of viewing data. These products are used by millions of users every month. And being in the service of the public means that this should be done in a way that does not violate user privacy. The analysis of the data is needed by a range of people with across the organization from authors and editors to developers and data analysts.
The nature of data architecture is that there is no one right solution. It’s often necessary to make assumptions and trade-offs to move forward. The data team learned to bridge the gap between all of SVT’s teams from editors and TV producers to developers.
To handle all of this complexity, it was necessary (but hard) to simplify the problem and start as simple as possible. Ismail Elouafiq walks you through the approach: choose what to prioritize by making the scope as simple as possible but not simpler, make a guess about what will work and decide on trade-offs when making technology choices, build the solutions to validate those guesses and have people use them, and realize what fails along the way, then understand why and reiterate again. This allowed the team to provide feedback to other teams through collecting data and having a common data model used across all of SVT (in other words, a common definition of what data points should be sent from the user-facing products).
Ismail outlines the team’s journey from zero (being a burden to all other teams) to building a data ecosystem that serves teams and makes people excited to work with it. He covers the experiments the team ran to make architectural choices and the trade-offs made along the way; how they came up with a central definition of events (independent of the products and how users interact with them), what they had to go through technically in terms of serialization formats (from JSON to Protobuf), and handling automated jobs; how these experiments made the company’s technical solutions and organizational processes fail as the team added more to its ecosystem, how it made the team see their deliveries more like “data products” as they scaled; and what the team did to transition smoothly with newer versions and handle breaking changes for the “data products” SVT offers.
What you'll learn
- Understand the experiments run to make architectural choices and the trade-offs made along the way
- Discover SVT's process in scaling up
Swedish Television (SVT)
Ismail Elouafiq is a data scientist and developer at Sweden’s Television. Ismail started as a developer by making and selling video games at a young age. He’s worked with data, machine learning, and artificial intelligence before it was popular. Ismail also writes online about data science, security, and machine learning. He also speaks five languages, and when he’s not looking at a screen, you’ll most likely find him rock climbing, freediving, or practicing partner acrobatics.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
Premier Diamond Sponsor
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires