Building a data ecosystem at Sweden's Television: Lessons and pitfalls

Description

If you had to determine which users read a news article and how many actually enjoyed it, you’ll most likely realize that these questions are very loosely defined.

At Sweden’s Television (SVT), the data team started as an effort to provide support in answering those and similar questions. The data team takes care of the whole of data infrastructure from collection to analysis. The team supports SVT in creating content that engages, entertains, and educates through its online products. Through collecting and analyzing data, it provides support for making informed decisions on both an operative and strategic level. This comes with a number of challenges.

Each of SVTs online products has a different goal, handles different types of content, and is interested in serving different users (e.g., one product delivers sports news and another delivers movies and TV shows online). These products exist across multiple platforms (web, Android, iOS, etc.), each requiring different programming languages and a different way of viewing data. These products are used by millions of users every month. And being in the service of the public means that this should be done in a way that does not violate user privacy. The analysis of the data is needed by a range of people with across the organization from authors and editors to developers and data analysts.

The nature of data architecture is that there is no one right solution. It’s often necessary to make assumptions and trade-offs to move forward. The data team learned to bridge the gap between all of SVT’s teams from editors and TV producers to developers.

To handle all of this complexity, it was necessary (but hard) to simplify the problem and start as simple as possible. Ismail Elouafiq walks you through the approach: choose what to prioritize by making the scope as simple as possible but not simpler, make a guess about what will work and decide on trade-offs when making technology choices, build the solutions to validate those guesses and have people use them, and realize what fails along the way, then understand why and reiterate again. This allowed the team to provide feedback to other teams through collecting data and having a common data model used across all of SVT (in other words, a common definition of what data points should be sent from the user-facing products).

Ismail outlines the team’s journey from zero (being a burden to all other teams) to building a data ecosystem that serves teams and makes people excited to work with it. He covers the experiments the team ran to make architectural choices and the trade-offs made along the way; how they came up with a central definition of events (independent of the products and how users interact with them), what they had to go through technically in terms of serialization formats (from JSON to Protobuf), and handling automated jobs; how these experiments made the company’s technical solutions and organizational processes fail as the team added more to its ecosystem, how it made the team see their deliveries more like “data products” as they scaled; and what the team did to transition smoothly with newer versions and handle breaking changes for the “data products” SVT offers.