Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

Lessons learned building a scalable and extendable data pipeline for Call of Duty

Yaroslav Tkachenko (Activision)
1:15pm–1:55pm Wednesday, 09/12/2018
Data engineering and architecture
Location: 1A 23/24 Level: Intermediate
Secondary topics:  Data Integration and Data Pipelines
Average rating: ****.
(4.67, 3 ratings)

Who is this presentation for?

  • Data and software engineers and architects

Prerequisite knowledge

  • A basic understanding of data pipelines and Apache Kafka

What you'll learn

  • Learn how to design and build data pipelines with Apache Kafka, how to properly name and structure your topics, what kind of clusters to use, and how to build reliable data producers


What’s easier than building a data pipeline? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse. . .wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too.

Join Yaroslav Tkachenko to learn best practices for building a data pipeline, drawn from his experience at Demonware/Activision. Yaroslav shares lessons learned about scale pipelines, not only in terms of messages per second but also in terms of supporting more games and more use cases, as well as message schemas, Apache Kafka organization and tuning, topics naming conventions, structure and routing, reliable and scalable producers and the ingestion layer, and stream processing.

Photo of Yaroslav Tkachenko

Yaroslav Tkachenko


Yaroslav Tkachenko is a senior data engineer at Activision, where he is working on a large-scale data pipeline. Yaroslav is interested in distributed systems, microservices, functional programming, modern cloud infrastructure, and DevOps practices. Previously, Yaroslav held various leadership roles in multiple startups, where he was responsible for designing, developing, delivering, and maintaining platform services and cloud infrastructure for mission-critical systems.