Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

Lessons learned building a scalable and extendable data pipeline for Call of Duty

Yaroslav Tkachenko (Activision)
1:15pm–1:55pm Wednesday, 09/12/2018
Data engineering and architecture
Location: 1A 23/24 Level: Intermediate
Secondary topics:  Data Integration and Data Pipelines
Average rating: ****.
(4.67, 3 ratings)

Who is this presentation for?

  • Data and software engineers and architects

Prerequisite knowledge

  • A basic understanding of data pipelines and Apache Kafka

What you'll learn

  • Learn how to design and build data pipelines with Apache Kafka, how to properly name and structure your topics, what kind of clusters to use, and how to build reliable data producers


What’s easier than building a data pipeline? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse. . .wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too.

Join Yaroslav Tkachenko to learn best practices for building a data pipeline, drawn from his experience at Demonware/Activision. Yaroslav shares lessons learned about scale pipelines, not only in terms of messages per second but also in terms of supporting more games and more use cases, as well as message schemas, Apache Kafka organization and tuning, topics naming conventions, structure and routing, reliable and scalable producers and the ingestion layer, and stream processing.

Photo of Yaroslav Tkachenko

Yaroslav Tkachenko


Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, functional programming, modern cloud infrastructure, and DevOps practices. Currently, Yaroslav is a Software Architect at Activision Blizzard, working on a Data Platform for Activision games like Call of Duty franchise.

Prior to joining Activision Yaroslav held various leadership roles in multiple startups. He was responsible for designing, developing, delivering and maintaining platform services and cloud infrastructure for mission-critical systems.