Making Open Work
May 8–9, 2017: Training & Tutorials
May 10–11, 2017: Conference
Austin, TX

Scaling massive, real-time data pipelines with Go

Jean de Klerk (Pivotal)
11:50am12:30pm Thursday, May 11, 2017
Infrastructure, Performance
Location: Meeting Room 9 A/B
Level: Intermediate
Average rating: ***..
(3.50, 4 ratings)

Who is this presentation for?

  • Software engineers

Prerequisite knowledge

  • A good understanding of concurrency
  • Go knowledge not required

What you'll learn

  • Learn how to receive, parse, and store huge amounts of data with high availability, low-to-zero data drops, and low latency


Passing 10,000 messages a second between apps and data stores is tough. Add on filtering and parsing and you’re bound to run into problems. Pivotal Cloud Foundry handles several factors of that load each day, which means its programs need to be highly concurrent and very careful with state.

Jean de Klerk explains what it takes to pipe, parse, filter, and store 10,000 messages a second with Go, sharing some patterns and best practices that can be taken advantage of in order to handle this load.

Photo of Jean de Klerk

Jean de Klerk


Jean de Klerk is a consultant and open source evangelist at Pivotal helping companies achieve success through the agile process, moving to cloud-ready apps, and enabling developers on technologies. Jean also works on CloudFoundry, Pivotal’s open source PaaS, building highly concurrent microservices in Go. Outside of work, Jean contributes to open source libraries, tinkers with Arduinos and Raspberry Pis, and enjoys the Colorado outdoors.