Mar 15–18, 2020

BEAST: Building an event processing library to handle millions of events

Maulik Soneji (Gojek), Dinesh Kumar (Gojek)
11:00am11:40am Wednesday, March 18, 2020
Location: LL21B
Secondary topics:  Data Management and Storage

Who is this presentation for?

Data engineers, data architects, developers

Level

Beginner

Description

Building an event processing library comes with baggage. Gojek created BEAST, its own event-processing library to consume events from Kafka and push it to BigQuery.

Maulik Soneji and Dinesh Kumar explain what Gojek learned, including why it had to build its custom event processing tool BEAST—customizing code for each input and output combination and old way of deployment—limitations with existing systems for its use case; setting quick context on Kafka and consumer and how it had to use consumers without autocommit and manual sync; how Gojek ensures no data loss, tests the application for data loss and scenarios, and monitors data loss in BigQuery and alerting; how Gojek achieved high performance, handling high throughput with acceptable latency; how consumer and producer threads communicate using a blocking queue, why it didn’t pick Redis as a store, and why it couldn’t write in Go; and how it achieved scalability using Kubernetes. Maulik and Dinesh also provide a demo for you.

Prerequisite knowledge

  • A basic understanding of Kafka and big data concepts (useful but not required)

What you'll learn

  • Learn the internals of Kafka and leveraging Kafka for event sourcing capabilities
  • Discover how to create highly scalable systems by isolating different components and individually scaling them
  • Understand production readiness, which involves deployment, CI/CD, testing with chaos, monitoring, and alerting
Photo of Maulik Soneji

Maulik Soneji

Gojek

Maulik Soneji is a product engineer at Gojek, where he works with different parts of data pipelines for a hypergrowth startup. Outside of learning about mature data systems, he’s interested in Elasticsearch, Go, and Kubernetes.

Photo of Dinesh Kumar

Dinesh Kumar

Gojek

Dinesh Kumar is a product engineer at Gojek. He has experience of building high-scale distributed systems and working with event-driven systems and components around Kafka.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires