BEAST: Building an event processing library to handle millions of events
Who is this presentation for?Data engineers, data architects, developers
Building an event processing library comes with baggage. Gojek created BEAST, its own event-processing library to consume events from Kafka and push it to BigQuery.
Maulik Soneji and Dinesh Kumar explain what Gojek learned, including why it had to build its custom event processing tool BEAST—customizing code for each input and output combination and old way of deployment—limitations with existing systems for its use case; setting quick context on Kafka and consumer and how it had to use consumers without autocommit and manual sync; how Gojek ensures no data loss, tests the application for data loss and scenarios, and monitors data loss in BigQuery and alerting; how Gojek achieved high performance, handling high throughput with acceptable latency; how consumer and producer threads communicate using a blocking queue, why it didn’t pick Redis as a store, and why it couldn’t write in Go; and how it achieved scalability using Kubernetes. Maulik and Dinesh also provide a demo for you.
- A basic understanding of Kafka and big data concepts (useful but not required)
What you'll learn
- Learn the internals of Kafka and leveraging Kafka for event sourcing capabilities
- Discover how to create highly scalable systems by isolating different components and individually scaling them
- Understand production readiness, which involves deployment, CI/CD, testing with chaos, monitoring, and alerting
Maulik Soneji is a product engineer at Gojek, where he works with different parts of data pipelines for a hypergrowth startup. Outside of learning about mature data systems, he’s interested in Elasticsearch, Go, and Kubernetes.
Dinesh Kumar is a product engineer at Gojek. He has experience of building high-scale distributed systems and working with event-driven systems and components around Kafka.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
Premier Diamond Sponsors
Premier Exhibitor Plus
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires