Transactional event streaming with Apache Pulsar
Who is this presentation for?Data engineers, data architects, developers
The highest message delivery guarantee that Apache Pulsar provides is “exactly-once,” producing at a single partition via Idempotent Producer. You’re guaranteed that every message produced to a single partition via an Idempotent Producer will be persisted exactly once without data loss. However, there’s no “atomicity” when a producer attempts to produce messages to multiple partitions. From the consumer side, acknowledgement is a best-effort operation, which results in message redelivery, hence the consumer receiving duplicate messages. Pulsar only guarantees “at-least-once” consumption for consumers. It creates inconvenience and brings complexity when you use Pulsar to build mission-critical services (such as billing services).
Pulsar introduces transaction support in 2.5.0 version to simplify the process of building reliable and fault resilient services using Apache Pulsar and Pulsar Functions. It also provides the capability to achieve end-to-end exactly-once for streaming jobs in other stream processing engines. Sijie Guo and Yong Zhang lead a deep dive into the details of Pulsar transaction and how Pulsar transaction is used in Pulsar Functions and other processing engines to achieve transactional event streaming. You’ll see a real-world example on how Apache Pulsar is used for processing tens of billions of transactions every day in the internet giant’s core billing service.
- A basic understanding of Apache Pulsar and messaging and streaming technologies
What you'll learn
- Discover how Pulsar transaction works, how Pulsar Functions offer transaction support, and how to build an end-to-end exactly-once Flink job
Sijie Guo is the founder and CEO of StreamNative, a data infrastructure startup offering a cloud native event streaming platform based on Apache Pulsar for enterprises. Previously, he was the tech lead for the Messaging Group at Twitter and worked on push notification infrastructure at Yahoo. He’s also the VP of Apache BookKeeper and PMC Member of Apache Pulsar.
Yong Zhang is a software engineer at StreamNative. He’s also a Pulsar and BookKeeper contributor, where he contributes a lot at Pulsar transaction, storage, and tools.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
Premier Diamond Sponsors
Premier Exhibitor Plus
Diversity & Inclusion Sponsor
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires