Mar 15–18, 2020

Transactional event streaming with Apache Pulsar

Sijie Guo (StreamNative), Yong Zhang (StreamNative)
4:15pm4:55pm Wednesday, March 18, 2020
Location: 230 A

Who is this presentation for?

Data engineers, data architects, developers

Level

Beginner

Description

The highest message delivery guarantee that Apache Pulsar provides is “exactly-once,” producing at a single partition via Idempotent Producer. You’re guaranteed that every message produced to a single partition via an Idempotent Producer will be persisted exactly once without data loss. However, there’s no “atomicity” when a producer attempts to produce messages to multiple partitions. From the consumer side, acknowledgement is a best-effort operation, which results in message redelivery, hence the consumer receiving duplicate messages. Pulsar only guarantees “at-least-once” consumption for consumers. It creates inconvenience and brings complexity when you use Pulsar to build mission-critical services (such as billing services).

Pulsar introduces transaction support in 2.5.0 version to simplify the process of building reliable and fault resilient services using Apache Pulsar and Pulsar Functions. It also provides the capability to achieve end-to-end exactly-once for streaming jobs in other stream processing engines. Sijie Guo and Yong Zhang lead a deep dive into the details of Pulsar transaction and how Pulsar transaction is used in Pulsar Functions and other processing engines to achieve transactional event streaming. You’ll see a real-world example on how Apache Pulsar is used for processing tens of billions of transactions every day in the internet giant’s core billing service.

Prerequisite knowledge

  • A basic understanding of Apache Pulsar and messaging and streaming technologies

What you'll learn

  • Discover how Pulsar transaction works, how Pulsar Functions offer transaction support, and how to build an end-to-end exactly-once Flink job
Photo of Sijie Guo

Sijie Guo

StreamNative

Sijie Guo is the founder and CEO of StreamNative, a data infrastructure startup offering a cloud native event streaming platform based on Apache Pulsar for enterprises. Previously, he was the tech lead for the Messaging Group at Twitter and worked on push notification infrastructure at Yahoo. He’s also the VP of Apache BookKeeper and PMC Member of Apache Pulsar.

Photo of Yong Zhang

Yong Zhang

StreamNative

Yong Zhang is a software engineer at StreamNative. He’s also a Pulsar and BookKeeper contributor, where he contributes a lot at Pulsar transaction, storage, and tools.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires