Fast Data architectures provide an answer to the increasing need for the enterprise to process and analyze continuous streams of data, which helps accelerate decision making and enables faster responses to changing characteristics of their market. Apache Spark is a popular framework for data analytics. Its capabilities in the streaming domain are represented by two APIs: The low-level Spark Streaming and the more declarative Structured Streaming, which builds upon the recent advances in Spark SQL query optimization and code generation.
After a quick introduction to both APIs, we will discuss their virtues, capabilities and key differences:
- How to get started: ease of development.
- How to deal with time: both at the processing and event level
- How to deal with state: locally, distributed and its relation with time
- How to migrate: functional coding strategies
- How to do ML: machine learning capabilities
Using practical examples from actual applications, we will provide guidance on how to choose one or even combine both APIs to implement functional and resilient streaming pipelines.
Gerard Maas contributes to Lightbend Fast Data Platform as a Senior SW Engineer, where he focuses on the integration of stream processing technologies. Previously, he has held leading roles at several startups and large enterprises, building data science governance, cloud-native IoT platforms and scalable APIs.
He enjoys giving tech talks, contributing to small and large open source projects, tinkering with drones and building personal IoT projects.
Gerard is the co-author of ‘Learning Spark Streaming’, a book from O’Reilly Media.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org