Presented By O'Reilly and Cloudera
Make Data Work
September 26–27, 2016: Training
September 27–29, 2016: Tutorials & Conference
New York, NY

How to achieve zero-latency IoT and FSI data processing with Spark

yaron haviv (
2:55pm–3:35pm Thursday, 09/29/2016
IoT & real-time
Location: 1 E 12/1 E 13 Level: Intermediate
Average rating: **...
(2.00, 1 rating)

Prerequisite knowledge

  • A basic understanding of Spark and stream processing
  • What you'll learn

  • Understand how to leverage Spark with the latest data-processing technologies to achieve a breakthrough in real-time applications for FSI and the IoT
  • Description

    IoT and financial trading platforms share some commonality: they intercept massive amounts of sensor or event data and must provide insights and actions in real time. The technology challenge is huge since we need to combine fast event streams, historical state, and consistent data update transactions (e.g., time series data and statistical aggregators) with data science and machine learning and present results through real-time dashboards or drive immediate corrective actions.

    Traditional solutions like the Lambda Architecture or stream processing can’t meet the requirement, but with the latest advancements in Spark 2.0 coupled with Data Frames coprocessing in external real-time engines, we can analyze millions of complex events per second and deliver true real-time dashboards or actionable insights.

    Yaron Haviv explains how to design real-time IoT and FSI applications, leveraging Spark with advanced data frame acceleration. Yaron then presents a detailed, practical use case, diving deep into the architectural paradigm shift that makes the powerful processing of millions of events both efficient and simple to program.

    yaron haviv

    Yaron Haviv is the CTO and founder of as well as a serial entrepreneur with deep technological experience in the fields of big data, cloud, storage, and networking. Previously, Yaron was the vice president of datacenter solutions at Mellanox, where he led technology innovation, software development, and solution integrations and was the key driver of open source initiatives and new solutions with leading database and storage vendors, enterprise organizations, and cloud and Web 2.0 customers. Before Mellanox, Yaron was the CTO and vice president of R&D at Voltaire, a high-performance computing, I/O, and networking company. Yaron is a thought leader who often speaks at big data and cloud technology events and writes a popular technology blog.