IoT and financial trading platforms share some commonality: they intercept massive amounts of sensor or event data and must provide insights and actions in real time. The technology challenge is huge since we need to combine fast event streams, historical state, and consistent data update transactions (e.g., time series data and statistical aggregators) with data science and machine learning and present results through real-time dashboards or drive immediate corrective actions.
Traditional solutions like the Lambda Architecture or stream processing can’t meet the requirement, but with the latest advancements in Spark 2.0 coupled with Data Frames coprocessing in external real-time engines, we can analyze millions of complex events per second and deliver true real-time dashboards or actionable insights.
Yaron Haviv explains how to design real-time IoT and FSI applications, leveraging Spark with advanced data frame acceleration. Yaron then presents a detailed, practical use case, diving deep into the architectural paradigm shift that makes the powerful processing of millions of events both efficient and simple to program.
Yaron Haviv is the CTO and founder of iguaz.io as well as a serial entrepreneur with deep technological experience in the fields of big data, cloud, storage, and networking. Previously, Yaron was the vice president of datacenter solutions at Mellanox, where he led technology innovation, software development, and solution integrations and was the key driver of open source initiatives and new solutions with leading database and storage vendors, enterprise organizations, and cloud and Web 2.0 customers. Before Mellanox, Yaron was the CTO and vice president of R&D at Voltaire, a high-performance computing, I/O, and networking company. Yaron is a thought leader who often speaks at big data and cloud technology events and writes a popular technology blog.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.