With the growth of mobile, the cloud, and the Internet of Things, the world is becoming more connected every year. Much-hyped big data is currently the popular method to deal with the data that is created. Large amounts of historical data are stored in Hadoop or other platforms, and big data frameworks leverage machine-learning frameworks, such as R, Apache Spark, or H2O, to draw new knowledge and to find patterns from this data (for example, for promotions, cross-selling, or fraud detection). The key challenge is how these findings can be integrated from historical data into new transactions in real time to make customers happy, increase revenue, or prevent fraud.
“Fast data” via stream processing is the solution to embed patterns—obtained from analyzing historical data—into future transactions in real time. Kai Wähner explores “fast data” frameworks and explains how they are strongly related to microservices. Kai uses several real-world success stories to explain the concepts behind stream processing and its relation to Hadoop and other big data platforms and discusses how the patterns and statistical models of R, Spark MLlib, H2O, and other technologies can be integrated into real-time processing. Along the way, Kai points out why a microservices architecture helps solve the Agile requirements for these kind of projects, offers a brief overview of available open source frameworks and commercial products, such as Apache Storm, Apache Flink, Spark Streaming, IBM InfoSphere Streams, or TIBCO StreamBase, and shares a live demo showing how to implement stream processing, how to integrate machine learning, and how human operations can be enabled in addition to the automatic processing via a web UI and push events.
Kai Wähner is a technology evangelist at Confluent. Kai’s areas of expertise include big data analytics, machine learning, deep learning, messaging, integration, microservices, the internet of things, stream processing, and blockchain. He’s regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture, and ApacheCon and has written a number of articles for professional journals. Kai also shares his experiences with new technologies on his blog.
©2016, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org