Presented By O'Reilly and Cloudera
Make Data Work
Dec 4–5, 2017: Training
Dec 5–7, 2017: Tutorials & Conference

Spark Structured Streaming helps smart manufacturing

Xiaochang Wu (Intel)
4:15pm4:55pm Wednesday, December 6, 2017
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • IoT system architects and streaming application developers

Prerequisite knowledge

  • A basic understanding of streaming and IoT concepts

What you'll learn

  • Learn how to use Spark Structure Streaming to build an end-to-end solution for smart manufacturing


Xiaochang Wu explains how to design and implement a real-time processing platform using the Spark Structured Streaming framework to intelligently transform production lines in the manufacturing industry.

Traditional production lines created a variety of isolated structured, semistructured, and unstructured data, such as sensor data, machine screen output, log output, and database records. There are two main data scenarios: picture and video data with low frequency but in large amounts or continuous data with high frequency. Although the amount of data per unit is not in itself large, taken together, the total is very large. This data has many of the characteristics of streaming data: it’s real time, volatile, burst, disordered, and infinite. Making effective real-time decisions to retrieve values from this data is critical to smart manufacturing.

The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to Spark, we are able to build a low-latency, high-throughput, reliable operation system involving data acquisition, transmission, analysis, and storage. This system greatly improves the production process for predictive fault repair and production line material tracking efficiency and can reduce about half of the labor force for the production lines.

Photo of Xiaochang Wu

Xiaochang Wu


Xiaochang Wu is a senior software engineer on Intel’s big data engineering team, where he helps deliver the best Spark performance on Intel platforms. Xiaochang has more than 10 years’ experience in performance optimization for Intel architecture. He holds a master’s degree in computer science from Xiamen University of China.