Presented By
O’Reilly + Cloudera
Make Data Work
29 April–2 May 2019
London, UK
Please log in

Processing 10M samples a second to drive smart maintenance in complex IIoT systems

Geir Engdahl (Cognite), Daniel Bergqvist (Google)
14:55–15:35 Wednesday, 1 May 2019
Data Engineering and Architecture, Expo Hall, Streaming and IoT
 Location: Expo Hall 2 (Capital Hall N24)
Secondary topics:  AI and Data technologies in the cloud, IoT and its applications, Streaming and realtime analytics
Average rating: ****.
(4.00, 2 ratings)
Download slides (PDF)

Who is this presentation for?

  • Developers of IoT systems who need to handle high-velocity data streams and store significant quantities of time series data to drive ML and analytics

Level

Intermediate

Prerequisite knowledge

  • A basic understanding of cloud computing and distributed systems, such as RPC, queues, key-value stores, and containerization, and scaling of stateless services
  • No prior knowledge of Google Cloud Platform required

What you'll learn

  • Gain a detailed view of a live, battle-proven architecture for high performance, robust streaming sensor data ingest, and cost-effective storage of large volumes of time series data while using fully managed services for all things stateful
  • Learn how to add aggregation and fast queries to this architecture and how to achieve high performance with machine learning algorithms on top of this data store

Description

Today’s industrial IoT (IIoT) systems generate huge volumes of data—data that can be both difficult to both manage and make effective use of.

Geir Engdahl and Daniel Bergqvist discuss a Cognite-developed IIoT setup, based on the Google Cloud Bigtable NoSQL database, that’s currently being used to process multivalue time series data at rates of up to 10M samples a second. This system is being used to drive ML-based production optimization and predictive maintenance in industrial systems comprising many thousands of sensors, replacing costly scheduled maintenance with targeted, proactive alerts to operators when system anomalies are detected.

Geir and Daniel describe the key elements of the Cognite-developed system and demonstrate how to configure the underlying Google Cloud Bigtable NoSQL database to process high-velocity IoT data streams. They conclude by sharing some thoughts on data optimization to further increase the efficiency of the system.

Photo of Geir Engdahl

Geir Engdahl

Cognite

Geir Engdahl is CTO at Cognite, where he leads the R&D department in developing the Cognite industrial IoT data platform. Previously, Geir was founder and CEO/CTO at Snapsale, a machine learning classifieds startup (acquired by Schibsted), and a senior software engineer at Google in Canada, where he worked on machine learning for AdWords and AdSense, resulting in the Conversion Optimizer product. Geir holds an MSc in computational science from the University of Oslo. He won a silver medal from the International Olympiad in Informatics.

Photo of Daniel Bergqvist

Daniel Bergqvist

Google

Daniel Bergqvist works on the Cloud Platform team at Google. In his more than 10 years of experience in the software industry, Daniel has held positions at companies such as Ericsson and Opera Software. Daniel holds a bachelor’s degree in computer science from Uppsala University. He lives in Stockholm and likes to spend his spare time freediving.