Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

IoT in the enterprise: A look at Intel (IoT) Inside

Moty Fania (Intel)
1:50pm–2:30pm Wednesday, 03/30/2016
IoT and Real-time

Location: 210 C/G
Average rating: ***..
(3.20, 5 ratings)

Recent years have seen significant evolution of the Internet of Things (IoT). It has become increasingly easy to connect devices to the Internet and send readings to the public cloud in real time. However, it’s quite evident that the adoption of IoT platforms within the enterprise is lagging and less prevalent, due, in part, to the challenges of an on-premises deployment, the lack of related infrastructure, and in some cases, the lack of domain expertise required to deploy analytical solutions that demonstrate high value through real-life use cases.

Intel IT has implemented a platform that enables near-real-time stream analytics at scale, using Docker and Core OS for on-premises deployments. Unlike other IoT analytics implementations, which settle for basic statistics or make many assumptions on the collected data, usually relevant only to specific domains, Intel implemented a generic analytics layer that uses machine-learning techniques and advanced statistical tests to provide meaningful insights to enterprise users in different use cases and business domains.

Moty Fania shares Intel’s IT experience implementing this on-premises big data IoT platform for internal use cases. Intel’s ultimate goal was to allow users and organizations in the enterprise to gain insights and business value while hiding the complexity and avoiding the tremendous efforts in building, maintaining, and stabilizing such systems. For this purpose, Intel developed a “smart data pipe,” which allows analysis of massive data streams from many devices simultaneously in near real time. Based on several open source technologies including Kafka, Spark, and Hadoop (HBase), this platform is optimized to be portable and easily deployed with Docker and Core OS with the click of a button.

To enable stream analytics at scale, Intel implemented a sophisticated stream-processing framework, Pigeon, based on Akka. This model enables us to build a distributed, multithreaded system without the complexities of traditional multithreaded programming. Pigeon implements a cluster capable of processing channels named topologies that process the data according to any arbitrary logic determined by the users. It handles the creation of topologies, balancing them across the cluster, and allows nodes to join or leave dynamically. In the event that one or more topologies fail, they are automatically recovered without any human intervention.

Intel also provides an analytical layer on top of this smart data pipe that yields meaningful insights for diverse users and address situations where there is no prior knowledge about the time series data before it starts arriving and there is no way to assume “normal” behavior or distribution. This analytical layer includes four main components:

  1. Time series classification—a component that checks for each sensor whether its data is stationary, periodic, or neither
  2. Periodicity removal—a component that models the period (if one exists) and cancels it out
  3. Change detection—an ensemble of tests that either monitor a single sensor or a collection of device sensors
  4. Alert classification—a set of nonparametric statistical tests that enable pointing out the most significant changes causing each alert (in terms of changes in individual sensors or in sensor correlations)

Moty shares the results when the presented analytic capabilities were applied to several datasets—both internally and externally. For example, in one POC, they helped identify a fab tool which caused a yield problem; in another POC, they showed malfunctions of electrical network voltage-sensors. This is a promising direction for enabling the anticipated IOT analytics revolution in the enterprise.

Photo of Moty Fania

Moty Fania


Moty Fania is a principle engineer for big data analytics at Intel IT, where he drives the overall technology and architectural roadmap and owns development and architecture. Moty has over 13 years of experience in BI, data warehousing, and decision-support solutions. He holds a bachelor’s degree in computer science and economics and a master’s degree in business administration from the Ben-Gurion University in Israel.