Presented By O'Reilly and Cloudera
Make Data Work
September 25–26, 2017: Training
September 26–28, 2017: Tutorials & Conference
New York, NY

An open source architecture for the IoT

Dave Shuman (Cloudera), James Kirkland (Red Hat)
5:25pm6:05pm Wednesday, September 27, 2017
Secondary topics:  Architecture, IoT

Who is this presentation for?

  • Solution architects, IoT leads, big data architects, and line of business managers and directors

Prerequisite knowledge

  • A basic understanding of key IoT components and architecture and the Apache Hadoop ecosystem

What you'll learn

  • Explore an end-to-end reference architecture for implementing open source IoT stacks and a specific open source stack for gateways and the cloud
  • Understand key considerations from a data management perspective for deploying IoT use cases
  • Hear examples and customer use cases on how the open source IoT architecture was deployed


The IoT arena is exploding with a diverse set of technologies, features, standards, and architectures, each of which promises to be the panacea and the path forward for future enterprise adoption. However, given the speed of change, innovation, and the wide variety of use cases that IoT enables, it would be highly unlikely for any single proprietary IoT solution maintained by any one vendor to keep up with all the new standards, requirements, and technologies to be incorporated.

Given these challenges, an end-to-end IoT architecture based on open source is starting to emerge as an attractive option for organizations embarking their IoT journey. In the coming years, the overall IoT market, like the enterprise software market, will pivot toward an open source, open standards model that enables innovation and is interoperable, flexible, and secure.

Eclipse IoT is a community that is working together to establish an IoT architecture based on open source technologies and standards. Dave Shuman and James Kirkland showcase an end-to-end IoT architecture, from the device/edge to the cloud, specifically highlighting two key stacks in the ecosystem: Eclipse Kura, an open source stack for gateways and the edge, and Eclipse Kapua, the first open source IoT cloud platform.

Eclipse Kura provides a general purpose middleware and application container for IoT gateway services and includes everything for the edge, such as an application container or runtime environment, communication and connectivity, data management and messaging, and remote management. Eclipse Kapua is a modular and cloud-based platform that provides services required for real-time data management and analytics and management of the IoT gateways and smart edge devices. Kapua also provides a core integration framework and an initial set of core IoT services, including a device registry, device management services, messaging services, data management, and application enablement.

Apart from an end-to-end open IoT architecture, Dave and James explain how organizations can utilize an open source architecture for data management and analytics and how some of the leading open source projects, such as Apache Spark and Apache Kudu, can enable machine learning and drive advanced analytics on IoT data. Along the way, they share real-life examples of how some of the leading organizations are utilizing this open source stack to power some of the most compelling IoT use cases.

Photo of Dave Shuman

Dave Shuman


Dave Shuman is the industry lead for the IoT and manufacturing at Cloudera. Dave has an extensive background in big data analytics, business intelligence applications, database architecture, logical and physical database design, and data warehousing. Previously, Dave held a number of roles at Vision Chain, a leading demand signal repository provider enabling retailer and manufacturer collaboration, including chief operations officer, vice president of field operations responsible for customer success and user adoption, vice president of product responsible for product strategy and messaging, and director of services. He also served at such top CG companies as Kraft Foods, PepsiCo, and General Mills, where he was responsible for implementations; was vice president of operations for enews, an ecommerce company acquired by Barnes and Noble; was executive vice president of management information systems, where he managed software development, operations, and retail analytics; and developed ecommerce applications and business processes used by, Yahoo, and Excite and pioneered an innovative process for affiliate commerce. He holds an MBA with a concentration in information systems from Temple University and a BA from Earlham College.

Photo of James Kirkland

James Kirkland

Red Hat

James Kirkland is the advocate for Red Hat’s initiatives and solutions for the internet of things (IoT) and is the architect of Red Hat’s strategy for IoT deployments. This open source architecture combines data acquisition, integration, and rules activation with command and control data flows among devices, gateways, and the cloud to connect customers’ operational technology environments with information technology infrastructure and provide agile IoT integration. James serves as the head subject-matter expert and global team leader of system architects responsible for accelerating IoT implementations for customers worldwide. Through his collaboration with customers, partners, and systems integrators, Red Hat has grown its IoT ecosystem, expanding its presence in industries including transportation, logistics, and retail and accelerating adoption of IoT in large enterprises. James has a deep knowledge Unix and Linux variants that spans the course of his 20-year career at Red Hat, Racemi, and Hewlett-Packard. He is a steering committee member of the IoT working group for, a member of the IIC, and a frequent public speaker and author on a wide range of technical topics.