Presented By O'Reilly and Cloudera
December 5-6, 2016: Training
December 6–8, 2016: Tutorials & Conference

Architecting a hybrid cloud application using a global publish-subscribe streaming message system

Mathieu Dumoulin (McKinsey & Company)
1:45pm–2:25pm Wednesday, December 7, 2016
IoT and intelligent real-time applications
Location: Summit 2 Level: Beginner
Average rating: ***..
(3.00, 2 ratings)

Prerequisite Knowledge

  • A basic knowledge of Hadoop and message publish-subscribe systems (such as Kafka)

What you'll learn

  • Explore hybrid cloud use cases and how publish-subscribe message streams can enable these use cases in a natural and simple way
  • Understand how Kafka-esque message streams and stream processing technologies can be used in concert to support real-time IoT applications at scale
  • Discover the trade-offs between several competing message stream technologies and stream processing frameworks


Hybrid cloud architectures offer the opportunity to build applications and workflows that marry the security and control of an on-premises platform with the flexibility to move workloads to a public cloud to scale capacity as needed. Hybrid cloud implementations are useful in a variety of scenarios, including scaling existing applications to meet growing demands, building a disaster recovery solution, powering cloud-native and mobile applications that need to scale with traffic spikes while maintaining security and compliance requirements, and facilitating Agile development practices.

Underlying all hybrid cloud use cases is the need for an efficient and scalable stream message delivery system to connect the on-premises data center with the public cloud deployment(s). Kafka and other publish-subscribe message systems offer a natural paradigm for integrating on-premises and on-cloud workloads. Moreover, the integration of stream processing frameworks, such as Spark Streaming and Flink, with these Kafka-based message systems enables real-time analytics to be performed at scale in hybrid cloud environments.

Mathieu Dumoulin describes how a publish-subscribe message system can be used to architect a hybrid cloud solution, how a streaming message system can be used in a complex hybrid cloud application, and how to combine sensor data from global IoT applications running in multiple public clouds. Along the way, Mathieu also covers how to leverage on-premises analysis in a stream processing framework on Hadoop, such as Spark Streaming or Flink, and the trade-offs between some the competing message systems and stream processing frameworks.

Photo of Mathieu Dumoulin

Mathieu Dumoulin

McKinsey & Company

Mathieu Dumoulin is a Digital Expert at McKinsey & Company’s Tokyo office, where he advises large enterprises for big data, enterprise architecture and advanced analytics solutions.
Current areas of interest are creating production systems which optimize industrial processes on operational data and real-time IoT sensor data.