Hybrid cloud architectures offer the opportunity to build applications and workflows that marry the security and control of an on-premises platform with the flexibility to move workloads to a public cloud to scale capacity as needed. Hybrid cloud implementations are useful in a variety of scenarios, including scaling existing applications to meet growing demands, building a disaster recovery solution, powering cloud-native and mobile applications that need to scale with traffic spikes while maintaining security and compliance requirements, and facilitating Agile development practices.
Underlying all hybrid cloud use cases is the need for an efficient and scalable stream message delivery system to connect the on-premises data center with the public cloud deployment(s). Kafka and other publish-subscribe message systems offer a natural paradigm for integrating on-premises and on-cloud workloads. Moreover, the integration of stream processing frameworks, such as Spark Streaming and Flink, with these Kafka-based message systems enables real-time analytics to be performed at scale in hybrid cloud environments.
Mathieu Dumoulin describes how a publish-subscribe message system can be used to architect a hybrid cloud solution, how a streaming message system can be used in a complex hybrid cloud application, and how to combine sensor data from global IoT applications running in multiple public clouds. Along the way, Mathieu also covers how to leverage on-premises analysis in a stream processing framework on Hadoop, such as Spark Streaming or Flink, and the trade-offs between some the competing message systems and stream processing frameworks.
Mathieu Dumoulin is a Digital Expert at McKinsey & Company’s Tokyo office, where he advises large enterprises for big data, enterprise architecture and advanced analytics solutions.
Current areas of interest are creating production systems which optimize industrial processes on operational data and real-time IoT sensor data.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.