Big data architectures—those using large frameworks like Spark, YARN, HBase or Cassandra, HDFS, and Kafka—have been slow to embrace microservices. Everything else—i.e., enterprise architectures (whether microservice-based or not)—have been less concerned with large data volumes and more interested in reactive tools that are flexible, adaptive, scalable, resilient, and event/message driven.
These two spheres are now slowly converging, as data teams need answers faster (hence the growing interest in streaming architectures) and enterprises become more data driven (hence the need for sophisticated, scalable data processing options that are still event-driven).
Another trend affecting both spheres is the need to optimize resource utilization and lower costs, which has led to the growth of virtualized services on flexible, efficient clusters that are capable of running all services. While Hadoop has dominated the data world, it is a first-generation architecture that isn’t well suited for more general enterprise needs.
Dean Wampler explores the SMACK stack—Spark, Mesos, Akka, Cassandra, and Kafka—discussing the role each tool plays in addressing the needs of fast data and enterprise environments, as well as what’s missing and what areas need to mature.
Dean Wampler is an expert in streaming data systems, focusing on applications of machine learning and artificial intelligence (ML/AI). He’s head of developer relations at Anyscale, which is developing Ray for distributed Python, primarily for ML/AI. Previously, he was an engineering VP at Lightbend, where he led the development of Lightbend CloudFlow, an integrated system for building and running streaming data applications with Akka Streams, Apache Spark, Apache Flink, and Apache Kafka. Dean is the author of Fast Data Architectures for Streaming Applications, Programming Scala, and Functional Programming for Java Developers, and he’s the coauthor of Programming Hive, all from O’Reilly. He’s a contributor to several open source projects. A frequent conference speaker and tutorial teacher, he’s also the co-organizer of several conferences around the world and several user groups in Chicago. He earned his PhD in physics from the University of Washington.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org