Big data architectures—those using large frameworks like Spark, YARN, HBase or Cassandra, HDFS, and Kafka—have been slow to embrace microservices. Everything else—i.e., enterprise architectures (whether microservice-based or not)—have been less concerned with large data volumes and more interested in reactive tools that are flexible, adaptive, scalable, resilient, and event/message driven.
These two spheres are now slowly converging, as data teams need answers faster (hence the growing interest in streaming architectures) and enterprises become more data driven (hence the need for sophisticated, scalable data processing options that are still event-driven).
Another trend affecting both spheres is the need to optimize resource utilization and lower costs, which has led to the growth of virtualized services on flexible, efficient clusters that are capable of running all services. While Hadoop has dominated the data world, it is a first-generation architecture that isn’t well suited for more general enterprise needs.
Dean Wampler explores the SMACK stack—Spark, Mesos, Akka, Cassandra, and Kafka—discussing the role each tool plays in addressing the needs of fast data and enterprise environments, as well as what’s missing and what areas need to mature.
Dean Wampler is the vice president of fast data engineering at Lightbend, where he leads the Lightbend Fast Data Platform project, a distribution of scalable, distributed stream processing tools including Spark, Flink, Kafka, and Akka, with machine learning and management tools. Dean is the author of Programming Scala and Functional Programming for Java Developers and the coauthor of Programming Hive, all from O’Reilly. He’s a contributor to several open source projects. A frequent Strata speaker, he’s also the co-organizer of several conferences around the world and several user groups in Chicago.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org