Engineering the Future of Software
November 13–14, 2016: Training
November 14–16, 2016: Tutorials & Conference
San Francisco, CA

An architecture for merging fast data and enterprise applications: The SMACK stack

Dean Wampler (Lightbend)
3:50pm–4:40pm Wednesday, 11/16/2016
Integration architecture
Location: Georgian Level: Intermediate
Average rating: ****.
(4.40, 5 ratings)

Prerequisite knowledge

  • Prior experience building architectures for data-centric or enterprise systems (useful but not required)

What you'll learn

  • Understand the weaknesses of current, popular approaches and how the SMACK Stack addresses current needs in both data-centric systems and general enterprise systems

Description

Big data architectures—those using large frameworks like Spark, YARN, HBase or Cassandra, HDFS, and Kafka—have been slow to embrace microservices. Everything else—i.e., enterprise architectures (whether microservice-based or not)—have been less concerned with large data volumes and more interested in reactive tools that are flexible, adaptive, scalable, resilient, and event/message driven.

These two spheres are now slowly converging, as data teams need answers faster (hence the growing interest in streaming architectures) and enterprises become more data driven (hence the need for sophisticated, scalable data processing options that are still event-driven).

Another trend affecting both spheres is the need to optimize resource utilization and lower costs, which has led to the growth of virtualized services on flexible, efficient clusters that are capable of running all services. While Hadoop has dominated the data world, it is a first-generation architecture that isn’t well suited for more general enterprise needs.

Dean Wampler explores the SMACK stack—Spark, Mesos, Akka, Cassandra, and Kafka—discussing the role each tool plays in addressing the needs of fast data and enterprise environments, as well as what’s missing and what areas need to mature.

Photo of Dean Wampler

Dean Wampler

Lightbend

Dean Wampler is the vice president of fast data engineering at Lightbend, where he leads the creation of the Lightbend Fast Data Platform, a distribution of scalable, distributed stream processing tools including Spark, Flink, Kafka, and Akka, with machine learning and management tools. Dean is the author of Programming Scala and Functional Programming for Java Developers and the coauthor of Programming Hive, all from O’Reilly. He is a contributor to several open source projects. A frequent Strata speaker, he’s also the co-organizer of several conferences around the world and several user groups in Chicago.