Build & maintain complex distributed systems
17–18 October 2017: Training
18–20 October 2017: Tutorials & Conference
London, UK

Building and running serverless data pipelines on AWS

Mike Roberts (Symphonia)
15:4016:20 Friday, 20 October 2017
Location: Park Suite
Average rating: ***..
(3.50, 4 ratings)

Who is this presentation for?

  • Tech leads and data engineering architects

Prerequisite knowledge

  • Basic familiarity with Amazon Web Services and distributed systems

What you'll learn

  • Explore a real-life example where an existing data platform was rearchitected and reengineered to provide several improvements


Serverless services are those that fit into the classes of backends as a service (BaaS) (vendor-hosted, highly scalable data and logic components) and functions as a service (FaaS) (the ability to write custom server-side software in small, event-triggered functions, deployed to a fully managed platform). Serverless architectures are those systems that make significant use of either or both of these types of service. Data pipelines are an excellent use case for serverless architecture because of their inherently event-driven behavior and often volatile nature of scaling.

Mike Roberts describes a real-life example where an existing data platform was rearchitected and reengineered to provide several improvements—significantly increased data capacity, reduced cost, and vastly improved development cycle time—all while reducing operational effort. Such gains were possible by using the vast array of serverless services provided by AWS, such as Kinesis, Lambda, and S3. At the same time, “always on” server components were included within the pipeline where appropriate to create a hybrid architecture of serverless and traditional architecture.

Building such an architecture is just part of the story, of course; running it without headaches is just as crucial. Therefore activities like continuous delivery and monitoring also need to be part of the picture. Mike explains how to implement these using further AWS services, including CodePipeline, Serverless Application Model, and CloudFormation. You’ll learn what is possible when building and running scalable serverless data pipelines on AWS and leave ready to start your own engineering of such systems.

Photo of Mike Roberts

Mike Roberts


Mike Roberts is a partner at Symphonia, a cloud technology consultancy based in New York City. Mike’s a longtime proponent of Agile and DevOps values and is excited by the role that cloud technologies have played in enabling such values for many high-functioning software teams. Mike can be reached at