Build & maintain complex distributed systems
17–18 October 2017: Training
18–20 October 2017: Tutorials & Conference
London, UK

Building and running serverless data pipelines on AWS

Mike Roberts (Symphonia)
15:4016:20 Friday, 20 October 2017
Serverless
Location: Park Suite

Who is this presentation for?

  • Tech leads and data engineering architects

Prerequisite knowledge

  • Basic familiarity with Amazon Web Services and distributed systems

What you'll learn

  • Explore a real-life example where an existing data platform was rearchitected and reengineered to provide several improvements

Description

Serverless services are those that fit into the classes of backends as a service (BaaS) (vendor-hosted, highly scalable data and logic components) and functions as a service (FaaS) (the ability to write custom server-side software in small, event-triggered functions, deployed to a fully managed platform). Serverless architectures are those systems that make significant use of either or both of these types of service. Data pipelines are an excellent use case for serverless architecture because of their inherently event-driven behavior and often volatile nature of scaling.

Mike Roberts describes a real-life example where an existing data platform was rearchitected and reengineered to provide several improvements—significantly increased data capacity, reduced cost, and vastly improved development cycle time—all while reducing operational effort. Such gains were possible by using the vast array of serverless services provided by AWS, such as Kinesis, Lambda, and S3. At the same time, “always on” server components were included within the pipeline where appropriate to create a hybrid architecture of serverless and traditional architecture.

Building such an architecture is just part of the story, of course; running it without headaches is just as crucial. Therefore activities like continuous delivery and monitoring also need to be part of the picture. Mike explains how to implement these using further AWS services, including CodePipeline, Serverless Application Model, and CloudFormation. You’ll learn what is possible when building and running scalable serverless data pipelines on AWS and leave ready to start your own engineering of such systems.

Photo of Mike Roberts

Mike Roberts

Symphonia

Mike Roberts is an engineering leader and cofounder of Symphonia, a serverless and cloud technology consultancy. Mike is a longtime proponent of Agile and DevOps values and is excited by the role that cloud technologies have played in enabling such values for many high-functioning software teams. He sees serverless architectures as the next technological evolution of cloud systems and is optimistic about their ability to help teams be awesome. Mike can be reached at mike@symphonia.io.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)