Engineering the Future of Software
Feb 3–4, 2019: Training
Feb 4–6, 2019: Tutorials & Conference
New York, NY

Challenges with reuse within a large and diverse engineering organization: A case study

Andrey Utis (Capital One)
3:50pm–4:40pm Tuesday, February 5, 2019
DevOps & Continuous Delivery
Location: Sutton Center/Sutton South
Secondary topics:  Case Study, Framework-focused
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Engineering leaders and architects

Level

Non-technical

Prerequisite knowledge

  • Basic knowledge of continuous delivery patterns and tools

What you'll learn

  • Learn how a reusable CI/CD framework can be structured, how to empower teams within the company to contribute code to a framework, and how to encourage reuse and consistency without discouraging diversity and innovation

Description

Capital One embarked on a journey to become a leading technology company several years ago. The company has embraced open source, the public cloud, continuous delivery, DevOps, and microservices. It regularly competes for talent with companies like Amazon and Google, and engineers often jump between these companies (particularly in DC, Seattle, and NYC).

Capital One has already seen great results from this transformation, but it has come with its own challenges. Capital One used to have a large enterprise architecture team, which made decisions about which frameworks, platforms, patterns should be used. As the company hired more engineering talent, that did not work well anymore. Great engineers don’t enjoy the ivory tower approach and want to make their own decisions about the applications they’re building. As a result, Capital One allowed much more freedom to individual engineering teams and now has a truly polyglot and diverse environment.

As Capital One made a push to public cloud and embraced continuous delivery, teams that adapted different technologies naturally started building their own CI/CD frameworks and pipelines. At the end of 2017, Capital One engineering leadership picked the most mature of these frameworks and mandated that everyone must use it. This was done to minimize the amount of “plumbing” work teams were doing and to avoid additional audit risk. (Capital One is in a highly regulated industry and production deployments have a multitude of audit controls attached to them.) This decision was initially met with disappointment and pushback. While the framework that was picked was the most mature, it did not immediately support everyone’s diverse patterns: ECS versus Kubernetes; all the programming languages used to build services (Java, Go, Clojure, Node.js, Python, etc.); containerized versus serverless patterns; Angular versus React for single-page applications; etc. Other teams that built their own frameworks felt disrespected and underappreciated, and everyone’s morale suffered.

Andrey Utis details the architecture of this reusable CI/CD framework and explains how Capital One addressed this challenging situation:

  • Openness and explicitly stating the reason for controversial decisions really helps improve morale. Leaders met in person with many teams that were unhappy with the decision, explained the reasoning behind it, and asked for their help and support.
  • An open source-style contribution process helps alleviate the feeling of ivory tower, top-down decisions. Teams that are unhappy with what’s being offered are encouraged to contribute changes to the framework.
  • Having good product management in place helps teams that build internal tools meet their customer needs better. Capital One’s product owners conduct empathy interviews with engineers to find out their pain points and help prioritize a roadmap for the framework.

Finding the right balance between supporting diverse patterns and consistency and reuse can be a challenge. Andrey details how the company built enough trust with engineering teams to be able to say “don’t do this” without affecting team morale.

Photo of Andrey Utis

Andrey Utis

Capital One

Andrey Utis is a senior director of software engineering at Capital One, where he leads the retail bank architecture and platform teams. Andrey has held several leadership positions on engineering teams within Capital One, delivering features like online account opening and fraud and customer security protection and building a state-of-the-art CI/CD framework and transforming the organization to have a DevOps mindset. Previously, he spent several years as a consultant building large-scale distributed systems. Andrey studied computer science at the University of Maryland and Case Western Reserve University.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)