Challenges with reuse within a large and diverse engineering organization: A case study

Description

Capital One embarked on a journey to become a leading technology company several years ago. The company has embraced open source, the public cloud, continuous delivery, DevOps, and microservices. It regularly competes for talent with companies like Amazon and Google, and engineers often jump between these companies (particularly in DC, Seattle, and NYC).

Capital One has already seen great results from this transformation, but it has come with its own challenges. Capital One used to have a large enterprise architecture team, which made decisions about which frameworks, platforms, patterns should be used. As the company hired more engineering talent, that did not work well anymore. Great engineers don’t enjoy the ivory tower approach and want to make their own decisions about the applications they’re building. As a result, Capital One allowed much more freedom to individual engineering teams and now has a truly polyglot and diverse environment.

As Capital One made a push to public cloud and embraced continuous delivery, teams that adapted different technologies naturally started building their own CI/CD frameworks and pipelines. At the end of 2017, Capital One engineering leadership picked the most mature of these frameworks and mandated that everyone must use it. This was done to minimize the amount of “plumbing” work teams were doing and to avoid additional audit risk. (Capital One is in a highly regulated industry and production deployments have a multitude of audit controls attached to them.) This decision was initially met with disappointment and pushback. While the framework that was picked was the most mature, it did not immediately support everyone’s diverse patterns: ECS versus Kubernetes; all the programming languages used to build services (Java, Go, Clojure, Node.js, Python, etc.); containerized versus serverless patterns; Angular versus React for single-page applications; etc. Other teams that built their own frameworks felt disrespected and underappreciated, and everyone’s morale suffered.

Andrey Utis details the architecture of this reusable CI/CD framework and explains how Capital One addressed this challenging situation:

• Openness and explicitly stating the reason for controversial decisions really helps improve morale. Leaders met in person with many teams that were unhappy with the decision, explained the reasoning behind it, and asked for their help and support.
• An open source-style contribution process helps alleviate the feeling of ivory tower, top-down decisions. Teams that are unhappy with what’s being offered are encouraged to contribute changes to the framework.
• Having good product management in place helps teams that build internal tools meet their customer needs better. Capital One’s product owners conduct empathy interviews with engineers to find out their pain points and help prioritize a roadmap for the framework.

Finding the right balance between supporting diverse patterns and consistency and reuse can be a challenge. Andrey details how the company built enough trust with engineering teams to be able to say “don’t do this” without affecting team morale.