Managing data chaos in the world of microservices

Description

Microservices is one of the hottest topics in recent years, and the industry is shifting toward splitting applications into smaller and smaller independent units. This is all happening for very good reasons; you can gain a lot both in terms of technologies and organizational scalability. Many infrastructure tools to support the movement have been developed, from schedulers, deploy automation, and services discovery systems to development tools, like distribute tracers, log aggregators, and analyzers, and we’ve invented and reinvented protocols to make microservices communication even more efficient. However, one problem is often overlooked: the data layer is being diluted due to active encapsulation, which is essential for microservices to grow and evolve.

As we move toward more independently encapsulated services, we’re experiencing dramatically increased challenges managing data, including:

• Observability, knowledge sharing, and data discovery (Who owns that piece of the data? Where can I find that thing?)
• Querying the data (What API should I expose for others? How can I get this info from that dataset? Should I cache this or re-query when necessary?)
• Structural and semantic changes in the datasets (Can I add a new field here? Who’s using this record, and how should I update one not breaking any other services?)

These problems are common, but most of our effort and attention is directed at infrastructure, which is easier to find generic solutions for. On the other hand, making sense of the data is hardly a generalizable problem. There have been many attempts to tame the chaos associated with independent dataset management. Alexey Kachayev discusses high-level approaches to build a sharable abstraction layer separating “physical” details from logical concerns as well as specific technologies you can leverage.

The growing complexity of your data layer may overshadow the benefits of microservices architecture you deployed, so the sooner you start working on the solution, the easier it will be to manage the chaos.