Health checking: A not-so-trivial task in the distributed containerized world

Description

People usually think of a health check as a simple sequence: performing a specific action and judging whether the target application is healthy based on the outcome. This becomes trickier when the application consists of multiple containers managed by a cluster orchestrator and monitored by third-party tooling. In this situation, a number of questions arise, including:

• What entity should interpret the result? Should the reasoning about the health of a task be done locally (less context) or globally (greater overhead)?
• How often should health status be delivered to balance excessive network overhead against an up-to-date status?
• Should health checks be aware of environment-specific intricacies such as namespaces and software-defined networks?
• How do you keep the overhead imposed by health checks manageable and reasonable?

Alexander Rukletsov discusses the perils of modern health checking and shares lessons learned during the revamp of the Apache Mesos health checks subsystem. Alexander explores challenges and trade-offs and offers an overview of how the modern distributed systems, such as AWS, Apache Mesos, and Kubernetes, tackle the problem of health checking, as well as alternative solutions.