A Perfect DNS Storm
tl;dr - a combination of ndots, the DNS resolver implementation in the musl library, and a flawed DNS resolution implementation in Netty caused repeated failures in DNS resolution for Kubernetes-deployed services.
DNS resolution can be a tricky problem to diagnose when it happens inconsistently. Adding Kubernetes and container-based applications