Iterating Towards a More Scalable Ingress

Summary

Shopify’s infrastructure uses Kubernetes to serve millions of requests per minute, and its growth exposed ingress performance problems during frequent application deployments. After replacing Google Cloud Load Balancer Controller (glbc) with endpoint-aware ingress-nginx, the Edgescale team focused on reducing the NGINX reloads that caused latency increases. They added dynamic configuration: the controller POSTs endpoint lists to an NGINX HTTP endpoint, where lua-nginx-module updates an in-memory shared dictionary instead of regenerating the configuration file and reloading NGINX. Reload avoidance preserves keepalive connections and load-balancing state while reducing the resource effects of replacing worker processes. Tests found ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration produced its largest gains during deploys, especially beyond the 80th percentile.

Takeaways

1. 1

   NGINX reloads during deployments increased latency and resource consumption by replacing worker processes, resetting keepalive connections, and losing load-balancing state.

2. 2

   Dynamic configuration uses lua-nginx-module and an internal shared dictionary to update service endpoints without reloading NGINX.

3. 3

   Ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration improved results most clearly during deployments.

FAQ