The Graviton Trade: Price-Performance vs. Migration Drag
AWS Graviton migration trades strong price-performance gains against real coordination costs. This map shows the benefit, the cost, and the conditions that decide when Arm adoption pays off.
Red HatBala Tripura BodapatiTutorial
Red Hat Quay is a container image registry with multiple configurable components, and a large enterprise customer experienced pull failures under load across 160 nodes. The team developed a systematic profiling method to answer how many concurrent pulls Quay can handle, testing a specific configuration with 2 Quay replicas, NooBaa storage, and HAProxy. They identified three bottlenecks: NooBaa storage saturation, test client limitations due to Python’s GIL, and kubelet PID exhaustion. Scaling NooBaa from 4 to 6 endpoints more than doubled throughput and recovered from 49% failure to 100% success. Final recommendations for 2 vCPU Quay were 25 concurrent pulls with 0.22 img/s, scaling to 16 vCPU with 300 concurrency and 0.56 img/s, though the 16 vCPU ceiling reflects a single test client limitation.
A large enterprise customer running Red Hat Quay across 160 nodes in 2 edge clusters experienced pull failures under load and asked how many concurrent pulls Quay can handle. The team aimed to produce general sizing recommendations for a standard Quay deployment, usable as a baseline by customers and TAMs. Quay is a system of 10 configurable components, five of which directly affect pull throughput: quay, postgres, objectstorage, redis, and route. Any one component can cap throughput, so there is no universal maximum; numbers are tied to a specific configuration.
The team profiled a Red Hat Quay deployment by running load tests with quay-performance-scripts, an open source framework that deploys an orchestrator pod on Red Hat OpenShift. Worker pods generated load by authenticating, fetching manifests, and downloading layer blobs over HTTP, with 6 layers in parallel per image. They monitored all components using Dittybopper with Grafana dashboards backed by Prometheus, tracking CPU, memory, network, and connection counts. They systematically eliminated bottlenecks by scaling NooBaa endpoints, fixing test client issues (including Python GIL limitations and token refresh), and raising kubelet podPidsLimit. Final sizing recommendations were produced for a specific configuration with 2 Quay replicas and NooBaa storage.
In a multicomponent system like Quay, any component can cap throughput; profiling must cover storage, database, and ingress, not just application pods.
Scaling NooBaa object storage endpoints from 4 to 6 more than doubled pull throughput and recovered from 49% failure to 100% success, showing storage must be sized proportionally with Quay.
Test client limitations, such as Python’s GIL causing TCP zero-window at high concurrency, can invalidate performance numbers; distributing load across multiple client pods can recover throughput.
The test client's Python GIL limited concurrency. At 400+ threads, lock overhead overwhelmed HTTP work, causing TCP zero-window packets. Splitting 400 concurrency across 4 pods recovered throughput to ~1.5 img/s, a 21x improvement.
NooBaa storage endpoints hit CPU and connection limits before Quay pods. Scaling from 4 to 6 endpoints more than doubled throughput and recovered from 49% failure to 100% success. Storage must be sized proportionally with Quay.
At 5,000 concurrency on a single pod, the kubelet's podPidsLimit of 16,384 was hit. They raised it to 65,536 using a KubeletConfig CR and set corresponding cgroup limits. This is a platform configuration issue for high-concurrency workloads.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
AWS Graviton migration trades strong price-performance gains against real coordination costs. This map shows the benefit, the cost, and the conditions that decide when Arm adoption pays off.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.