---
title: "Sizing Red Hat Quay: Profiling components to find real limits"
description: "Red Hat Quay is a container image registry with multiple configurable components, and a large enterprise customer experienced pull failures under load across 160 nodes. The team developed a systematic profiling method to answer how many concurrent pulls Quay can handle, testing a specific configuration with 2 Quay replicas, NooBaa storage, and HAProxy. They identified three bottlenecks: NooBaa storage saturation, test client limitations due to Python’s GIL, and kubelet PID exhaustion. Scaling NooBaa from 4 to 6 endpoints more than doubled throughput and recovered from 49% failure to 100% success. Final recommendations for 2 vCPU Quay were 25 concurrent pulls with 0.22 img/s, scaling to 16 vCPU with 300 concurrency and 0.56 img/s, though the 16 vCPU ceiling reflects a single test client limitation."
---

# Sizing Red Hat Quay: Profiling components to find real limits

[Red Hat](https://yomu.fyi/company/red-hat) · Bala Tripura Bodapati · Oct 29, 2026

**Type:** Tutorial

## Summary

Red Hat Quay is a container image registry with multiple configurable components, and a large enterprise customer experienced pull failures under load across 160 nodes. The team developed a systematic profiling method to answer how many concurrent pulls Quay can handle, testing a specific configuration with 2 Quay replicas, NooBaa storage, and HAProxy. They identified three bottlenecks: NooBaa storage saturation, test client limitations due to Python’s GIL, and kubelet PID exhaustion. Scaling NooBaa from 4 to 6 endpoints more than doubled throughput and recovered from 49% failure to 100% success. Final recommendations for 2 vCPU Quay were 25 concurrent pulls with 0.22 img/s, scaling to 16 vCPU with 300 concurrency and 0.56 img/s, though the 16 vCPU ceiling reflects a single test client limitation.

## Context

A large enterprise customer running Red Hat Quay across 160 nodes in 2 edge clusters experienced pull failures under load and asked how many concurrent pulls Quay can handle. The team aimed to produce general sizing recommendations for a standard Quay deployment, usable as a baseline by customers and TAMs. Quay is a system of 10 configurable components, five of which directly affect pull throughput: quay, postgres, objectstorage, redis, and route. Any one component can cap throughput, so there is no universal maximum; numbers are tied to a specific configuration.

## Approach / What changed

The team profiled a Red Hat Quay deployment by running load tests with quay-performance-scripts, an open source framework that deploys an orchestrator pod on Red Hat OpenShift. Worker pods generated load by authenticating, fetching manifests, and downloading layer blobs over HTTP, with 6 layers in parallel per image. They monitored all components using Dittybopper with Grafana dashboards backed by Prometheus, tracking CPU, memory, network, and connection counts. They systematically eliminated bottlenecks by scaling NooBaa endpoints, fixing test client issues (including Python GIL limitations and token refresh), and raising kubelet podPidsLimit. Final sizing recommendations were produced for a specific configuration with 2 Quay replicas and NooBaa storage.

## Takeaways

- In a multicomponent system like Quay, any component can cap throughput; profiling must cover storage, database, and ingress, not just application pods.
- Scaling NooBaa object storage endpoints from 4 to 6 more than doubled pull throughput and recovered from 49% failure to 100% success, showing storage must be sized proportionally with Quay.
- Test client limitations, such as Python’s GIL causing TCP zero-window at high concurrency, can invalidate performance numbers; distributing load across multiple client pods can recover throughput.

## FAQ

### Why did 16 vCPU Quay collapse at 400 concurrency?

The test client's Python GIL limited concurrency. At 400+ threads, lock overhead overwhelmed HTTP work, causing TCP zero-window packets. Splitting 400 concurrency across 4 pods recovered throughput to ~1.5 img/s, a 21x improvement.

### What caused the first major bottleneck?

NooBaa storage endpoints hit CPU and connection limits before Quay pods. Scaling from 4 to 6 endpoints more than doubled throughput and recovered from 49% failure to 100% success. Storage must be sized proportionally with Quay.

### How was PID exhaustion resolved?

At 5,000 concurrency on a single pod, the kubelet's podPidsLimit of 16,384 was hit. They raised it to 65,536 using a KubeletConfig CR and set corresponding cgroup limits. This is a platform configuration issue for high-concurrency workloads.

**Tags:** [OpenShift](https://yomu.fyi/topic/openshift), [Performance](https://yomu.fyi/topic/performance), [Red Hat Quay](https://yomu.fyi/topic/red-hat-quay)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/10/29/sizing-red-hat-quay-profiling-components-find-real-limits)
- Source URL: https://developers.redhat.com/articles/2026/10/29/sizing-red-hat-quay-profiling-components-find-real-limits
- Ingested by Yomu: 2026-10-29T15:00:15.042Z

[Read original post](https://developers.redhat.com/articles/2026/10/29/sizing-red-hat-quay-profiling-components-find-real-limits)
