Red HatJann Fischer, Pradeep Surisetty, Sho Weimer, Jan Hutař, Alex Krzos, Mike NgBenchmark
Red Hat conducted a performance and scale test of the argocd-agent pull architecture for GitOps, using 42 physical machines, 500 spoke clusters, and software including OpenShift 4.21 and OpenShift GitOps 1.20. The pull model decentralizes reconciliation to lightweight agents on each managed cluster, which pull configurations from a central hub, improving security by eliminating inbound administrative ports and reducing cross-cluster network traffic by up to 80%. In the bootstrap scenario, generating and syncing 31,200 applications across 480 clusters completed in 13 minutes 3 seconds, while the sync-to-commit scenario, which includes a polling delay, took 13 minutes. The test revealed bottlenecks including Kubernetes API limits on the hub, informer processing slowdowns causing watch restarts, event writer retry issues under load, and ApplicationSet controller RAM consumption. The team also deleted all 31,200 applications in 4 minutes 53 seconds using cascading deletion, and the insights are informing future OpenShift GitOps releases.
Traditional push-based GitOps requires the hub to hold high-level credentials for every cluster and creates bottlenecks as the hub observes all resources. The pull model with Argo CD Agent decentralizes reconciliation to agents on each spoke cluster, improving security and network efficiency.
Deploy a lightweight Argo CD Agent on each managed cluster that pulls configuration from a central hub. The hub generates ApplicationSets and Applications, and agents reconcile locally. Testing used 500 spoke clusters with up to 31,200 applications and 3.12 million resources across two scenarios: bootstrap and sync-to-commit.
The pull architecture reduced cross-cluster network traffic by up to 80% compared to the traditional push model.
Bootstrap sync of 31,200 applications across 480 clusters completed in 13 minutes 3 seconds; sync-to-commit took 13 minutes.
The Kubernetes API and etcd on the hub became the primary rate-limiting factor at scale, requiring QPS limits to avoid overwhelming the API.
In push GitOps, a central hub pushes configurations to spoke clusters, requiring high-level credentials for each spoke. In pull GitOps, a lightweight agent on each spoke pulls configurations from the hub, improving security and reducing network traffic.
The test used up to 480 spoke clusters with 31,200 applications and 3.12 million resources. The environment included 42 physical machines and 500 virtual machine clusters.
The main bottlenecks were Kubernetes API limits on the hub, informer processing slowdowns causing watch restarts, event writer retry issues under load, and ApplicationSet controller RAM consumption.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Buildkite splits CI into a hosted control plane and agents you run on your own machines. This comparison weighs that self-hosted agent model against fully hosted runners so you can pick the right fleet.
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
