Pass@1 vs. pass@k: Two Ways to Read One Benchmark
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
MongoDBBenchmark
A benchmark of the Icon Payments Framework (IPF), trusted by Citi, NatWest, and BNP Paribas, on MongoDB Atlas demonstrated extreme performance and resilience for instant payment processing. The system sustained 6,000 end-to-end payments per second, requiring approximately 430,000 simultaneous database operations, with a mean end-to-end latency of 0.51 seconds and persistence latency below 30 milliseconds. Upgrading from MongoDB Atlas 8.0 to 8.3 reduced persistence latency by roughly 50%, from 40 to 20 milliseconds at 4,000 payments per second, and throughput scaled near-linearly across the 500 to 6,000 range while latency remained flat until peak and hot-shard CPU utilization sat at about half. During a primary failover, a new node was elected in five seconds with zero data loss and no application restart. An application node failure recovered to 3,500 transactions per second in 90 seconds with zero data loss.
As instant payment rails replace legacy batch systems worldwide, banks need public proof that modern data architectures can scale without compromising resilience. The Icon Payments Framework (IPF), trusted by Tier 1 banks including Citi, NatWest, and BNP Paribas, was tested on MongoDB Atlas under extreme load and failure conditions.
A rigorous ladder test evaluated key performance metrics across a range of 500 to 6,000 payment transactions per second, with simulated disaster conditions including primary failover and application node failure.
MongoDB Atlas on the IPF sustained 6,000 end-to-end payment transactions per second with a mean latency of 0.51 seconds and 430,000 database operations per second.
Upgrading from MongoDB Atlas 8.0 to 8.3 reduced persistence latency by approximately 50%, e.g., from 40ms to 20ms at 4,000 transactions per second.
During a primary failover, a new node was elected in five seconds with zero data loss and no application restart; an application node failure recovered to 3,500 transactions per second in 90 seconds with zero data loss.
The system achieved 6,000 end-to-end payment transactions per second, with 430,000 database operations per second and sub-second latency.
During a primary failover, a new node was elected in five seconds with zero data loss and no application restart. An application node failure temporarily reduced throughput but recovered to 3,500 transactions per second in 90 seconds with zero data loss.
Upgrading from version 8.0 to 8.3 reduced persistence latency by approximately 50%, for example from 40ms to 20ms at 4,000 transactions per second.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.