Stack Benchmarking

Summary

Ramp’s Stack is an AI-native accounting suite designed to automate repetitive book-closing work, including reconciliations, variance analysis, data entry, and schedules and accruals. To avoid overfitting to individual design partners, Ramp built a benchmark of synthetic business worlds, realistic accounting tasks, and accountant-written grading criteria, including standard and roll-forward worlds for testing memory transfer. The benchmark contains 237 tasks and 3,469 grading criteria in the analyzed slice, and supports repeated runs to compare models, prompts, tools, skills, harness changes, and memory behavior. Optimization included ablating skills, shrinking a spreadsheet skill from 14,000 to 5,000 characters, and tuning the system against end-to-end task performance rather than narrow evaluations. The resulting Stack system achieved the highest agent performance, with 4% higher accuracy and 3% better Pass@1, while remaining on the latency frontier with GPT 5.4; schedules and accruals were harder for raw models than variance analysis.

Context

Accounting book-closing work involves repetitive, time-sensitive tasks across scattered systems, while design-partner feedback can be slow, imprecise, and prone to overfitting an agent to one customer’s environment.

Approach / What changed

Ramp created an accounting benchmark using synthetic businesses, realistic close tasks, accountant-written grading criteria, standard worlds, and roll-forward worlds. It used repeated task runs to evaluate models, skills, prompts, tools, harness changes, and memory, then pruned and shortened skills and optimized Stack against end-to-end performance.

Takeaways

• The benchmark’s analyzed slice contains 237 tasks and 3,469 grading criteria across standard and roll-forward synthetic business worlds.
• A bookkeeping spreadsheet skill was reduced from 14,000 to 5,000 characters, a 64% reduction, with removed content moved into a separate narrower module.
• Ramp Stack had the highest reported agent performance, with 4% higher accuracy and 3% better Pass@1; schedules and accruals remained harder for raw models than variance analysis.

Related reading