Continually improving our agent harness

Summary

Cursor describes its agent harness as an iterative product process: vision-driven hypotheses are tested through offline evals, real-world A/B experiments, and instrumentation. Its context strategy has shifted from static codebase context and guardrails toward dynamic retrieval as models improve, while model-specific prompts and tools match training conventions. Quality measurement combines CursorBench with latency, token efficiency, tool-call errors, cache behavior, code Keep Rate, and language-model analysis of user responses. The team classifies expected and unknown tool errors, uses per-tool and per-model anomaly baselines, and automates log investigations and backlog tickets; one sprint reduced unexpected tool-call errors by an order of magnitude. The harness also supports model switching and subagents, although switching can cause cache misses and summary information loss, and the post argues that multi-agent orchestration will make harness engineering increasingly central.

Takeaways

1. 1

   Code Keep Rate and language-model analysis of user responses complement operational metrics such as latency, token efficiency, tool-call count, and cache hit rate when assessing agent quality.

2. 2

   Unknown tool errors are treated as harness bugs, while expected errors are classified by cause and monitored with per-tool, per-model anomaly baselines to detect regressions.

3. 3

   Switching models mid-conversation changes prompts and tools, can cause provider- and model-specific cache misses, and may lose details when the conversation is summarized; subagents provide a fresh-context alternative.