Improving agent with semantic search

Summary

Cursor describes semantic search as a complement to grep for helping coding agents locate relevant code in large codebases. Its evaluation used Cursor Context Bench across widely used models, comparing tool sets with and without semantic search, and found an average 12.5% accuracy improvement, with gains ranging from 6.5% to 23.5%. In an online A/B test, semantic search increased code retention by 0.3% overall and 2.6% on codebases with at least 1,000 files, while its absence produced a 2.2% increase in dissatisfied follow-up requests. The retrieval system uses a custom embedding model trained from agent session traces, with an LLM ranking helpful content and training aligning embedding similarity scores to those rankings. The conclusion is that combining semantic search with grep produces the best outcomes, especially for large codebases.

Takeaways

1. 1

   Semantic search improved answer accuracy by an average of 12.5% in offline evaluations, with model-dependent gains from 6.5% to 23.5%.

2. 2

   In the A/B test, code retention rose 0.3% overall and 2.6% for codebases with 1,000 or more files when semantic search was available.

3. 3

   The strongest results came from combining semantic search with grep rather than relying exclusively on either search method.