---
title: "Improving agent with semantic search"
description: "Cursor describes semantic search as a complement to grep for helping coding agents locate relevant code in large codebases. Its evaluation used Cursor Context Bench across widely used models, comparing tool sets with and without semantic search, and found an average 12.5% accuracy improvement, with gains ranging from 6.5% to 23.5%. In an online A/B test, semantic search increased code retention by 0.3% overall and 2.6% on codebases with at least 1,000 files, while its absence produced a 2.2% increase in dissatisfied follow-up requests. The retrieval system uses a custom embedding model trained from agent session traces, with an LLM ranking helpful content and training aligning embedding similarity scores to those rankings. The conclusion is that combining semantic search with grep produces the best outcomes, especially for large codebases."
---

# Improving agent with semantic search

[Cursor](https://yomu.fyi/company/cursor) · Stefan Heule, Emily Jia & Naman Jain · Nov 6, 2025

**Type:** Benchmark

## Summary

Cursor describes semantic search as a complement to grep for helping coding agents locate relevant code in large codebases. Its evaluation used Cursor Context Bench across widely used models, comparing tool sets with and without semantic search, and found an average 12.5% accuracy improvement, with gains ranging from 6.5% to 23.5%. In an online A/B test, semantic search increased code retention by 0.3% overall and 2.6% on codebases with at least 1,000 files, while its absence produced a 2.2% increase in dissatisfied follow-up requests. The retrieval system uses a custom embedding model trained from agent session traces, with an LLM ranking helpful content and training aligning embedding similarity scores to those rankings. The conclusion is that combining semantic search with grep produces the best outcomes, especially for large codebases.

## Context

Coding agents need to understand a codebase by reading files and searching for relevant information. The work evaluates whether semantic search improves on traditional regex-based tools such as grep, particularly for large codebases and real user tasks.

## Approach / What changed

A custom embedding model is trained from agent session traces. An LLM ranks the content that would have been most helpful at each step, and the embedding model is trained to align its similarity scores with those rankings. The approach is evaluated through Cursor Context Bench and an online A/B test comparing agents with and without semantic search.

## Takeaways

- Semantic search improved answer accuracy by an average of 12.5% in offline evaluations, with model-dependent gains from 6.5% to 23.5%.
- In the A/B test, code retention rose 0.3% overall and 2.6% for codebases with 1,000 or more files when semantic search was available.
- The strongest results came from combining semantic search with grep rather than relying exclusively on either search method.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Search](https://yomu.fyi/topic/search)

- Source: [Cursor](https://cursor.com/blog/semsearch)
- Source URL: https://cursor.com/blog/semsearch
- Ingested by Yomu: 2026-09-02T11:45:15.342Z

[Read original post](https://cursor.com/blog/semsearch)