Loading post…
CursorJeremy StriblingProblem & solution
Securely indexing large codebases
Summary
Cursor describes a secure way to index large codebases for semantic search, where naive processing can delay availability for hours. The system builds a Merkle tree of file and directory hashes, synchronizes only divergent branches, and caches embeddings by chunk content to reduce repeated work. For new users, a client-generated simhash finds a similar index within the same team or user's indexes, which is copied while background synchronization reconciles differences. Content proofs derived from the client's full Merkle tree filter search results so users see only files they can prove they possess. Reported time-to-first-query falls from 7.87 seconds to 525 milliseconds for the median repository, and from 4.03 hours to 21 seconds at the 99th percentile.
Context
Large repositories with tens of thousands of files can take hours to index naively, and semantic search is unavailable until at least 80% of the indexing work is complete. New users working from near-identical copies of an existing codebase would otherwise repeat that processing.
Approach / What changed
Cursor uses Merkle trees to identify changed files and directories, caches embeddings by syntactic chunk content, and derives a simhash to locate a similar existing index within the same team or user's indexes. It copies that index in the background and uses content proofs from the client's Merkle tree to filter search results until synchronization completes.
Takeaways
Merkle-tree comparisons synchronize only branches whose hashes differ; unchanged entries are skipped, while missing client or server entries are deleted or added.
A simhash is searched against other current index simhashes in a vector database, and matches above a threshold can provide the initial index for a new codebase.
Content proofs prevent results from being returned for files the client cannot prove it has, while background synchronization reconciles the copied index with local differences.
Source and freshness
- Source
- cursor.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.