---
title: "Securely indexing large codebases"
description: "Cursor describes a secure way to index large codebases for semantic search, where naive processing can delay availability for hours. The system builds a Merkle tree of file and directory hashes, synchronizes only divergent branches, and caches embeddings by chunk content to reduce repeated work. For new users, a client-generated simhash finds a similar index within the same team or user's indexes, which is copied while background synchronization reconciles differences. Content proofs derived from the client's full Merkle tree filter search results so users see only files they can prove they possess. Reported time-to-first-query falls from 7.87 seconds to 525 milliseconds for the median repository, and from 4.03 hours to 21 seconds at the 99th percentile."
---

# Securely indexing large codebases

[Cursor](https://yomu.fyi/company/cursor) · Jeremy Stribling · Jan 27, 2026

**Type:** Problem & solution

## Summary

Cursor describes a secure way to index large codebases for semantic search, where naive processing can delay availability for hours. The system builds a Merkle tree of file and directory hashes, synchronizes only divergent branches, and caches embeddings by chunk content to reduce repeated work. For new users, a client-generated simhash finds a similar index within the same team or user's indexes, which is copied while background synchronization reconciles differences. Content proofs derived from the client's full Merkle tree filter search results so users see only files they can prove they possess. Reported time-to-first-query falls from 7.87 seconds to 525 milliseconds for the median repository, and from 4.03 hours to 21 seconds at the 99th percentile.

## Context

Large repositories with tens of thousands of files can take hours to index naively, and semantic search is unavailable until at least 80% of the indexing work is complete. New users working from near-identical copies of an existing codebase would otherwise repeat that processing.

## Approach / What changed

Cursor uses Merkle trees to identify changed files and directories, caches embeddings by syntactic chunk content, and derives a simhash to locate a similar existing index within the same team or user's indexes. It copies that index in the background and uses content proofs from the client's Merkle tree to filter search results until synchronization completes.

## Takeaways

- Merkle-tree comparisons synchronize only branches whose hashes differ; unchanged entries are skipped, while missing client or server entries are deleted or added.
- A simhash is searched against other current index simhashes in a vector database, and matches above a threshold can provide the initial index for a new codebase.
- Content proofs prevent results from being returned for files the client cannot prove it has, while background synchronization reconciles the copied index with local differences.

**Tags:** [Cursor](https://yomu.fyi/topic/cursor), [Search](https://yomu.fyi/topic/search), [Security](https://yomu.fyi/topic/security)

- Source: [Cursor](https://cursor.com/blog/secure-codebase-indexing)
- Source URL: https://cursor.com/blog/secure-codebase-indexing
- Ingested by Yomu: 2026-09-02T11:44:39.954Z

[Read original post](https://cursor.com/blog/secure-codebase-indexing)