Fast regex search: indexing text for agent tools

Summary

Agentic coding has revived grep as a critical way to gather source-code context, but ripgrep can stall on large monorepos because it scans every file. The proposed remedy is a local inverted index of overlapping character trigrams, with regular expressions parsed into required and alternative trigram constraints that narrow candidate files before exact matching. The index stores posting lists in one disk file and a sorted hash-to-offset lookup table in another, memory-mapping only the table for binary-search lookups; hash collisions can broaden results but cannot produce incorrect ones. Freshness is prioritized because agents often search for text they have just written, and the client index layers user and agent changes over a Git commit. The reported conclusion is that instant local search creates qualitative workflow improvements, especially for bug investigation and large Enterprise repositories.

Takeaways

1. 1

   Trigrams are used as inverted-index tokens because bigrams create very large posting lists, while quadgrams create billions of possible keys; trigrams provide a middle ground.

2. 2

   Hashing n-grams in the lookup table saves space: a collision may broaden a posting list, but the excerpt states it cannot produce incorrect results.

3. 3

   Fresh local indexing matters more for regex search than semantic indexing because failing to find text that an agent just wrote can trigger wasted tokens and a misleading investigation.