Building high-performance full-text search for object storage

Summary

ClickHouse redesigned its native full-text index to address latency bottlenecks caused by random read patterns on remote object storage. Instead of using a Finite State Transducer dictionary, the new architecture organizes indexed tokens into fixed-size sorted blocks compressed with front-coding. Each data part separates index data across a small in-memory sparse index file, a sequential dictionary file, and a posting list file. This layout allows the query engine to pinpoint token locations with minimal random I/O and resolve multiple full-text functions directly from index files. Consequently, ClickHouse Cloud maintains predictable search latency across shared object storage while matching local disk performance and scaling through parallel query execution.

Takeaways

1. 1

   The text index splits data across three separate files per part: the dictionary file (text_idx.dct.idx), the sparse dictionary index file (text_idx.idx), and the posting list file (text_idx.pst.idx).

2. 2

   Dictionary tokens are grouped into fixed-size blocks of 512 entries by default and compressed using front-coding to replace full strings with prefix lengths and suffixes.

3. 3

   Negation functions such as !=, NOT IN, and NOT LIKE are unsupported because the text index only tracks terms and posting lists that exist in the table.