Building high-performance full-text search for object storage

Elmi Ahmadov, Jimmy Aguilar, and George LarionovMar 24, 2026

Source: Clickhouse
Published: Mar 24, 2026
Added to Yomu: Aug 28, 2026

Summary

ClickHouse redesigned its native full-text index to address latency bottlenecks caused by random read patterns on remote object storage. Instead of using a Finite State Transducer dictionary, the new architecture organizes indexed tokens into fixed-size sorted blocks compressed with front-coding. Each data part separates index data across a small in-memory sparse index file, a sequential dictionary file, and a posting list file. This layout allows the query engine to pinpoint token locations with minimal random I/O and resolve multiple full-text functions directly from index files. Consequently, ClickHouse Cloud maintains predictable search latency across shared object storage while matching local disk performance and scaling through parallel query execution.

Context

Remote object storage in ClickHouse Cloud provides high throughput but significantly higher access latency than local disks. The previous text index used a Finite State Transducer dictionary structure that relied on small, scattered random reads, causing end-to-end query latency bottlenecks when reading from remote object storage.

Approach / What changed

ClickHouse redesigned the text index into three separate files per data part: an in-memory sparse index, a dictionary file, and a posting list file. The dictionary groups sorted tokens into fixed-size blocks compressed with front-coding, referencing offsets in posting lists. This block-based layout prioritizes sequential access, allowing lookups to navigate via the sparse index and answer queries without reading the raw indexed text column.

Takeaways

The text index splits data across three separate files per part: the dictionary file (text_idx.dct.idx), the sparse dictionary index file (text_idx.idx), and the posting list file (text_idx.pst.idx).
Dictionary tokens are grouped into fixed-size blocks of 512 entries by default and compressed using front-coding to replace full strings with prefix lengths and suffixes.
Negation functions such as !=, NOT IN, and NOT LIKE are unsupported because the text index only tracks terms and posting lists that exist in the table.

Architecture Performance Search

Read original post

Building high-performance full-text search for object storage

Elmi Ahmadov, Jimmy Aguilar, and George LarionovMar 24, 2026

Source: Clickhouse
Published: Mar 24, 2026
Added to Yomu: Aug 28, 2026

Summary

Context

Approach / What changed

Takeaways

The text index splits data across three separate files per part: the dictionary file (text_idx.dct.idx), the sparse dictionary index file (text_idx.idx), and the posting list file (text_idx.pst.idx).
Dictionary tokens are grouped into fixed-size blocks of 512 entries by default and compressed using front-coding to replace full strings with prefix lengths and suffixes.
Negation functions such as !=, NOT IN, and NOT LIKE are unsupported because the text index only tracks terms and posting lists that exist in the table.

Architecture Performance Search

Read original post

Related reading

Clickhouse · Mar 10, 2026
Announcing General Availability of ClickHouse Full-text Search
ClickHouse has reached general availability for its native Full-text Search feature, designed to accelerate token-based text filtering across large datasets. The capability relies on deterministic inverted indexes rather than probabilistic Bloom filter skip indexes, mapping individual tokens directly to row numbers. It supports plain strings, string arrays, and map keys or values while allowing configurable pre-processing and tokenization using standard SQL expressions. The system does not implement relevance scoring or store positional data for ranking, focusing instead on rapid filtering paired with large-scale analytical aggregations. Testing shows significant reductions in scanned granules alongside substantial latency improvements, though text indexes require more storage than Bloom filters.
Melvyn Peignon
Clickhouse · Apr 16, 2026
April 2026 newsletter
ClickHouse 26.3 LTS introduces experimental materialized common table expressions, WebAssembly user-defined functions, and enables asynchronous inserts by default. Alongside the release, Alasdair Brown launched clickhousectl to manage local installations and cloud deployments with built-in agentic development features and authentication safeguards. User case studies show Goldsky accelerating historical blockchain backfills by twelve times through direct ClickHouse reads using Apache Arrow, replacing their prior Kafka and Avro architecture. Additionally, the engineering team redesigned the full-text search index with front-coding compression, a sparse lookup index, and adaptive posting lists to handle expensive random reads on object storage. ClickHouse also previewed community architecture patterns for lightweight SIEM setups using Clickdetect and Wazuh log collection.
Mark Needham
Clickhouse · Apr 23, 2026
Do you still need Elasticsearch for log analytics? ClickHouse says no.
ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.
Tom Schreiber, Lionel Palacin
Clickhouse · Mar 19, 2026
March 2026 newsletter
The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl_only_drop_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.
Mark Needham