GitTrends: A Google Trends style view of the GitHub ecosystem

Lionel PalacinMar 10, 2026

Source: Clickhouse
Published: Mar 10, 2026
Added to Yomu: Aug 28, 2026

Summary

GitHub produces billions of events across issues, pull requests, and comments, but turning this vast stream into a real-time trends analyzer presents significant technical challenges. GitTrends is an open-source demo application designed to search and compare technology keywords across nearly ten billion GitHub events. Built to showcase the full-text search capabilities recently released in ClickHouse, the application indexes raw text without pre-computed answers or intermediate data transformation. ClickHouse executes full-text searches and aggregations within a single engine in one pass, eliminating cross-system joins and data movement. The demo includes live comparisons toggling between full-text search, bloom filters, and full table scans to illustrate query performance at scale.

Context

Capturing GitHub's continuous stream of billions of issues, pull requests, and comments to analyze technology lifecycles and keyword trends in real time without heavy pre-computation is difficult.

Approach / What changed

ClickHouse indexes raw text directly using its new full-text search index feature, enabling simultaneous raw text searches and aggregations across nearly 10 billion GitHub events in a single query pass without cross-system data movement.

Takeaways

ClickHouse combines full-text search and analytical aggregation in the same engine, executing queries in a single pass without moving data between systems.
GitTrends lets users toggle between full-text search, bloom filters, and full table scans to compare query performance on raw GitHub event data.
GitHub Events API payload changes starting in October 2025 altered certain fields, affecting historical trend accuracy across the complete dataset.

Open Source Performance Search

Read original post

GitTrends: A Google Trends style view of the GitHub ecosystem

Lionel PalacinMar 10, 2026

Source: Clickhouse
Published: Mar 10, 2026
Added to Yomu: Aug 28, 2026

Summary

Context

Capturing GitHub's continuous stream of billions of issues, pull requests, and comments to analyze technology lifecycles and keyword trends in real time without heavy pre-computation is difficult.

Approach / What changed

Takeaways

ClickHouse combines full-text search and analytical aggregation in the same engine, executing queries in a single pass without moving data between systems.
GitTrends lets users toggle between full-text search, bloom filters, and full table scans to compare query performance on raw GitHub event data.
GitHub Events API payload changes starting in October 2025 altered certain fields, affecting historical trend accuracy across the complete dataset.

Open Source Performance Search

Read original post

Related reading

Clickhouse · Mar 10, 2026
Announcing General Availability of ClickHouse Full-text Search
ClickHouse has reached general availability for its native Full-text Search feature, designed to accelerate token-based text filtering across large datasets. The capability relies on deterministic inverted indexes rather than probabilistic Bloom filter skip indexes, mapping individual tokens directly to row numbers. It supports plain strings, string arrays, and map keys or values while allowing configurable pre-processing and tokenization using standard SQL expressions. The system does not implement relevance scoring or store positional data for ranking, focusing instead on rapid filtering paired with large-scale analytical aggregations. Testing shows significant reductions in scanned granules alongside substantial latency improvements, though text indexes require more storage than Bloom filters.
Melvyn Peignon
Clickhouse · Mar 19, 2026
March 2026 newsletter
The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl_only_drop_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.
Mark Needham
Clickhouse · Apr 23, 2026
Do you still need Elasticsearch for log analytics? ClickHouse says no.
ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.
Tom Schreiber, Lionel Palacin
Clickhouse · Mar 24, 2026
Building high-performance full-text search for object storage
ClickHouse redesigned its native full-text index to address latency bottlenecks caused by random read patterns on remote object storage. Instead of using a Finite State Transducer dictionary, the new architecture organizes indexed tokens into fixed-size sorted blocks compressed with front-coding. Each data part separates index data across a small in-memory sparse index file, a sequential dictionary file, and a posting list file. This layout allows the query engine to pinpoint token locations with minimal random I/O and resolve multiple full-text functions directly from index files. Consequently, ClickHouse Cloud maintains predictable search latency across shared object storage while matching local disk performance and scaling through parallel query execution.
Elmi Ahmadov, Jimmy Aguilar, and George Larionov