Unordered mode for GCS ClickPipes is now available

Marta PaesMar 16, 2026

Source: Clickhouse
Published: Mar 16, 2026
Added to Yomu: Aug 28, 2026

Summary

ClickHouse Cloud has added unordered mode to its Google Cloud Storage (GCS) ClickPipes connector, enabling continuous data ingestion regardless of file naming or arrival sequence. Standard GCS ClickPipes rely on polling every 30 seconds and require files to arrive in strict lexicographical order, which causes out-of-order data and backfills to be ignored. Unordered mode replaces polling by consuming OBJECT_FINALIZE notifications from Google Cloud Pub/Sub whenever objects land in a bucket. State tracking is managed through a metadata store, and the connector automatically reprocesses batches upon failure while maintaining exactly-once insertion guarantees. Setup requires configuring a Pub/Sub topic and bucket notifications via tools like gcloud, followed by configuring the ClickPipe via the console or Terraform provider.

Context

By default, GCS ClickPipes poll buckets every 30 seconds and assume files arrive in sequential lexicographical order. Consequently, files that are lexically smaller than the last ingested file—such as backfills, retries, or late-arriving data—are ignored.

Approach / What changed

Unordered mode utilizes Google Cloud Pub/Sub notifications triggered by OBJECT_FINALIZE events from the GCS bucket. ClickPipes listens to the Pub/Sub subscription, tracks state using a metadata store, handles failures with automatic batch retries, and delivers exactly-once semantics into target ClickHouse Cloud tables.

Takeaways

Default GCS ClickPipes poll every 30 seconds and drop files that are not lexically greater than previously processed files.
Unordered mode relies on GCS bucket notifications publishing OBJECT_FINALIZE events to a Google Cloud Pub/Sub topic.
The connector uses a metadata store to track file state, automatically retries failed batches, and guarantees exactly-once insertion.

Data Pipelines Google Cloud Reliability Streaming

Read original post

Unordered mode for GCS ClickPipes is now available

Marta PaesMar 16, 2026

Source: Clickhouse
Published: Mar 16, 2026
Added to Yomu: Aug 28, 2026

Summary

Context

Approach / What changed

Takeaways

Default GCS ClickPipes poll every 30 seconds and drop files that are not lexically greater than previously processed files.
Unordered mode relies on GCS bucket notifications publishing OBJECT_FINALIZE events to a Google Cloud Pub/Sub topic.
The connector uses a metadata store to track file state, automatically retries failed batches, and guarantees exactly-once insertion.

Data Pipelines Google Cloud Reliability Streaming

Read original post

Related reading

Clickhouse · Apr 27, 2026
ClickHouse at Google Cloud Next '26
ClickHouse announced several product updates and integrations designed to deepen native compatibility with Google Cloud. Migration of ClickHouse Cloud to Google Axion ARM-based processors demonstrated a 30 to 55 percent query speed increase and reduced data load times from 64 to 26 seconds on ClickBench benchmarks. An integration with Google Lakehouse Runtime Catalog enables querying Iceberg tables on Google Cloud Storage via the DataLakeCatalog engine without data movement. Furthermore, Bring Your Own Cloud reached general availability on Google Cloud Marketplace, and ClickPipes added an unordered mode using Pub/Sub notifications for GCS ingestion. Google Antigravity integrated ClickHouse through an MCP server to translate natural language into SQL and generate interactive React charts.
ClickHouse
Clickhouse · Apr 20, 2026
Terraform & OpenAPI for ClickPipes is now Generally Available
ClickHouse Cloud announced the general availability of ClickPipes resources in Terraform and OpenAPI, expanding infrastructure-as-code support across its managed data ingestion platform. The release resolves previous beta gaps by adding full connector coverage, including Change Data Capture for Postgres, MySQL, and MongoDB, alongside support for BigQuery, Azure Blob Storage, and reverse private endpoints. Operators can provision new data ingestion pipelines, pause or resume sync operations, and scale resources programmatically through OpenAPI endpoints or the ClickHouse Terraform provider starting at version 3.14.0. Existing ClickPipes can also be imported into Terraform state using native import blocks and automated configuration generation. Although SSH tunneling remains under active development, nearly all UI capabilities now map cleanly to programmatic interfaces.
Marta Paes
Clickhouse · Apr 21, 2026
ClickHouse expands strategic collaboration with Google Cloud, bringing deeper integration, cloud flexibility, and next-generation performance
ClickHouse announced an expanded strategic collaboration with Google Cloud introducing four core architectural milestones. First, native integration with Google Cloud Lakehouse allows direct sub-second queries across data lakes and ClickHouse deployments without ETL pipelines or data duplication. Second, the Bring Your Own Cloud deployment model lets organizations run managed ClickHouse Cloud inside their own Google Cloud Virtual Private Cloud to retain data sovereignty and network controls. Third, ClickHouse Cloud is migrating automatically to Google's custom Arm-based Axion processors, boosting query throughput and cost efficiency for analytical workloads without requiring application changes. Finally, a new ClickHouse MCP server integration connects ClickHouse with Google Antigravity to enhance developer and analyst workflows.
Tanya Bragin
Clickhouse · Mar 26, 2026
Top 10 best practices tips for ClickHouse
ClickHouse delivers rapid analytical queries across massive datasets, but suboptimal schemas and queries can significantly reduce performance and storage efficiency. Aligning table definitions with ClickHouse storage and execution mechanics resolves these bottlenecks. Key practices include placing low-cardinality columns first in the ORDER BY clause to enable granule pruning, and selecting optimal data types while avoiding Nullable wrappers to cut memory overhead. Developers should treat partitioning as a data management tool for TTL drops or merge engines rather than a generic speed booster, preventing fragmented parts from slowing query execution. Recent improvements like the Analyzer query planner have also made joins viable, though dictionaries remain faster for static lookups.
Yonatan Dolan