BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
Loading…
Topic
BigQuery
7 posts about BigQuery. Every summary links to the original.
- 7 posts
- 6 companies
- 1 explainer
See the idea
BigQuery explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Three layers, one repo: a tour of google/adk-recipes
The google/adk-recipes repository serves as the single source of truth for ADK content across Google Cloud’s Agent Platform, including the Agents CLI, Agent Garden, and Gemini Enterprise app. It is organized into three top-level folders: core/ with reference patterns curated by the Agents CLI and DevRel teams, plugins/ with installable vertical skills packaged as Plugin Recipes conforming to the Agent Plugins v1.0.0 spec, and contrib/ with end-to-end deployable services in five languages. Every recipe ships a validated manifest.yaml and an AGENTS.md written primarily for AI coding assistants, and CI enforces the standards on every pull request. The core/ layer features Horizon, a reference harness for long-horizon memory agents that handles cross-session memory, per-user sandboxing, guardrails, and multi-step task orchestration. The retail Plugin Recipe includes product-search and virtual-tryon skills using Vector Search 2.0 and Gemini models, with deployable counterparts in contrib/ that can be surfaced on five deployment personas.
Why BigQuery can’t match ClickHouse Cloud for real-time analytics
ClickHouse Cloud delivered 438× better end-to-end performance per dollar than BigQuery under Capacity pricing, and 512× under On-demand pricing, while continuously ingesting 113.2 billion stock-market quotes. The CostBench methodology measures preparation cost, query cost, and query runtime as data arrives. ClickHouse Cloud led in all three components: preparation cost was 8.5× lower than BigQuery Capacity, normalized query cost was 94× lower, and accumulated query runtime was 50.8× lower. The difference originates in how each system handles incoming data: ClickHouse orders raw data and builds current summaries during inserts, while BigQuery makes new rows queryable before background clustering and materialized view refresh complete. This leaves BigQuery aggregate queries to include rows that refresh has not yet summarized, increasing cost and runtime.
How fast is the web? Explore billions of real-user measurements with BEACON
Cloudflare has released the BEACON dataset, an anonymized collection of billions of real-user performance measurements from 10,000 of the largest websites. The dataset reports Core Web Vitals (LCP, CLS, INP) with full histograms, enabling analysis of any percentile including the long tail. It reveals that WebKit browsers underperform Blink in 46 countries, and that resource-discovery and render-blocking contribute more to slow LCP than download time. Soft navigations in single-page applications load two to three times faster than hard navigations, but landing pages remain heavy. The dataset is publicly available on Google BigQuery and includes differential privacy via the RAPPOR algorithm to protect individual user data.
Three principles for building a vector platform at Thumbtack
Thumbtack describes a vector platform that lets ML engineers stand up production vector search without separately negotiating database access, building ETL, or writing a query service. The design uses an existing Aurora Postgres cluster with pgvector, while Airflow jobs generate embeddings into BigQuery and managed reverse-ETL syncs them into vector tables; a query API validates inputs and returns ranked results. Three principles guide the layers: reuse trusted infrastructure, treat embeddings as ordinary data flowing through established pipelines, and lower adoption costs for subsequent teams. Production examples include semantic customer-support escalation using cosine similarity and an internal knowledge assistant combining vector retrieval, often with keyword search, before a language model responds; planned directions include self-serve provisioning, vector-backed feature retrieval, and framework-neutral agent memory.
Kafka App? There’s a Skill for That
Etsy uses Kafka streams to provide fresh features and embeddings for machine learning models, which infer shopper intent and listing meaning from visits and listing data. Previously, engineers manually assembled Java and YAML applications, selected schemas and base classes, and handled testing and local execution. Etsy introduced conversational Kafka Skills that use language models to interpret requests, pass structured inputs into versioned scaffolding and validation tools, run applications locally, and open pull requests. Telemetry records Skill invocation and pull-request completion through protobuf messages in Kafka and BigQuery, while least-privilege ACLs, repository permissions, and deployment controls constrain agent actions. Initial adoption was low because developers did not know the Skills existed, so Etsy added vocabulary-specific and repository-level entry points; it now has Skills for seven Kafka workflows and is focusing on discoverability.
Interoperability between Unity Catalog and Google BigQuery via catalog federation
Databricks and Google Cloud announce bidirectional catalog federation that lets customers access the same data from Unity Catalog or BigQuery without duplication. The interoperability uses Apache Iceberg REST catalog APIs, enabling both platforms to read and write the same tables through a common open interface. Google Cloud's catalog federation is in preview for reading Unity Catalog-managed tables from BigQuery, while Google Cloud's Lakehouse federation is in private preview for Databricks users to govern and read foreign Iceberg tables. Unity Catalog's policies, fine-grained access controls, and lineage tracking apply to federated data, and users can query it with Genie using natural language. The announcement positions open standards as a path to interoperability across data platforms and previews future catalog metadata exchange through Iceberg REST catalog APIs.
Building Etsy Buyer Profiles with LLMs
Etsy is experimenting with large language models (LLMs) to create anonymous buyer profiles from browsing and purchasing behavior, capturing interests such as styles, categories, and shopping missions. The pipeline combines recent searches, item views, purchases, favorites, session data, and listing information from internal feature-store and BigQuery sources, then prompts an LLM to generate structured profiles with explanations and confidence information. To scale processing, Etsy moved listing data to clustered and partitioned BigQuery tables, shortened input history from two years to nine months, increased batch sizes and compute, added managed concurrency, and used Airflow to stagger tasks. For 10 million users, these changes reduced generation time from 21 days to three and cut estimated cost per million users by 94%; experiments apply profiles to query rewriting and refinement pills.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.