The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Granite 4.1 LLMs: How They’re Built
Granite 4.1 is a family of dense decoder-only language models in 3B, 8B, and 30B sizes, trained on approximately 15T tokens. Its five-phase pre-training pipeline moves from broad web-scale data to progressively curated mixtures, increases math and code emphasis, and extends context from 4K to 512K tokens through staged long-context training. Supervised fine-tuning uses about 4.1M curated samples evaluated with LLM-as-Judge and rule-based checks, while reinforcement learning uses on-policy GRPO with DAPO loss. The 8B instruct model matches or surpasses the earlier Granite 4.0-H-Small 32B-A9B MoE model despite its simpler dense architecture and fewer parameters, and the Granite 4.1 models are released under Apache 2.0.
Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI
Thumbtack evaluates generative AI features across customer and service professional experiences to prevent unsupported claims, tone inconsistencies, and safety issues. The engineering organization transitioned from decentralized evaluation efforts to a dedicated cross-functional Evals team that provides shared infrastructure while preserving product team autonomy. System architectures combine rule-based checks, AI-as-a-judge scorers powered by DeepEval, trust and safety assessments, and crowdsourced human reviews. Teams leverage three parallel orchestration paths, including an MLflow-tracked stack, nightly Databricks batch jobs integrated with data warehouses, and multi-layer human-in-the-loop pipelines. Future milestones target centralized LLM judge registries, full task journey evaluations, automated judge-writing workflows, and cost-aware evaluation orchestration.
From months to minutes: Building real-time clinical data pipelines with natural language
Healthcare data teams often spend months integrating EHR systems, normalizing HL7, CCD, and X12, and routing data through intermediary storage before analytics, creating latency and maintenance burden. Databricks and Redox describe a pipeline model that combines Redox MCP Server, natural-language prompts, and Databricks Zerobus Ingest to build integrations inside Databricks and stream clinical data directly into Unity Catalog managed tables. The post says Zerobus provides subsecond latency, while the MCP Server identifies environments, suggests workflow steps, executes integration tasks, and surfaces validation signals such as logs and performance summaries. A demonstrated workflow retrieved a recent patient admission as structured data with a plain-language summary, and Redox writeback can return AI outputs to EHRs for point-of-care action. The same foundation is presented as enabling real-time use cases and Redox Agents built with Databricks Genie Spaces.
Agentic data engineering with Genie Code and Lakeflow
Genie Code is presented as a natural-language assistant for developing, orchestrating, deploying, and debugging data pipelines and jobs. It uses pipeline and job context, including code, configuration, run results, Unity Catalog metadata, lineage, popularity, and code samples, to help engineers discover datasets and understand data flows. Engineers can describe pipelines or jobs, and Genie Code can generate Spark Declarative Pipelines with Bronze, Silver, and Gold layers, sources, transformations, data quality expectations, and outputs, then configure orchestration, schedules, dependencies, Auto Loader, AutoCDC flows, and Declarative Automation Bundles. It also analyzes failures and unexpected row-count or schema changes, proposes cross-file updates with reviewable diffs, and supports extensions through custom instructions, agent skills, and MCP servers. The stated result is faster development and guided debugging while workflows remain aligned with Unity Catalog governance, performance, and data quality standards; future plans include background failure response and cluster right-sizing.
Introducing the OSSCAR Index
Supabase has launched the OSSCAR Index, a quarterly ranking of fastest-growing open source organizations, with the site, data, and scoring code released as open source. It addresses the limits of rankings based on accumulated stars, downloads, or contributors by measuring community growth within two divisions: Emerging organizations below 1,000 GitHub stars at quarter-start and Scaling organizations at or above that threshold. Scores combine normalized net new stars, unique contributors, and npm, PyPI, and Cargo downloads using an L² norm, with minimum thresholds and no penalty for missing or declining signals. The Q1 2026 edition identifies Openclaw as a breakout project, places AI-agent projects prominently in Emerging, and reports global representation; quarterly updates, expanded package coverage, an RFC process, and historical rankings are planned.
ClickHouse at Google Cloud Next '26
ClickHouse announced several product updates and integrations designed to deepen native compatibility with Google Cloud. Migration of ClickHouse Cloud to Google Axion ARM-based processors demonstrated a 30 to 55 percent query speed increase and reduced data load times from 64 to 26 seconds on ClickBench benchmarks. An integration with Google Lakehouse Runtime Catalog enables querying Iceberg tables on Google Cloud Storage via the DataLakeCatalog engine without data movement. Furthermore, Bring Your Own Cloud reached general availability on Google Cloud Marketplace, and ClickPipes added an unordered mode using Pub/Sub notifications for GCS ingestion. Google Antigravity integrated ClickHouse through an MCP server to translate natural language into SQL and generate interactive React charts.
Stop hand-coding change data capture pipelines
Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through __START_AT and __END_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.
Flow generation through natural language: An agentic modeling approach
Shopify built a tool-calling agent for Sidekick that converts natural-language requests into Shopify Flow workflows, addressing the cold-start problem by working backward from thousands of anonymized production workflows. The team generated synthetic queries and tool trajectories, fine-tuned Qwen3-32B, and changed Flow’s nested JSON DSL into an equivalent Python representation connected by a round-trip transpiler. Using Python improved syntactic correctness by 22 points and semantic correctness by 13 points on a 300-example benchmark, while matching production tool names, ordering, responses, and prompts proved essential to avoid degradation. Production evaluation exposed a 35% gap masked by benchmark parity, so Shopify built a weekly feedback loop that scores conversations, filters training data, identifies slice gaps, retrains, and deploys. The resulting agent serves most production traffic, runs 2.2 times faster at 68% lower cost, and outperforms closed models.
Building an Enterprise Data Management Strategy
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
Real-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First
Real-time decisioning for AI agents depends on more than customer records: it requires a customer context layer that captures what people are doing now and connects that behavior to identity. The post positions the data platform as the center of a composable martech architecture, with agents and applications operating on shared data in open formats such as Apache Iceberg and Delta Lake. Snowplow’s described approach places structured event collection, schema validation, enrichment, and identity resolution before data reaches the platform, while operating in the customer’s cloud environment. It distinguishes historical profiles from granular behavioral event streams and argues that agent interaction outcomes should return to the foundation as first-class events. The resulting four-stage loop—collect, resolve and enrich, serve, and learn—supports simultaneous real-time and historical context, with decision quality dependent on source data coherence.
AI Data Transformation Guide for Data Engineers and Data Scientists
AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.
Terraform & OpenAPI for ClickPipes is now Generally Available
ClickHouse Cloud announced the general availability of ClickPipes resources in Terraform and OpenAPI, expanding infrastructure-as-code support across its managed data ingestion platform. The release resolves previous beta gaps by adding full connector coverage, including Change Data Capture for Postgres, MySQL, and MongoDB, alongside support for BigQuery, Azure Blob Storage, and reverse private endpoints. Operators can provision new data ingestion pipelines, pause or resume sync operations, and scale resources programmatically through OpenAPI endpoints or the ClickHouse Terraform provider starting at version 3.14.0. Existing ClickPipes can also be imported into Terraform state using native import blocks and automated configuration generation. Although SSH tunneling remains under active development, nearly all UI capabilities now map cleanly to programmatic interfaces.
“A generational leap”: How Trio unified payment analytics and cut storage by 88% with ClickHouse Cloud
Brazilian payment processor Trio faced severe data challenges after scaling and acquiring PayBrokers, processing over 243 million payments in early 2025 across disparate legacy databases. Because financial reconciliation and compliance demand absolute precision, the engineering team unified their analytics layer by migrating to ClickHouse Cloud. To handle out-of-order and duplicate events from their Redpanda streaming pipelines, Trio implemented sliding-window refreshable materialized views alongside ReplacingMergeTree engines rather than recomputing entire historical datasets. A custom ETL service manages schema drift before inserting clean records into ClickHouse, while batch ingestion is coordinated through Airflow. Consequently, Trio reduced storage consumption by approximately 88% through columnar compression, ingested a 5-billion-row migration without issue, and maintained real-time query performance across more than one billion daily ledger data points.
Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating
dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.
April 2026 newsletter
ClickHouse 26.3 LTS introduces experimental materialized common table expressions, WebAssembly user-defined functions, and enables asynchronous inserts by default. Alongside the release, Alasdair Brown launched clickhousectl to manage local installations and cloud deployments with built-in agentic development features and authentication safeguards. User case studies show Goldsky accelerating historical blockchain backfills by twelve times through direct ClickHouse reads using Apache Arrow, replacing their prior Kafka and Avro architecture. Additionally, the engineering team redesigned the full-text search index with front-coding compression, a sparse lookup index, and adaptive posting lists to handle expensive random reads on object storage. ClickHouse also previewed community architecture patterns for lightweight SIEM setups using Clickdetect and Wazuh log collection.
Getting started with clickhousectl: the ClickHouse CLI for local and cloud
ClickHouse developers often face friction managing local versions, isolated instances, project scaffolding, and Cloud operations without relying on complex shell scripts. The open-source command-line tool clickhousectl addresses this by unifying local environment management and ClickHouse Cloud workflows. Locally, it downloads shared binaries, initializes standardized directory structures, and spins up named, isolated servers with automatic port assignments to facilitate upgrade testing. For cloud workflows, the CLI enforces a split authentication model where browser OAuth provides read-only access and API keys permit state mutations like service creation and scaling. Furthermore, the tool includes Claude Code agent skills covering best practices and deployment workflows, alongside support for CI/CD pipeline automation via environment variables and JSON output.
What is operational analytics?
Operational analytics uses real-time or near-real-time data to monitor day-to-day operations and support immediate decisions, unlike traditional analytics, which generally relies on batch data to explain past performance. Its workflow collects logs, events, clickstreams, telemetry, transactions, and other signals; streams them into a unified platform for cleaning, transformation, and enrichment; analyzes incoming data with rules, models, or anomaly detection; and sends results to dashboards, alerts, or operational applications. This can help teams detect issues earlier, reduce mean time to detect (MTTD) and mean time to respond (MTTR), improve forecasting, coordinate across departments, and act on inventory, customer, or system changes. The approach also requires integrating heterogeneous systems, maintaining data quality and schemas, embedding insights into existing workflows, and building reliable low-latency pipelines, with tools such as Lakeflow, Databricks SQL, and AI and machine learning capabilities presented as examples.
Mintlify boosts NPS 30% and saves 60% with real-time analytics on ClickHouse Cloud
Documentation platform Mintlify experienced major dashboard performance degradation and rate limiting under PostHog as multi-tenant analytics traffic grew rapidly. Because PostHog lacked incremental materialized views, queries scanned full raw datasets and took tens of seconds to load. Mintlify migrated its customer-facing analytics infrastructure to ClickHouse Cloud, ingesting events through Amazon MSK and ClickPipes into a single events table backing incremental materialized views. The team executed the transition over two weeks by dual-writing events and replicating the existing PostHog schema to preserve API surface compatibility. The migration reduced dashboard query latency to sub-one-second, decreased related support bug reports to zero, lifted NPS by an estimated 30%, and lowered analytics infrastructure costs by approximately 60%.
Building real-time product search on Databricks
Real-time product search for an online marketplace must retrieve, filter, rank, and respond in milliseconds while balancing latency, relevance, revenue, and click-through rate. The Databricks architecture divides the pipeline into ingestion, retrieval, and refinement, using Lakeflow pipelines and AI Functions for source processing, AI Search for embeddings and hybrid retrieval, and Lakebase for sub-10ms operational signals. It recommends validating p95/p99 latency, throughput, retrieval quality, and engagement metrics, with MLflow supporting systematic evaluation and caching helping meet strict latency requirements. As an example, FOX Sports used AI Search for thousands of QPS and achieved a 2x improvement in query success rate, combining streaming ingestion, two-phase retrieval, model serving, caching, and trending searches.
The three villains to agentic observability: retention, sampling and rollups
Observability practices increasingly depend on data sampling, short retention windows, and rollups to control storage costs. While human operators historically compensated for these compromises with intuition, AI agents lack implicit operational context and require full-fidelity telemetry to diagnose issues and justify conclusions. Head sampling drops unobserved spans early, tail sampling risks missing implicit non-error anomalies, and aggressive log expiration prevents long-term cyclical trend analysis. Addressing these constraints requires decoupling storage and compute via high-compression object storage and dedicated compute pools. Deploying real-time analytical databases such as ClickHouse enables high-concurrency, low-latency SQL querying, log clustering, and structured in-database summarization necessary for automated reasoning loops.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.