The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Indexing the Data Lake for Online Point Queries
Online services and AI agents require low-latency point queries over massive datasets that are economically impractical to store entirely in key-value systems like Bigtable or DynamoDB. While cloud object storage offers rapid request latency, traditional distributed query engines like Trino and BigQuery introduce seconds of planning and scheduling overhead along with dependent file reads. Random Access Parquet addresses this limitation by using an external index mapping keys directly to exact Parquet files and row locations. Readers execute O(1) index lookups and issue parallel ranged reads to fetch only the required bytes without scanning footers or metadata. By running directly against existing data lake files, this architecture eliminates separate storage copies and allows shared analytical data to serve interactive online lookups.
AI in healthcare: applications and best practices
Healthcare AI applies artificial intelligence, machine learning, deep learning, and generative AI to clinical, administrative, and research workflows, using patient data, EHRs, medical imaging, and clinical documentation to support decisions and operational efficiency. It distinguishes AI models from algorithms and generative models, and describes supervised classifiers, time-series, natural-language-processing, and computer-vision systems used across healthcare. The data section emphasizes standardized models, consistent coding, interoperability through APIs and FHIR, and unified data lakehouse architecture, noting reported EHR prediction accuracy of 70-72% that varies by condition, data quality, and population. For generative documentation, audio transcription, large language models, templates, and Retrieval-Augmented Generation can draft notes, but clinician review remains required because fluent output may contain factual errors. It also covers imaging AI, privacy practices, FDA and European AI Act obligations, and the expectation that clinical AI will augment rather than replace professional judgment.
Automate custom PII detection at scale with Amazon Macie and Step Functions
Regulated organizations ingesting large data volumes must detect and classify standard and domain-specific personally identifiable information to satisfy compliance mandates like GDPR, HIPAA, CCPA, and PCI DSS. To eliminate manual inspection, an event-driven architecture orchestrates Amazon Macie scans for files landing in Amazon S3. Amazon EventBridge captures upload events and invokes AWS Step Functions, which manages Lambda functions to stage objects, trigger Macie classification jobs with custom regex identifiers, and poll for results. The pipeline separates data across a three-bucket pattern representing raw, staged, and scanned states to prevent mixing unprocessed data with validated assets. Finally, the workflow exports timestamped compliance reports in CSV and JSON formats while alerting security teams via Amazon SNS on high-severity findings.
How Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale
Dow built the Carbon Footprint Ledger (CFL) on Databricks to provide a transparent, verifiable way to track and communicate product carbon footprints amid customer Scope 3 commitments and tighter environmental disclosure requirements. The system combines carbon-accounting methodology assured against ISO 14067 and the GHG Protocol Product Standard with a calculation engine and certificate-issuing ledger that maintains residual balances. Within Dow’s Integrated Data Hub, Apache Spark unifies supply-chain, sustainability, operations, and other source data, while Unity Catalog governs access and Delta tables provide ACID transactions, time travel, and efficient upserts. An advanced optimization model, developed and deployed with MLflow, identifies lowest-greenhouse-gas production pathways and sends versioned, monitored results through the pipeline with lineage tracking. The implementation reduced PCF processing from weeks to a fraction of that, enabled portfolio-scale calculations, and supports verifiable certificates and commercial use of decarbonization investments.
Why R&D Data Belongs in the Lakehouse - and Why Agents Need It There
cellcentric describes how it built a governed lakehouse foundation for R&D data so industrial AI agents can reason over engineering context, not isolated source-system records. On Azure and Databricks, the Data Hub combines Unity Catalog governance, Lakehouse Federation, Delta Sharing, Declarative Automation Bundles, and an MCP interface for agents. Its Fuel Cell Passport unifies five enterprise systems, models seven hierarchy levels, refreshes daily, and uses a state-based temporal model for point-in-time configuration and rework-history questions. The platform treats context coverage as a data-quality metric, with 27 published products averaging 90% column-comment coverage, while authenticated user identity and Unity Catalog authorization constrain agent access. For many R&D and process-development workflows, work that once took weeks now ships in days, though complex multi-source investigations still require substantial domain review.
Supabase Pipelines is now in Public Alpha
Supabase Pipelines has moved into public alpha as a service for replicating Supabase Postgres data to external analytical systems in near real time. The release adds automatic application of supported schema changes, faster initial synchronization through parallelism, and a destination request form for ClickHouse, Snowflake, and DuckLake; Google BigQuery is the first destination available to everyone in public alpha. Powered by the open-source Rust-based Supabase ETL, Pipelines uses Postgres logical replication to copy existing tables, stream inserts, updates, deletes, and truncates, and resume from the last acknowledged position after interruptions. It provides at-least-once delivery and managed pipeline controls, while remaining an alpha product with ongoing work on throughput, batching, backpressure, broader schema support, and additional destinations.
Grabette: an open system to record robot-manipulation data
Grabette is an open, low-cost system for recording robot-manipulation data without requiring a robot, lab, or teleoperation rig. It uses a handheld gripper with a wide fisheye observation camera, an RGBD tracking camera, an IMU, and gripper encoders to capture synchronized demonstrations and recover camera-local 6-DoF Cartesian pose plus gripper state. Users save episodes on a Raspberry Pi, then select them in a browser dashboard for upload, RTAB-MAP SLAM, trajectory verification, and conversion into a LeRobot dataset on the Hugging Face Hub. The release also includes a motorized Gripette gripper and an example that trains a Diffusion Policy on 200 demonstrations and evaluates it on an OpenArm 7-DoF arm via gRPC, demonstrating the recording-to-training workflow.
Content Ingestion & Podcast Video Incident Report
On June 24, video transcoding infrastructure reached maximum capacity, causing hours of publishing delays for Spotify video podcast episodes. Four converging factors drove the disruption: insufficient headroom for content spikes, an active routine batch processing job, higher compute costs from a recent video quality upgrade, and a scheduling bug reducing hardware throughput by roughly 10%. Creators exacerbated the backlog by re-uploading unconfirmed episodes, while four hours elapsed between the earliest alerts and formal incident response. Engineers halted the batch job, patched the scheduling defect, and brought an extra compute cluster online to clear all backlogs overnight. Spotify subsequently increased transcoding capacity by approximately 67% and initiated cross-team investments in workload prioritization, rate limiting, and earlier capacity alerting.
How our universal content processing platform Riviera evolved for AI and beyond
Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products.
From experiment to insight: how Dotmatics Luma and Databricks make AI-ready science a reality
Scientific workflows generate data across instruments, teams, and partner networks, but siloed systems can strip away metadata, lineage, and experimental context. Luma, Dotmatics’ scientific intelligence platform, captures outputs continuously and harmonizes them into structured, FAIR-compliant records, while Databricks supplies scalable storage, governance, and enterprise data and AI infrastructure. Running natively on Databricks, Luma preserves a continuous digital thread across experiment design, acquisition, analysis, reporting, and sharing, with Delta Sharing supporting governed exchange with collaborators. Chromatography illustrates the approach: connectivity across vendor systems and Virscidian’s Analytical Studio automate LC/MS processing while retaining context and adding dashboards, registration, and management tools. In one pharmaceutical deployment, Luma connected about 1,500 of more than 5,000 instruments across four LC/MS vendors, enabling cross-vendor performance trending, unified purity analysis, and a foundation for AI and machine learning.
Integrating Context-Aware Video AI Agents Into Enterprise Workflows
Video analytics AI agents often struggle to integrate into enterprise workflows because video platforms, organizational knowledge bases, and operational systems are isolated. To resolve this, NVIDIA NemoClaw orchestrates the Metropolis Video Search and Summarization blueprint alongside the Retrieval-Augmented Generation blueprint into an automated pipeline. Upfront human-in-the-loop interactions collect analysis parameters, objects of interest, and relevant retrieval queries before video ingestion begins. The system combines hierarchical video summaries with contextual enterprise reference documents to generate structured reports with timestamps and citations. Finally, NemoClaw executes programmatic downstream tasks, such as creating prioritized Jira tickets or dispatching industrial maintenance work orders directly into management tools.
Introducing Apache Spark 4.2
Apache Spark 4.2 extends the engine’s role in modern data and AI workloads with governed metrics, vector and top-K primitives, Arrow-first Python execution, native change data capture, and stronger streaming foundations. Metric views provide shared business definitions, while Spark Connect uses gRPC and Arrow to let remote clients submit logical plans without a full Spark runtime. Spark SQL adds vector similarity functions, NEAREST BY, geospatial types, sketches, and time-series features; Python interoperability includes Arrow UDFs and can move Spark DataFrames into supported Arrow-native tools without copying or serializing underlying data. Spark Declarative Pipelines adds Auto CDC for SCD Type 1 targets, while Data Source V2 standardizes change streams through CHANGES and expands row-level operations and schema evolution. The release also includes operational updates such as Web UI modernization, Kubernetes improvements, JDK 25 support, and dependency upgrades.
July 2026 newsletter
The July 2026 newsletter rounds up ClickHouse releases, community projects, benchmarks, engineering work, and upcoming events in real-time data warehousing. Version 26.6 adds hypothetical skip indexes, cascading refreshable materialized-view dependencies, experimental continuous queries, AI embedding functions, PNG output, CLI documentation, and geospatial improvements. A CostBench comparison running the full analytics path over 28 hours found better cost-performance for ClickHouse Cloud than Snowflake Interactive Tables, which required a continuously running refresh warehouse for pre-aggregated data. Other features cover conversational analytics with ClickHouse Cloud, LibreChat, and MCP; Postgres change-data capture through Debezium and Kafka; local agent queries with chDB; high-concurrency tuning; and Silk, whose fiber runtime targets lower I/O tail latency. Quick reads report ingestion, caching, security, and multi-tenant SQL developments.
Replacing the HDB: ClickHouse for historical ticker data
Historical market-data systems separate a durable archive from the real-time trading path, but the archive still must support research, backtesting, transaction-cost analysis, and regulatory reconstruction. The worked example uses Binance’s public archive to load trades and quotes for five USD-M perpetual-futures symbols into ClickHouse, with a MergeTree table ordered by (symbol, ts) and codecs including DoubleDelta and ZSTD. SQL queries calculate VWAP, hourly OHLC candles, and as-of joins, while ClickHouse’s sparse primary index skips most granules for symbol-and-time filters. Across ten months, 16.2 billion rows compress to 76 GiB on disk; the post reports a ClickHouse Cloud cost of $1.88 per month and argues that typical symbol-and-time queries retain low latency at that scale, while broader scans read more data.
Training Orchestrator: Unifying Model Training at Yelp
Yelp developed Training Orchestrator to standardize machine learning pipeline execution and eliminate divergent training scripts across its applied engineering teams. Previously, monolithic Spark scripts coupled training logic directly to cluster runtimes, preventing local testing, slowing debug cycles, and causing inconsistent configuration drift across projects. The new framework introduces a declarative configuration layer backed by Pydantic models, enabling static type checking and step schema validation before runtime execution starts. Training workflows are defined as modular step hierarchies that the orchestration engine compiles into directed acyclic graphs for execution in topological order. By injecting Spark and MLflow contexts into steps, the architecture ensures reproducible tracking, automated Slack alerts, and parity across local development and production environments.
The agentic marketing stack starts with the data layer
The piece argues that agentic marketing depends on modernizing the data and infrastructure foundation first, rather than starting with isolated AI use cases. At Acxiom, moving core products and client solutions from on-premises Hadoop to Databricks reportedly improved workload runtimes by 80 to 90 percent, reducing jobs that took 50+ hours or as long as 90+ hours to 2–3 hours. The migration also reduced manual pipeline and infrastructure work, freeing engineering capacity for products and client outcomes. Those capabilities now support agentic audience planning, media buying, campaign activation, performance analytics, code generation, testing, and ad variation, with a goal of connecting the full marketing value chain. Because workflows handle PII, generated content passes through legal approval, defined controls, and human oversight; Acxiom is also embedding its data in client platforms and privacy-safe clean rooms to make decisions more transparent and native to customer environments.
One Driver, One Format, Every Language: ADBC
ClickHouse announces an official ADBC driver, a modern Arrow-native alternative to ODBC and JDBC for analytics and AI applications. It provides end-to-end columnar movement: query results leave ClickHouse as Apache Arrow and arrive in applications without a row-oriented conversion or staging step. A single Rust-built implementation connects through ClickHouse's HTTP interface, supports TLS, and serves languages and tools with ADBC bindings, including Ruby, R, and C. Users can install precompiled, signed binaries through the ADBC Driver Foundry and connect with a clickhouse:// URI; dbt Fusion distributes the driver directly. The driver also supports streaming Arrow inserts, while schema discovery, improved bulk ingest, usage examples, and JSON support remain on the roadmap.
Synthetic Data Generation for Financial AI Research with NVIDIA NeMo
Real-world financial natural language processing datasets often overrepresent common events like earnings while underrepresenting rarer occurrences such as credit-rating changes. To address this imbalance, an iterative generation pipeline produced over 500,000 unique headlines across 13 categories using NVIDIA NeMo Data Designer, NeMo Curator, and Nemotron 3 Nano. Instead of relying on a single large batch that yielded 65 percent near-duplicates, the workflow executes iterative cycles of category-weighted generation, rule-based filtering, global semantic deduplication, and dynamic category reweighting. Farthest-from-centroid few-shot example selection guided subsequent rounds toward novel semantic spaces across 82 total iterations. Executed over approximately six days on an eight-way NVIDIA B200 node, the resulting corpus enabled fine-tuning compact student language models to achieve 95 percent of teacher model performance.
Running Low-Latency Analytical Workloads with GPU-Accelerated Presto on NVIDIA GB200 NVL72
GPU-accelerated Presto integrates NVIDIA cuDF algorithms and NVLink to deliver low-latency analytical query execution on massive datasets. Benchmarks derived from TPC-H across scale factors 1K and 3K reveal that a single NVIDIA DGX B200 system with eight GPUs outperforms 8- to 10-node Intel Xeon clusters by up to 8.2x. Scaling out across multinode NVIDIA GB200 NVL72 systems paired with IBM Storage Scale achieved additional performance gains. Enabling NVIDIA GPUDirect Storage, tuning I/O parameters, adjusting thread counts, and applying query rebatching delivered a cumulative 64% reduction in query runtimes. Topology-aware GPUDirect Storage cold reads proved roughly twice as fast as standard POSIX reads by avoiding CPU memory staging and NUMA boundary crossing penalties.
AI needs the best-of-breed data stack: Postgres and ClickHouse
AI-native applications generate large volumes of operational data, while agents and users increasingly require real-time answers from that data. This pressure raises both query concurrency and reliability requirements, with a cited sample of companies using Postgres and ClickHouse showing more than 1,000% data growth in six months and over 85 TB added. The proposed stack assigns Postgres low-latency transactional workloads and ClickHouse real-time analytics, connecting them through PeerDB and ClickPipes CDC with single-digit-second replication, pg_clickhouse query pushdown, and pg_stat_ch telemetry streaming. The conclusion is that AI makes the OLTP/OLAP distinction more important, favoring an open best-of-breed stack over a single platform while preserving each database's specialized role.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.