The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Designing effective Genie Agents from a single prompt
Generic agents may select the first relevant-looking revenue table rather than the finance-maintained source because they lack business context. Genie Agents address this by using a single prompt to assemble a domain-specific agent from governed Unity Catalog context, including structured data, documents, and files. The prompt identifies the desired outcome and relevant sources, while the agent can retrieve content from PDFs, Word documents, presentations, images, and tables under the requesting user’s permissions. The recommended process is to begin with one focused, repetitive job, test it against known questions or incidents, and expand only after measuring performance. Built-in benchmarks provide accuracy scores, while monitoring surfaces user questions and feedback that can reveal gaps in the curated context.
What's New with Monitoring in PostgreSQL 19
PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log_lock_waits changes to on, while log_min_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal_fpi_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg_get_multixact_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.
The prototyping tax is killing your AI roadmap
The post argues that the “prototyping tax”—delay between an AI-powered idea and a working MVP—causes initiatives to lose momentum, often because organizational context is fragmented across teams, tools, and codebases. Traditional encapsulation and siloed domain knowledge can leave agents with syntax and contracts but not business intent, making production codebases harder to navigate than personal projects. The proposed response is agentic development grounded in business semantics and governed data, with alignment occurring through working builds while CI/CD, code review, and production rigor remain unchanged. The post recommends tracking time-to-prototype, first-pass acceptance rate, and PoC-to-production rate to distinguish faster demos from durable delivery. It cites a benchmark of 401 data tasks and Abacus Insights results: 77% accuracy at roughly half the cost for a platform-native agent, plus about 40% less manual effort and onboarding to first value in roughly half the time.
Using AI_Functions in Your Data Warehouse: Top Use Cases
Databricks AI Functions bring model inference into SQL queries, addressing the need to combine warehouse-resident structured data with unstructured reviews, tickets, PDFs, and transcripts without exporting rows to a separate service. The post presents six Lakehouse use cases: document parsing and extraction, sentiment classification, translation, ticket routing, sales-call extraction, and generative drafting with ai_query. Task-specific functions such as ai_parse_document, ai_extract, ai_classify, and ai_translate handle specialized work, while ai_query sends prompts to an accessible Databricks-hosted Foundation Model serving endpoint. Unity Catalog permissions, existing pipelines, and system.billing.usage keep governance and billing within Databricks, while SQL execution manages planning, parallelization, and retries. Production guidance includes tagging jobs, sampling at least 10,000 rows, choosing models deliberately, requesting structured output, and versioning prompts.
The Future of Data Analytics: Why AI is rewriting the Analyst’s Job Description
The post argues that AI is not eliminating data analysts; it is automating the technical work that has crowded out their business impact. Natural-language tools such as Databricks AI/BI, Genie Code, and Genie One can generate dashboards, answer ad hoc questions, and accelerate tasks such as customer segmentation, which the post says one analyst completed in half a day instead of two months. This shifts the analyst’s focus toward problem framing, context, validation, storytelling, governance, and directing AI agents, while keeping intent, accountability, and decision ownership human. For organisations, the proposed response is to hire and develop curiosity, business acumen, and communication, embed analysts near decision-makers, retain human review, and measure decisions influenced rather than dashboards delivered.
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
Modern Risk Demands a Real-Time Foundation: The CRO’s Mandate
The piece argues that modern risk failures stem less from modeling weakness than from slow, fragmented data architecture. It uses Silicon Valley Bank, Archegos, and the UK LDI crisis to show how batch processing, siloed counterparty data, and static stress tests can obscure liquidity, aggregate exposure, and feedback loops until intervention is difficult. Its proposed shift is a unified Databricks Data and AI Platform combining governed data, real-time aggregation, lineage, scenario analysis, and AI-assisted investigation across risk functions. Specific capabilities described include Unity Catalog permissions and lineage, VaR or Expected Shortfall deltas computed in seconds, and SQL-auditable natural-language queries with MLflow-backed model provenance. The conclusion is that real-time, defensible risk visibility supports the CRO’s expanding role in capital allocation and growth, with Raiffeisen Bank International, Morgan Stanley, and State Street cited as production examples.
How Standardizing Product Telemetry Reduced Time to Insight by 97%
Salesforce faced scaling bottlenecks when individual engineering teams maintained fragmented, custom telemetry pipelines requiring manual data transformations and dashboard creation every release. To resolve these silos, the engineering organization established the Product Data Platform (PDP) using a standardized custom schema built atop Monitoring Cloud infrastructure. This framework mandates core telemetry fields for consistent metric generation while offering governed optional attributes for product-specific flexibility. An AI-driven Model Context Protocol tool assists developers by recommending compliant instrumentation patterns directly within their workflows. Today, the unified platform processes 45 billion rows of data daily across 19,000 distinct events and 2,000 product features, reducing dashboard refresh latency by 97 percent from one month to daily updates.
Introducing FILE type: a native column type for multimodal data
The beta of FILE introduces a native column type for storing unstructured data such as documents, images, audio, and video alongside structured table data. FILE applies fine-grained governance, row- and column-level controls, ABAC, and lifecycle synchronization, so deleting a row also deletes the corresponding binary in object storage. Its lightweight pointers fetch file content only when a query requires it, while SQL, Python UDFs, built-in AI functions, and materialized views support multimodal processing. The example uses sampled dashcam frames and object detection to identify stops with no hazard, then joins extracted results with trip metadata; the announcement also describes connectors, incremental pipelines, and an open effort around Parquet, Delta Lake, Apache Iceberg, and Apache Spark.
How Musinsa scaled its audience engine with ClickHouse Cloud and reduced TCO by 71.4%
Musinsa’s Audience Engine powers a customer data platform that targets audiences across 16.4 million members. It stores each user-to-audience relationship as a separate row, producing roughly 1.1 billion cohort-to-user mappings across 2,500 audiences and potentially 16 million new rows when one audience is added. Its self-hosted ClickHouse deployment coupled storage and compute, forced workloads to share a cluster, and created operational burdens as data and query demands grew. Musinsa migrated to ClickHouse Cloud on AWS, using separated S3-backed storage, workload-specific compute, automatic idling, vertical autoscaling, and ClickPipes for direct real-time ingestion. The company reports an 86.5% reduction in storage costs and a total cost of ownership reduction of up to 71.4%, while shifting engineering effort toward CDP business logic.
What is an AI Assistant?
AI assistants are software systems that use language models, data retrieval, and reasoning to understand natural-language requests and return information or perform tasks. For enterprise data teams, they can generate SQL, build dashboards, troubleshoot pipelines, and automate repetitive work, but their usefulness depends on integration with data catalogs and governance rather than generic text generation alone. The described pipeline captures input, parses intent and context, retrieves knowledge, generates an output, and either delivers it for review or executes an action; advanced systems incorporate tool and API use plus feedback from corrections and approvals. The article categorizes assistants as voice, conversational, general-purpose, domain-specific, data and developer, or autonomous agents, while noting risks including hallucinations, privacy exposure, integration complexity, over-reliance, and bias. It presents assistants as augmenting human work through faster, more accessible, and more scalable decision support, not replacing human oversight.
Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning
OfficeQA Pro V2 benchmarks whether AI agents generalize grounded reasoning to unfamiliar enterprise document collections rather than specialize in one corpus or task distribution. It contains 90 questions grounded in roughly 120,000 pages of U.S. Treasury accounting records spanning 1793–2024, including dense tables, charts, revised values, and changing reporting conventions. Questions were generated with asynth from seeded inputs, corpus evidence searches, traceable reasoning, and quality gates, then evaluated with deterministic exact match at 0.0% tolerance. Out-of-the-box frontier agents averaged 37.5% in one matched comparison, while competition systems averaged 41.1% and the winning team reached 63.3%; Databricks Genie raised matched-model accuracy by 24.0 percentage points on average, although failures remained in parsing, temporal reconciliation, and entity or category interpretation.
How Mercado Libre rebuilt its observability platform on ClickHouse Cloud with 50x faster trace queries
Mercado Libre rebuilt its O11y events observability platform to answer granular business-level questions such as why a payment failed across a large microservice environment. The previous storage-layer design performed transformation, filtering, and aggregation at query time, causing queries to run longer than five minutes and making high-cardinality analysis expensive. The team moved storage to ClickHouse Cloud on AWS while retaining its existing ingestion path, and shifted heavy work into the database with materialized views, layered retention, and a dedicated lookup table for high-cardinality filtering. O11y events uses OpenTelemetry instrumentation with business-flow attributes and samples all traffic for critical applications. The resulting system reduced trace-query time to about four seconds, a 50x speedup, achieved up to 89% compression, and scaled from 7 million to 400 million spans per minute in ClickHouse.
Ingest semi-structured data faster and more efficiently with Variant - Now Generally Available
Databricks announces general availability of the Variant data type for ingesting semi-structured JSON, XML, and CSV while avoiding the usual choice between flexible storage and fast, schematized queries. Variant Shredding, also generally available, stores common fields as columns in Parquet files, while Predictive Optimization uses workload and query patterns to identify important fields, collect statistics, and improve file skipping. More than 5,000 teams write Variant, with users executing over 500 million Variant queries monthly across more than 160 TB of data; shredding delivers nearly four-times-faster reads than unshredded Variant and 30-times-faster reads than JSON strings. Auto Loader and Zerobus can ingest Variant into Delta or Iceberg, and future plans include Liquid Clustering by Variant fields, expanded SQL functions, and additional integrations.
Using Triggers On Schemaless, Uber Engineering's Datastore Using MySQL
Schemaless triggers provides a scalable, fault-tolerant, and lossless way to listen for changes in Schemaless, Uber Engineering’s datastore using MySQL, and drives workflows including trip billing, analytics-warehouse ingestion, and cross-datacenter replication. Schemaless stores immutable cells and all their versions, allowing each shard to function as a partitioned change log with monotonically increasing added IDs while retaining random-access APIs. The framework tails shards through get_cells_for_shard, assigns them to worker processes under a leader, persists offsets in shared storage, and invokes column-specific callbacks such as bill_rider; handlers must be idempotent because retries can occur. Failed workers’ shards are reassigned, repeatedly failing cells can move to a separate queue, processing continues during leader or shared-storage outages although redistribution pauses, and replicas insulate triggers from database-node failures.
Modeling Device Capabilities for Analytics
Netflix supports diverse features, such as 4K streaming, immersive audio, and cloud gaming, across varied hardware platforms. Because hardware constraints like RAM, CPU cores, and display capabilities limit feature support on certain models, understanding device capabilities is crucial for managing user experiences. To address this challenge, Netflix built a comprehensive device capability data model integrated with internal feature flags. The architecture employs a cumulative table to capture the latest state of device capabilities alongside a histogram table tracking 28-day active device counts grouped by model and software version. These datasets power analytical products that evaluate feature reach and inform rollout decisions for capabilities like Netflix Spatial Audio and 4K Ultra HD.
Dispatches from O'Reilly: The best risk mitigation strategy in data? A single source of truth
Data organizations face severe operational risks from inconsistent metrics, sprawling access controls, and incomplete change management across disparate BI tools and warehouses. Traditional remediation relies on centralized BI gatekeepers and fragmented governance frameworks, which introduces costly bottlenecks and fails to scale. Implementing a semantic layer mitigates these risks by consolidating metric definitions, business logic, and calculations into a single governed location. Centralizing access shrinks the governance surface area and allows metric updates to propagate automatically to downstream consumers, including Tableau, Power BI, Excel, and AI tools. Structured metadata captures context directly alongside models, enabling reliable self-service and federated hub-and-spoke delivery while keeping definitions auditable under version control.
NBCUniversal’s Seamless Migration: Unlocking Scalable Analytics with Databricks
NBCUniversal sought to modernize its data infrastructure as growing data volumes, evolving analytics demands, and slot-based compute reservations created cost, contention, and flexibility concerns. Working with EXL and Databricks, it migrated workloads to a Databricks lakehouse architecture that unifies data engineering, analytics, machine learning, and business intelligence. The phased, risk-based program assessed workloads and dependencies, designed workspaces and governance with Unity Catalog, and used four custom accelerators for SQL translation, DAG migration, data transfer, and validation. Job-specific compute, autoscaling, Delta Lake layouts, MLflow, and Lakeflow Jobs supported workload-specific execution and operations. The reported outcome was a 30% reduction in data infrastructure costs, elastic scaling for high-traffic events, unified ML operations, and the onboarding of approximately 300 analysts to Databricks SQL.
Chief data officer: role, responsibilities, and career guide
The chief data officer (CDO) is a senior executive who treats enterprise data as a strategic asset, overseeing data strategy, management, quality, governance, and its use in analytics and AI. The role sits between business strategy, information technology, and data science, while distinguishing the CDO’s focus on data from the CIO’s responsibility for technology infrastructure. Responsibilities span the data lifecycle, governance frameworks, access controls, quality metrics, business intelligence, analytics prioritization, and organizational data capability. The role has shifted from compliance-focused stewardship toward enterprise AI strategy, with enablement-oriented governance making reliable, AI-ready data accessible while protecting its integrity. The guide also presents executive authority, data literacy, and leadership judgment as important conditions for delivering measurable business value.
The EU Digital Product Passport: a traceability deadline
The EU’s Ecodesign for Sustainable Products Regulation (ESPR) will require in-scope products sold in the single market to carry a machine-readable Digital Product Passport (DPP), with batteries first from February 2027. The passport links a unique identifier and data carrier to lifecycle information that must remain accurate, while operators retain responsibility for data stored on their own backend, tiered access, supplier inputs, and any Registry registration. Implementation is primarily a data-integration and governance problem, spanning tier-N suppliers, per-unit operational records, lineage, analytics, AI, and controlled sharing rather than QR-code generation. Databricks components including Lakebase, Unity Catalog, Lakeflow Spark Declarative Pipelines, Databricks Apps, and OpenSharing are presented with a battery-focused Solution Accelerator as an operator-side reference architecture. The stated benefits extend beyond market access to faster recalls, sourcing-risk visibility, and lineage-based sustainability reporting, but technology alone does not establish compliance.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.