Loading…
Topic
247 posts about Data Pipelines. Every summary links to the original.
See the idea
AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
ClickHouse has announced a GCP Pub/Sub connector for ClickPipes, now in Private Preview, to stream GCP infrastructure logs, application events, and operational signals into ClickHouse Cloud in real time. Previously, routing data from Pub/Sub into ClickHouse required GCS or a Dataflow pipeline; the connector instead manages subscription lifecycle, schema inference, and message delivery after a topic and target table are configured. It supports JSON, Avro, and Protobuf, Pub/Sub Schema Registry integration, automatic decompression, attribute-based filtering, and start positions of latest, earliest, or a specific timestamp. ClickPipes supports horizontal and vertical scaling, configurable replicas, automatic retries, built-in metrics and logs, and programmatic management through OpenAPI and the ClickHouse Terraform Provider. The connector is available upon request and free during Private Preview, with access offered through a waitlist or a ClickHouse Cloud account manager.
Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.
Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.
Open Electricity uses ClickHouse to serve live Australian energy data through a public platform and API spanning roughly one billion records. Migrating from Postgres and TimescaleDB addressed queries that could take more than five minutes, reducing a benchmark to just over three seconds and to 60 milliseconds with a materialized view. The new setup keeps heavy relational workloads in Postgres while ClickHouse handles analytics, using ReplacingMergeTree base tables and more than 20 materialized views for different aggregation patterns. It also uses substantially less hardware than the previous configuration: four cores, four GB RAM, and 200 GB disk versus 16 cores, 32 GB RAM, and 1.6 TB, while serving data added daily and API users.
Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability.
Production financial intelligence systems frequently rely on hand-engineered tabular features and rule sets that are brittle, expensive to maintain, and blind to historical sequential patterns. NVIDIA demonstrates an accelerated reference pipeline that replaces standard BPE tokenization with a GPU-based domain tokenizer, converting raw transactions into semantic tokens with an 8,192-token context window. Using the NeMo AutoModel library, a compact 29-million parameter decoder-only transformer is pretrained from scratch on unlabeled transaction sequences using causal language modeling. Learned sequence embeddings are extracted, compressed using PCA, and concatenated with raw tabular features to train a downstream GPU-accelerated XGBoost fraud detection model. On the IBM TabFormer benchmark dataset, this combined approach achieves a 41.76% lift in Average Precision over the baseline model relying solely on raw tabular features.
Sidekick’s customer-segmentation skill was fine-tuned on tens of thousands of successful, de-identified production queries, so it generated zero-result queries instead of refusing impossible requests such as finding customers by occupation. A small Toloka dataset of roughly 600 standard queries and 602 refusal annotations became a seed for four frontier LLM judges, which were calibrated against human-labeled examples and required to agree on both verdict and reasoning. The pipeline used mutually exclusive categories to resolve conflicting labels, filtered disagreements for human review, and fed accepted examples from later production traffic back into fine-tuning. With automated curation, the segmentation pass rate rose from 0.762 to 0.798 versus naive merging, while manual validation measured 86.3% refusal accuracy and a 4.6% false-positive rate. The resulting flywheel is being applied to other Sidekick skill models and data-quality problems.
Enterprise intelligence (EI) is presented as an organization-wide capability combining business intelligence, knowledge management, enterprise search and AI to turn structured and unstructured data into decisions and actions. Unlike traditional BI, which centers on dashboards, reports and structured data, EI connects these capabilities through a shared architecture and governed business context. The described stack includes a lakehouse-based data foundation, batch and streaming pipelines, governance, semantics, analytics, search, machine learning, generative AI and a decision layer. Shared definitions such as “active customer” and “monthly revenue” are intended to keep dashboards, queries and AI agents aligned, while maintained context addresses knowledge that becomes stale as the business changes. The result described is a common trusted source from which people, applications and agents can produce insights and initiate actions.
Databricks is formalizing its Forward Deployed Engineering (FDE) organization to address customers’ shift from migration and data-pipeline requests toward business outcomes with AI. FDE brings Professional Services together around an engineering-led model that embeds engineers with customers, supports modernization and production AI, and works with partners and Databricks R&D. In the cited examples, teams migrated five-plus petabytes of JPMC Consumer and Community Banking Risk data and more than 500 notebooks in four months, while Fox used Lakebase, AI Search, Databricks Apps, and Model Serving to redesign fan experiences. The organization says its engagements use shared OKRs, rapid prototype-to-production delivery, embedded engineering, outcome-aligned commercial options, and global partner coverage, with Fox reporting that Sports AI users spend approximately twice as long in the app.
As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.
ClickHouse and Datadog announce a partnership aimed at reducing the tradeoff between retaining complete telemetry and keeping it accessible through familiar observability workflows. New Preview capabilities let organizations route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer. Pipelines can parse, enrich, filter, transform, and redact data before storage, while ClickHouse Cloud’s separation of storage and compute supports long-term retention, independently scaled query resources, and workload isolation. Federated log search queries data directly in ClickHouse, avoiding duplicate copies or re-ingestion while allowing high-volume logs to remain searchable and historical investigations to use full-fidelity telemetry. The partnership combines Datadog’s collection and investigation experience with ClickHouse’s storage and analytics foundation; the native integration and federated search are available in Preview.
Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.
To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.
Qube Research & Technologies (QRT), a global quantitative investment manager headquartered in London, uses ClickHouse Cloud for a data platform that serves researchers and for near-real-time risk monitoring, management, and P&L. Founded in 2018 as a cloud-native firm after the spinout of Credit Suisse’s proprietary trading arm, QRT chose a cloud-based approach rather than inheriting decades of on-premise infrastructure. The platform supports two major systems, while ClickStack consolidates logs, metrics, and traces for observability; alerts, log queries, and metric aggregations run against a single ClickHouse instance using standard SQL. QRT says its previous database was constrained by writer and reader counts and total cluster load, whereas ClickHouse provides greater speed and scalability, and that a unified monitoring backend simplifies maintenance while remaining reliably fast.
Plenitude and Databricks built an agent-based system that turns solar and wind plant maintenance PDFs into structured data for cross-plant analysis. Event-driven ingestion uses Databricks Jobs and the ai_parse_document AI Function to extract text, tables, figures, and metadata, then stores page- and object-level JSON records in Delta Lake with coordinates, version history, and links to source reports. A Genie space uses Unity Catalog metadata, knowledge-store instructions, and SQL generation to answer natural-language questions, produce visualizations, and export results, while Agent Bricks can orchestrate multi-step workflows and downstream actions. The design also applies automatic liquid clustering to dynamic queries and row-level security to restrict results by country. The resulting data layer supports historical trends, plant comparisons, recurring-fault analysis, and a foundation for predictive maintenance, although the source frames predictive use as a future improvement.
An enterprise data strategy connects organizational data assets to measurable business outcomes, while fragmented architectures can leave data investments uncoordinated and limit real-time analysis and action. The roadmap starts with purpose, scope, executive sponsorship, measurable objectives, KPI mapping, and use-case prioritization based on business impact, feasibility, time to value, and organizational readiness. It then organizes governance, lifecycle management, data quality, target-state architecture, integration, analytics, team structure, compliance, and measurement as interdependent capabilities, emphasizing owners, stewards, decision rights, executable quality rules, and automated cleansing. Implementation proceeds through a time-boxed cross-functional pilot, documented learnings, and incremental scaling, with steering-committee oversight and governance that evolves through feedback. The text gives indicative timelines of 60 to 90 days for a focused pilot, 12 to 18 months for a foundational platform across multiple business units, and multiple years for a mature data-driven culture.
Part 2 describes how to build a repeatable Apache Spark replay harness for testing high-impact microservices against historical, production-shaped inputs without reimplementing the service. The design turns dependency state—such as database snapshots, versioned rules, merchant configuration, eligibility state, and logged service responses—into datasets, while representing side effects as output rows and matching fidelity to the decision under test. A thin Spark wrapper reconstructs ReplayInput objects, invokes the same deterministic decision engine used online, compares current and candidate results, and writes traceable diffs with intermediate values and rule explanations. The source cautions that incomplete context, distributed state, asynchronous workflows, and external actors limit replay’s predictive power, so it should model explicit assumptions rather than claim full production simulation. It presents migrations as a practical point to establish these boundaries and use replay to reduce risk.
The playbook presents AI and data transformation as an enterprise capability built on trustworthy, well-governed data, rather than a model-only project. It explains how governance, scalable pipelines, and continuous quality monitoring address incompatible formats, missing values, duplicate records, and schema inconsistencies from systems such as CRM platforms, operational databases, IoT sensors, and cloud applications. Operational guidance covers KPI design, named pipeline ownership, review checkpoints, data mapping and lineage, automated de-duplication, deterministic enrichment, regulatory classification, and role-based access. It states that deterministic tests and human review remain necessary for AI-assisted code, while telemetry and quarterly audits help detect reliability, governance, and data quality degradation. The concluding position is that operational discipline—not model sophistication—supports reliable machine learning, predictive analytics, and generative AI outcomes.
Data + AI Summit 2026 presents a Telecommunications Industry Experience for operators responding to surging network traffic, regulatory pressure, cybersecurity threats, competition, and customer churn. The event, scheduled for June 15–18 in San Francisco, positions unified data and AI, governed workflows, and production use cases as the basis for operationalized, AI-native telecom models. Its June 17 Telecommunications Industry Forum features keynotes, presentations, and executive panels on customer experience, autonomous network operations, fraud prevention, secure agent deployment, and the return from modernizing legacy data warehouses. Breakout sessions cover automated metadata generation for Genie, conversational AI/BI, Lakeflow pipelines with Agent Bricks, and data exfiltration protection with egress monitoring. The industry lounge will demonstrate Model as a Service and agentic real-time decisioning, while the agenda emphasizes peer examples and architectural blueprints for scaling AI under telecom governance and compliance.
Healthcare X12 pipelines can normalize remittance and enrollment data without giving medical billers a place to act on denials, leaving short-pays in SQL queries and spreadsheets while filing deadlines approach. Genpact and Databricks address that workflow gap with a two-layer operational workbench inside Databricks: production extensions to the open-source x12-edi-parser and a secure web application over Databricks SQL. The engine adds contextual 834 enrollment tracking and decoded 835 CAS adjustments, while the interface provides claim and service-line drill-downs, denial queues, dashboards, quality gates, and appeal or correction drafting. Live Unity Catalog gold-view queries avoid ETL shadow copies and synchronized caches; PHI remains inside the secure perimeter, with row-level security and audit logging for reveals and workflow changes. The proposed next step is Claude-based appeal drafting through Databricks Foundation Model APIs, and a two-week configuration-based trial uses an organization's schema and twenty existing denials.
The best engineering reads, once a week.
