How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading…
Topic
Data Engineering
23 posts about Data Engineering. Every summary links to the original.
- 23 posts
- 9 companies
- 10 explainers
See the idea
Data Engineering explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Build a Governed Azure Data-to-Agent Workflow
A practical Azure workflow for turning fragmented operational data into governed analytics and agent experiences, with checkpoints for layering, context, access, and deployment.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.
How iSAM Funds built an options research platform for a 10,000x data problem with ClickHouse Cloud
iSAM Funds, a UK-based systematic asset manager, built an options research platform on ClickHouse Cloud to handle the 10,000x data fan-out from underlying instruments to related options. Migrating from Postgres, the team achieved 35-50x compression and increased single-core ingestion from 8,000 to 800,000 rows per second, while ClickHouse’s native Apache Arrow support eliminated encoding reshuffling. ClickHouse’s columnar storage and vectorized execution made queries fast enough to run on demand, allowing the team to drop pre-computed rollups and materialized views and remove a layer of pipeline complexity. The managed service integrates with iSAM’s AWS footprint and enterprise authentication, pushes backups to iSAM’s own buckets for developer use, and lets them scale instances up for large backfills then down to control costs. The platform gives researchers fast, interactive access to full options history, shortening the time from hypothesis to tested strategy.
Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day
Salesforce Data 360 processes about eight million Apache Spark executions daily, where successful completion alone does not confirm efficient resource use. The team instrumented the shared Data Processing Controller (DPC) gateway with a DataFlint Spark listener plugin to capture per-executor CPU, memory, shuffle, and zero-task metrics without application code changes. Metrics stream through Kafka into an Iceberg lakehouse, and a curated Silver analytic layer ensures analysis uses only complete, comparable runs and correctly handles retries and missing data. Investigation revealed workloads that repeatedly allocated executors performing zero tasks, enabling targeted reductions like lowering initial executors from forty to four. After staged production rollout monitored for runtime, spill, retries, and platform pressure, weekly compute-cost run rate fell 31% while DPC job starts grew over 7%.
Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging
IFCO’s data platform team, with Databricks Forward Deployed Engineering, optimized a large dbt project on Databricks to handle billions of daily tracking events for reusable packaging assets. The pipeline faces scale, unpredictable late-arriving data, and the need to reconcile history when late events change previously reported asset timelines. The team tuned incremental model settings to match Delta Lake write behaviors, including liquid clustering, deliberate merge vs. delete+insert strategies, and bounding writes to recent ingestion windows. Diagnosing performance from actual query plans rather than compiled SQL revealed that the consolidation model was scanning far more rows than necessary. These changes cut the core semantic layer job’s daily runtime by over 60 percent, retired a costly nightly full refresh, and reduced daily compute cost by 58 percent.
The 24 most commonly misunderstood marketing data terms
Marketing and data engineering teams often interpret the same data terms differently, leading to campaign rework and misaligned expectations. The guide covers 24 commonly misunderstood terms organized around five practical questions about campaign execution, including where data comes from, who a customer record represents, and what makes an audience ready. For example, ‘customer’ might mean a person to marketing but an entity at a specific grain like account or household to engineering, while ‘real time’ can mean soon enough not to feel broken to marketing but sub-second streaming with always-on architecture to engineering. The post provides paired definitions for each term as conversation starters to surface assumptions and agree on meanings before work begins. It also recommends documenting business definitions in Unity Catalog, connecting agreed meanings to datasets or workflows, and using them for AI agents to produce consistent results.
Automated merchant review summary system with integrated feedback
The system uses representative sampling, large language model (LLM) summarization, automated quality checks, and consumer feedback to generate concise summaries from written merchant reviews. Consumers can scan an overall summary or focus on specific topics such as taste, packaging, and portion size, while merchants can identify recurring themes without reading every comment. The pipeline calculates the merchant’s rating distribution, uses a sentiment model to score reviews, and applies stratified sampling to select a representative subset before sending them to an LLM for summary generation. A separate LLM validates that each claim in the generated summary is supported by the source reviews, and summaries that fail this automated assessment are withheld from publication to reduce unsupported claims. The system regenerates summaries when new reviews or negative consumer feedback reach configured thresholds, and the feedback-driven regeneration includes anti-abuse controls to maintain trustworthiness.
How Fountain rebuilt its data plane on ClickHouse Cloud to power Cue, the Frontline Superintelligence
Fountain rebuilt its data plane on ClickHouse Cloud to power Cue, its agentic AI platform for frontline hiring, moving off a multi-vendor BigQuery and S3/Iceberg batch stack. The migration cut analytics latency from three hours to under two minutes, with most queries returning in under a second, at roughly 66% lower cost. Data from 13 Postgres production databases and four MongoDB databases streams into ClickHouse Cloud via ClickPipes managed CDC, landing as Iceberg and Parquet in S3 before reaching BigQuery and ClickHouse. The new architecture uses two symmetrical pipelines—one for Postgres, one for MongoDB—each running a short path: source, ClickPipes, materialized views, and Cue. The system now carries 8.48 billion CDC rows replicated continuously by ClickPipes and 10.3 billion analytics rows across 15 deployments, served by over 2,000 incremental materialized views.
The 2026 CDP Shift: Why AI Agents Are Only as Smart as Your Data Infrastructure
The 2026 Customer Data Platform Report identifies a critical enterprise challenge: while AI tools grow individually smarter, fragmented customer data causes a worsening, not better, experience. The shift is from computational capacity to contextual continuity, as disconnected pipelines force AI agents to start every interaction from scratch. Three failures arise from fragmented systems: agent amnesia, escalating operational costs from wasted tokens, and trapped intelligence that never reaches active workflows. Modern CDPs have evolved from passive audience builders into API-first context engines that support three pillars: pre-execution context with working memory, unified behavioral and conversational data streams, and infrastructure-level governance enforcing compliance before data reaches AI models. Organizations like Twilio, Trustpilot, and Quartz demonstrate that embedding real-time customer context into the conversation layer outperforms deploying the smartest standalone models.
Tevogen Bio’s Journey to Streamlining Life
Tevogen Bio faced drug development costing upwards of $3 billion and requiring 10–12 years; selecting targets for its single-HLA-restricted SARS-COV2 product took 18–24 months of wet-lab work. Tevogen.AI partnered with Microsoft and Databricks to build and curate a multi-terabyte protein-sequence library and parallelize processing for machine-learning training. Databricks Data + AI Platform, Medallion Architecture, Unity Catalog, and distributed computing cut processing from 50 days to 24 hours under governed, fine-grained access controls. The dataset reached 24 million proteins, 16 billion datapoints, ~700 million unique peptides, and ~37 million cross-matched expert articles. An MLOps framework automated training, inference, monitoring, and retention; the alpha PredicTcell model used XGBoost and ESM models and reported 93–97% recall and 38–43% accuracy, while beta PredicTcell and alpha AdapTcell work began.
A multi agent approach audience intelligence
Advertisers often lose strategic intent when campaign briefs become data queries, while planners and data teams also miss insights hidden across large customer datasets. An AI-powered audience generation solution on the Databricks Data + AI Platform addresses this gap by combining Genie, custom tool-calling agents, Agent Bricks, Unity Catalog, and a Databricks App. Planners describe audiences in natural language; Genie produces governed SQL segments, an Affinity Agent calculates statistically validated lift against the full population, and a Supervisor Agent routes or chains these capabilities. The resulting workflow saves audiences in Unity Catalog for auditability and downstream activation, and is described as compressing planning and iteration cycles from days or weeks to minutes while enabling faster response to market shifts and pacing issues.
What is Data Transformation?
Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.
Welcoming Matt Gilbert, our new Data Analyst
Resend announces Matt Gilbert as a Data Analyst on its Trust & Safety team. He spent 10 years as a Deliverability Engineer at Mailchimp, handling Delivery-team data and maintaining sending infrastructure that pushed around a billion emails a day, after earlier work on the Mandrill transactional mail service. Gilbert describes email delivery as a multidisciplinary field spanning MTA operations, data engineering, analytics, mail-queue performance, and troubleshooting delivery and inbox-placement problems, which led him to specialize in data engineering and analytics. He says he joined Resend because email remains an open, globally available way for services to communicate, and plans to help build data-driven capabilities and intelligence that enable customers to send better and reach inboxes quickly.
Simplifying Data and Product Integrations with a Data Abstraction Layer
Uber built a domain-agnostic Data Abstraction Layer (DAL) to decouple data consumers from changing physical datasets and simplify advertiser reporting. The DAL is an RPC service that accepts a logical table, schema, time range, and filters; logical tables define the interface, while configuration describes locations, relationships, availability windows, and rollup transforms. It generates database-specific queries, runs them in parallel across systems such as an internal OLAP database, Docstore, and Apache Hive, and assembles responses with aggregation and rollup behavior determined by the request. Removing a dimension can produce an account-level response with metrics automatically rolled up, while mixed realtime and daily tables can cover different portions of a requested time range. For advertiser reporting, the approach reduced new report delivery from several weeks or months to under two days and enabled dynamic filters, report builders, and third-party API integrations.
Databricks at SIGMOD 2026
Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.
Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy
Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.
Announcing the Databricks analytics engineer learning pathway
Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions.
Announcing Databricks student fellows
Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications.
Data Science vs Data Engineering: Choosing Analysis or Infrastructure
This guide distinguishes data engineering from data science by their roles in the data value chain. Data engineers build and maintain ingestion pipelines, warehouses, lakes, access controls, and production systems, using tools such as SQL, Python, Scala, Java, Apache Spark, Kafka, orchestration platforms, and cloud infrastructure. Data scientists work with clean, accessible data to perform exploratory analysis, engineer features, train and validate statistical or machine-learning models, and communicate findings through visualization and storytelling. The roles collaborate through feedback on data quality, shared documentation, model deployment infrastructure, and versioned schemas, feature pipelines, and experiments. For choosing a path, the guide recommends trying a two-week pipeline project and a two-week ML project; it also outlines portfolio deliverables for each and says both roles require strong analytical and computer-science fundamentals.
From firefighting to building: How AI agents restored our team’s core productivity
Grab’s Analytics Data Warehouse team supports more than 1,000 monthly users and 15,000 tables, but repetitive questions about definitions, lineage, quality, and basic enhancements consumed about 40% of engineering time. It deployed a multi-agent AI system that routes Slack requests into enhancement or investigation pathways, using FastAPI and LangGraph for stateful, cyclical collaboration, Redis for caching and sessions, and PostgreSQL for persistent memory. Specialized agents query Trino, Hive, or Delta Lake, inspect GitLab code, check incidents and pipeline health, or prepare semi-automated code changes, while human reviewers approve production modifications. The system autonomously resolves most standard inquiries and some common enhancements, reducing resolution time by an order of magnitude and bringing simple answers to users within minutes. The resulting capacity shift reclaimed several FTEs worth of engineering bandwidth and moved hundreds of hours from reactive support to roadmap work.
User foundation models for Grab
Grab describes a foundation model that learns from both tabular data and sequential clickstream interactions to represent users, merchants, and drivers across its superapp. Traditional recommender systems depended on manually engineered, task-specific features, which required substantial effort and did not capture evolving behavior holistically. The custom architecture addresses heterogeneous modalities including text, numerical values, IDs, images, and location data, combining long-term profiles with immediate intent. Grab pre-trains the model on a diverse corpus, then either extracts embeddings for downstream recommendation and prediction tasks or fine-tunes it to predict directly. The resulting reusable representations are reported to support applications such as ad optimisation, dual app prediction, fraud detection, and churn probability, while future work targets unified data, improved architecture, and greater training and inference efficiency.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.