Loading…
Spark
19 posts about Spark. Every summary links to the original.
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
Josh Seidel, Shanelle Roman, Sudhanva HuruliIntroducing FILE type: a native column type for multimodal data
The beta of FILE introduces a native column type for storing unstructured data such as documents, images, audio, and video alongside structured table data. FILE applies fine-grained governance, row- and column-level controls, ABAC, and lifecycle synchronization, so deleting a row also deletes the corresponding binary in object storage. Its lightweight pointers fetch file content only when a query requires it, while SQL, Python UDFs, built-in AI functions, and materialized views support multimodal processing. The example uses sampled dashcam frames and object detection to identify stops with no hazard, then joins extracted results with trip metadata; the announcement also describes connectors, incremental pipelines, and an open effort around Parquet, Delta Lake, Apache Iceberg, and Apache Spark.
Michael Armbrust, Burak Yavuz, Dejan Krakovic, John SpencerHow Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale
Dow built the Carbon Footprint Ledger (CFL) on Databricks to provide a transparent, verifiable way to track and communicate product carbon footprints amid customer Scope 3 commitments and tighter environmental disclosure requirements. The system combines carbon-accounting methodology assured against ISO 14067 and the GHG Protocol Product Standard with a calculation engine and certificate-issuing ledger that maintains residual balances. Within Dow’s Integrated Data Hub, Apache Spark unifies supply-chain, sustainability, operations, and other source data, while Unity Catalog governs access and Delta tables provide ACID transactions, time travel, and efficient upserts. An advanced optimization model, developed and deployed with MLflow, identifies lowest-greenhouse-gas production pathways and sends versioned, monitored results through the pipeline with lineage tracking. The implementation reduced PCF processing from weeks to a fraction of that, enabled portfolio-scale calculations, and supports verifiable certificates and commercial use of decarbonization investments.
Jesse Grekowicz, Tim Licquia, Varun MahajanBuilding a soccer coaching app on Databricks
Coach’s Corner, also called La Pizarra, turns high-frequency soccer tracking data into a bench-side application for replay, tactical analysis, scouting, standings, and agent-generated dossiers. Built as a Databricks App, it ingests NDJSON feeds at 25 frames per second through Auto Loader and Spark Declarative Pipelines, enforcing 46 data-quality expectations across bronze, silver, and gold layers. Liquid clustering supports 1–3-second DBSQL queries, while Lakebase synchronizes gold data to Postgres for millisecond replay reads and separates sequential playback from exploratory analytics. The scouting layer grounds Genie, Vector Search, a Unity Catalog-registered xG model, and an Agent Bricks supervisor in governed data, with Claude calls routed through the Unity AI Gateway, MLflow tracing, and a deterministic fallback. Together, these components are presented as a way to deliver traceable insights within seconds without forcing coaches to interpret raw tables or analysts to relay every result.
Samwel Emmanuel, Sheridan Harris, Andrew Helmreich, Kush Patel, Nick RagoneseA Decision Framework for ETL Migration to Databricks
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
Rafael AieloFrom test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Dr. Thomas Bonfert, Jonathan Bräuer, Fabian Ade, Maxim Hammer, Florian Gorzitzke, David Crescence, Christa Simon, Jörg Zimmermann, Hannes SchneiderLakeflow: A new era of agentic data engineering
Databricks announces a major evolution of Lakeflow, its unified platform for data engineering across ingestion, transformation, and orchestration, with capabilities centrally governed by Unity Catalog. Genie Code and generally available Lakeflow Designer support agentic and no-code pipeline development, while Genie ZeroOps monitors production assets, analyzes failures, proposes fixes, and validates them in a governed sandbox before human approval. Lakeflow Connect expands to more than 100 managed connectors, and Zerobus Ingest adds Kafka-compatible, gRPC, REST, SDK, and OpenTelemetry interfaces for high-volume event ingestion. Real-Time Mode for Spark Declarative Pipelines reaches Public Preview with end-to-end latency as low as 5 milliseconds, alongside declarative APIs and expanded Lakeflow Jobs integrations. The release also adds data-readiness triggers and external orchestration for systems including Snowflake, REST APIs, Slack, and PagerDuty.
Bilal Aslam, Ray Zhu, Manish Dalwadi, Saad Ansari, Giselle GoicocheaWhat is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Lisa CaoDatabricks at SIGMOD 2026
Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.
Indrajit RoyScaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy
Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.
Saad Ali, David Poulet, Daniel Taylor, Ismail MakhloufDatabricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Priyanka Mehta, Shaunak SenIntroducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs
Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.
Ruifeng Zheng, Yicong HuangHow to Build Real-Time Fraud Detection using Spark Real-Time Mode and Lakebase
Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.
Sixuan He, Navneeth NairRethinking Distributed Systems for Serverless Performance and Reliability
Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.
Aaron Davidson, Roland Fäustlin, Zach WilliamsPeril predicts: Precision payouts for a volatile world
Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.
Anindita Mahapatra, Timo Roest, Justin MonaldoApproximate answers, exact decisions: New sketch functions for analytics
Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.
Daniel Tenedorio, Kent Marten, Gengliang Wang, Chenhao LiStop hand-coding change data capture pipelines
Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through __START_AT and __END_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.
Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray ZhuGrab ·
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
Huong VuongShopify ·
Categorizing Products at Scale
Shopify describes a product-categorization system for more than one billion products across the Google Product Taxonomy, which contains over 5,000 hierarchical categories. The model uses product titles, descriptions, collections, tags, vendors, and merchant-provided types, with text preprocessing and a fixed-length term-frequency HashingTF featurizer in PySpark rather than image features or vocabulary-based methods such as Word2Vec. Kesler’s Construction transforms multiclass training into a binary dataset by embedding each candidate class into feature tokens, while logistic regression provides a single, interpretable classifier that can use taxonomy relationships and reduce computational load. The resulting baseline supports more than 20 teams and downstream experiences, while schematized Kafka events and an internal annotation platform provide human feedback; the post identifies class imbalance, multilingual text, and image features as improvement areas.
2023-10-18
