Loading…
Topic
Spark
27 posts about Spark. Every summary links to the original.
- 27 posts
- 5 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Introducing Funke: Native HL7v2 Parsing on Databricks
HL7v2 is the dominant healthcare messaging standard but is difficult to parse due to its nested, delimiter-encoded structure. Existing workarounds like converting to FHIR or using third-party engines add cost and latency. Funke, the successor to the open-source Smolder library, is a new open-source Python and PySpark library under the Databricks License that parses HL7v2 messages directly into native Spark types, preserving the full hierarchy losslessly. It deploys as a Databricks Asset Bundle with a Declarative Pipeline that uses Auto Loader to ingest raw messages into a Unity Catalog volume, then parses them into structured silver tables following the medallion pattern. The library supports all HL7 message types and versions, includes accessor helpers for extracting fields by segment and component, and ships with a demo using synthetic ADT events to illustrate end-to-end ingestion and gold table creation, including a live census and bed utilization dashboard.
Google Cloud Use Cases: How Enterprises Run Data & AI on GCP with Databricks
The guide presents Google Cloud as a unified platform for enterprise data engineering, analytics, machine learning, and generative AI, with managed services intended to reduce infrastructure and integration work. It focuses on practical architectures, including medallion ETL pipelines that store Bronze, Silver, and Gold Delta Lake tables in Google Cloud Storage and expose Gold data to BigQuery through an optimized connector. For streaming workloads, Cloud Pub/Sub supplies ingestion while Apache Spark Structured Streaming processes continuous micro-batches; together with Delta Lake ACID transactions, this supports exactly-once results despite duplicates or out-of-order messages. The text also describes Cloud Run and Cloud Functions for containerized or event-driven workloads, and identifies retrieval-augmented generation applications on Vertex AI as a common starting point for generative AI. It concludes that integrated service combinations, rather than isolated tools, are the central strategic advantage for organizations building evolving data and AI platforms on Google Cloud.
Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks
Stateful Apache Spark Structured Streaming queries historically locked their state-store partition count into the checkpoint, so changing spark.sql.shuffle.partitions after a workload grew had no effect without rebuilding and losing accumulated state. Databricks Runtime 18 and later introduces on-demand state repartitioning, a Public Preview feature that preserves the checkpoint while moving state to a new partition layout. Operators set spark.sql.streaming.stateStore.partitions, stop the query, and restart it; the query finishes any pending microbatch, physically rehashes and redistributes state, then resumes with the requested count. The feature applies to aggregations, stream-stream joins, deduplication, sessionization, and transformWithState, provided the RocksDB state store is used. Repartitioning pauses processing and its duration is reported in controlBatch.REPARTITION; Coveo reports a 40% reduction in related Amazon S3 API costs.
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
Tech – Tools & craft – Notion
Notion’s engineering collection covers tools, product design, infrastructure, security, and data systems across a set of technical posts. It opens with an inside look at the hosted Notion MCP server, created in response to requests for AI agents to work directly with Notion workspaces; the path described runs from an open-source experiment to a hosted solution with custom tools for AI workflows. Other entries describe Lore’s conversion of conversations, decisions, tasks, and durable facts into Notion pages, a Spark-native lexical-search reindexer, and multi-region systems for search, analytics, AI, and event logging. The collection also reports CI guardrails for schema changes, security testing for Custom Agents, vector-search scaling by 10x with costs reduced by 90%, Spot Balancer for Spark on Kubernetes, and the architecture and data model behind Offline Mode.
Introducing FILE type: a native column type for multimodal data
The beta of FILE introduces a native column type for storing unstructured data such as documents, images, audio, and video alongside structured table data. FILE applies fine-grained governance, row- and column-level controls, ABAC, and lifecycle synchronization, so deleting a row also deletes the corresponding binary in object storage. Its lightweight pointers fetch file content only when a query requires it, while SQL, Python UDFs, built-in AI functions, and materialized views support multimodal processing. The example uses sampled dashcam frames and object detection to identify stops with no hazard, then joins extracted results with trip metadata; the announcement also describes connectors, incremental pipelines, and an open effort around Parquet, Delta Lake, Apache Iceberg, and Apache Spark.
How Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale
Dow built the Carbon Footprint Ledger (CFL) on Databricks to provide a transparent, verifiable way to track and communicate product carbon footprints amid customer Scope 3 commitments and tighter environmental disclosure requirements. The system combines carbon-accounting methodology assured against ISO 14067 and the GHG Protocol Product Standard with a calculation engine and certificate-issuing ledger that maintains residual balances. Within Dow’s Integrated Data Hub, Apache Spark unifies supply-chain, sustainability, operations, and other source data, while Unity Catalog governs access and Delta tables provide ACID transactions, time travel, and efficient upserts. An advanced optimization model, developed and deployed with MLflow, identifies lowest-greenhouse-gas production pathways and sends versioned, monitored results through the pipeline with lineage tracking. The implementation reduced PCF processing from weeks to a fraction of that, enabled portfolio-scale calculations, and supports verifiable certificates and commercial use of decarbonization investments.
Building a soccer coaching app on Databricks
Coach’s Corner, also called La Pizarra, turns high-frequency soccer tracking data into a bench-side application for replay, tactical analysis, scouting, standings, and agent-generated dossiers. Built as a Databricks App, it ingests NDJSON feeds at 25 frames per second through Auto Loader and Spark Declarative Pipelines, enforcing 46 data-quality expectations across bronze, silver, and gold layers. Liquid clustering supports 1–3-second DBSQL queries, while Lakebase synchronizes gold data to Postgres for millisecond replay reads and separates sequential playback from exploratory analytics. The scouting layer grounds Genie, Vector Search, a Unity Catalog-registered xG model, and an Agent Bricks supervisor in governed data, with Claude calls routed through the Unity AI Gateway, MLflow tracing, and a deterministic fallback. Together, these components are presented as a way to deliver traceable insights within seconds without forcing coaches to interpret raw tables or analysts to relay every result.
Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption
Grab is transitioning its petabyte-scale data lake from Hive Parquet tables managed through the Hive Metastore to a table-centric architecture centered on Apache Iceberg, while retaining storage-compute separation on S3. The move addresses catalog listing latency, severe small-file fragmentation, manual partition registration, absent native ACID operations, and catalog state becoming stale after direct S3 changes. Rather than converting the entire lake at once, Grab prioritizes high-value tables and uses features such as Z-ordering; representative workloads cut one query from 70 seconds to 6, reduced daily S3 API costs by up to 95%, and halved cluster resource usage. Its open-sourced UnifiedSparkCatalog detects table formats through metastore metadata or path inference and routes operations to the appropriate Spark catalog without format-specific prefixes. Deployment also required resolving Hive lock contention, timestamp incompatibilities, and storage-tier cost spikes.
A Decision Framework for ETL Migration to Databricks
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
From test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Lakeflow: A new era of agentic data engineering
Databricks announces a major evolution of Lakeflow, its unified platform for data engineering across ingestion, transformation, and orchestration, with capabilities centrally governed by Unity Catalog. Genie Code and generally available Lakeflow Designer support agentic and no-code pipeline development, while Genie ZeroOps monitors production assets, analyzes failures, proposes fixes, and validates them in a governed sandbox before human approval. Lakeflow Connect expands to more than 100 managed connectors, and Zerobus Ingest adds Kafka-compatible, gRPC, REST, SDK, and OpenTelemetry interfaces for high-volume event ingestion. Real-Time Mode for Spark Declarative Pipelines reaches Public Preview with end-to-end latency as low as 5 milliseconds, alongside declarative APIs and expanded Lakeflow Jobs integrations. The release also adds data-readiness triggers and external orchestration for systems including Snowflake, REST APIs, Slack, and PagerDuty.
What is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Databricks at SIGMOD 2026
Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.
Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy
Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.
Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs
Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.
How to Build Real-Time Fraud Detection using Spark Real-Time Mode and Lakebase
Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.
Rethinking Distributed Systems for Serverless Performance and Reliability
Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.
Peril predicts: Precision payouts for a volatile world
Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.