Introducing Apache Spark 4.2

Summary

Apache Spark 4.2 extends the engine’s role in modern data and AI workloads with governed metrics, vector and top-K primitives, Arrow-first Python execution, native change data capture, and stronger streaming foundations. Metric views provide shared business definitions, while Spark Connect uses gRPC and Arrow to let remote clients submit logical plans without a full Spark runtime. Spark SQL adds vector similarity functions, NEAREST BY, geospatial types, sketches, and time-series features; Python interoperability includes Arrow UDFs and can move Spark DataFrames into supported Arrow-native tools without copying or serializing underlying data. Spark Declarative Pipelines adds Auto CDC for SCD Type 1 targets, while Data Source V2 standardizes change streams through CHANGES and expands row-level operations and schema evolution. The release also includes operational updates such as Web UI modernization, Kubernetes improvements, JDK 25 support, and dependency upgrades.

Context

The stated motivation is to move more of the modern data and AI stack into Spark, improving the quality and freshness of data supplied to AI agents while making Spark easier for applications and agents to invoke as a remote execution service.

Approach / What changed

Spark 4.2 adds governed metric views, Spark Connect and broader Arrow-based Python integration, native SQL primitives for vector retrieval and geospatial analytics, Auto CDC in Spark Declarative Pipelines, and expanded Data Source V2 support for change data, row-level operations, schema evolution, and transactions.

Takeaways

• Metric views make dimensions and measures first-class Spark SQL objects, allowing dashboards, applications, and AI tools to use governed business metrics with consistent aggregation semantics.
• Spark Connect separates clients from the Spark server through gRPC and Arrow; clients submit logical plans and receive Arrow batches without requiring a full Spark runtime or colocated JVM.
• Auto CDC provides a Python API for maintaining SCD Type 1 target tables, while the CHANGES surface exposes change data from Data Source V2 connectors through SQL, DataFrame APIs, and PySpark.

Related reading