Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs

Summary

Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.

Context

Pandas UDFs still incur Pandas/Arrow conversion overhead, including additional data copies and deep copies for columns with NULL values. The source also identifies limited support for complex datatypes, including nested StructType outputs in aggregation use cases.

Approach / What changed

Native Arrow UDFs execute directly on PyArrow data, preserving its columnar layout and avoiding conversion to Pandas or NumPy objects. The APIs include scalar, aggregate, and table functions, plus DataFrame-level mapInArrow and applyInArrow operations, with decorator and type-hinted definitions.

Takeaways

• Native Arrow UDFs are introduced in Databricks Runtime 18.0 and operate directly on Arrow data instead of converting inputs into Pandas or NumPy objects.
• Scalar, aggregate, and table-function interfaces support direct and iterator-based input modes; iterator variants can amortize one-time setup such as model loading or regex compilation.
• The reported comparison found Arrow UDFs approximately 10% faster than Pandas UDFs and using approximately 40% less memory, while also providing better support for complex datatypes.

Related reading