DatabricksRuifeng Zheng, Yicong HuangExplainer
Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.
Pandas UDFs still incur Pandas/Arrow conversion overhead, including additional data copies and deep copies for columns with NULL values. The source also identifies limited support for complex datatypes, including nested StructType outputs in aggregation use cases.
Native Arrow UDFs execute directly on PyArrow data, preserving its columnar layout and avoiding conversion to Pandas or NumPy objects. The APIs include scalar, aggregate, and table functions, plus DataFrame-level mapInArrow and applyInArrow operations, with decorator and type-hinted definitions.
Native Arrow UDFs are introduced in Databricks Runtime 18.0 and operate directly on Arrow data instead of converting inputs into Pandas or NumPy objects.
Scalar, aggregate, and table-function interfaces support direct and iterator-based input modes; iterator variants can amortize one-time setup such as model loading or regex compilation.
The reported comparison found Arrow UDFs approximately 10% faster than Pandas UDFs and using approximately 40% less memory, while also providing better support for complex datatypes.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Apache Arrow is a columnar in-memory format that lets databases, analytics engines, and Python libraries share data without serialization overhead or format conversion, cutting memory peaks and speeding up common workflows.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
