---
title: "Declarative Pipelines"
description: "1 posts about Declarative Pipelines, summarised, each linking to the original."
---

# Declarative Pipelines
> 1 posts about Declarative Pipelines, summarised, each linking to the original.

## Articles

### [Built for Agents: The Case for Spark Declarative Pipelines](https://yomu.fyi/post/built-for-agents-the-case-for-spark-declarative-pipelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Lisa Cao, Sudhanva Huruli
- Published: Oct 23, 2026
- Type: Benchmark

AI agents now create over 70% of Spark data pipelines on Databricks, making cheap iteration and readable code critical. A benchmark comparing procedural PySpark with declarative pipelines tested 22 tasks across 12 datasets each, totaling 528 builds using Opus 4.8 for agent coding and an automated grader. Declarative pipelines enable dry-run validation that catches structural errors before execution, reducing cluster compute time by roughly 34x compared to procedural pipelines, which used 596 executor-seconds versus 17. Declarative code was about half the size (median 64 vs 131 lines) with less variability, while silent error rates were similar within about 15%. The declarative approach trades 2.3x higher LLM token costs ($0.50 vs $0.26 median) for eliminating wasted cluster time, a trade-off that improves as data volumes grow because compute waste scales while token costs are fixed.
