---
title: "Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day"
description: "Salesforce Data 360 processes about eight million Apache Spark executions daily, where successful completion alone does not confirm efficient resource use. The team instrumented the shared Data Processing Controller (DPC) gateway with a DataFlint Spark listener plugin to capture per-executor CPU, memory, shuffle, and zero-task metrics without application code changes. Metrics stream through Kafka into an Iceberg lakehouse, and a curated Silver analytic layer ensures analysis uses only complete, comparable runs and correctly handles retries and missing data. Investigation revealed workloads that repeatedly allocated executors performing zero tasks, enabling targeted reductions like lowering initial executors from forty to four. After staged production rollout monitored for runtime, spill, retries, and platform pressure, weekly compute-cost run rate fell 31% while DPC job starts grew over 7%."
---

# Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day

[Salesforce](https://yomu.fyi/company/salesforce) · Scott Nyberg · Oct 8, 2026

**Type:** Problem & solution

## Summary

Salesforce Data 360 processes about eight million Apache Spark executions daily, where successful completion alone does not confirm efficient resource use. The team instrumented the shared Data Processing Controller (DPC) gateway with a DataFlint Spark listener plugin to capture per-executor CPU, memory, shuffle, and zero-task metrics without application code changes. Metrics stream through Kafka into an Iceberg lakehouse, and a curated Silver analytic layer ensures analysis uses only complete, comparable runs and correctly handles retries and missing data. Investigation revealed workloads that repeatedly allocated executors performing zero tasks, enabling targeted reductions like lowering initial executors from forty to four. After staged production rollout monitored for runtime, spill, retries, and platform pressure, weekly compute-cost run rate fell 31% while DPC job starts grew over 7%.

## Context

Salesforce Data 360’s platform runs roughly eight million Apache Spark workload executions daily across ingestion, segmentation, and activation use cases. The team needed to optimize resource allocation without requiring application code changes, using execution evidence rather than successful completion alone.

## Approach / What changed

The team instrumented the Data Processing Controller (DPC) common gateway with a DataFlint listener plugin. Listeners collected per-executor CPU, memory, shuffle, spill, and zero-task-executor metrics, exported via Kafka to an Iceberg lakehouse. A Silver layer preserved semantics: used only complete runs within workload and environment scopes, correctly distinguishing retries and missing data. Queried via Trino and a natural-language agent to find expensive workloads with repeated zero-task executors. Changes included reducing initial executor count (e.g., from 40 to 4) or lowering the executor ceiling by ~20%. Validation monitored p95 runtime, retries, OOMs, spill, and platform pressure (pod churn, etcd) before advancing.

## Takeaways

- Successful Spark job completion does not indicate efficient resource use; compare requested capacity with actual useful work (e.g., zero-task executors) across complete, comparable runs to identify waste.
- Instrument your shared Spark submission gateway (e.g., DPC) with a listener plugin to collect per-executor metrics (CPU, memory, shuffle, spill) without modifying application code; export via Kafka to a data lake for centralized analysis.
- When reducing Spark executor count, validate by monitoring p95 runtime, spill, retries, OOMs, and cluster pressure (pod churn, etcd) in lower environments before production rollout; cost reduction alone is insufficient.

## FAQ

### How can you identify wasted Spark capacity without changing application code?

By instrumenting the shared Spark submission gateway (Data Processing Controller) with a DataFlint listener plugin. The plugin captures per-executor metrics like CPU idle time and zero-task executors, and exports through Kafka to Iceberg lakehouse for analysis, without any changes to user application code.

### What metrics distinguish a valid resource-optimization candidate from a misleading one?

Use only complete Spark application runs within defined workload and environment scopes. Distinguish retried stages (extra billed work but not distinct logical work) and avoid summing memory spill and disk spill as independent bytes. The team's Silver layer enforces these semantics.

### How did the team validate that reducing executor count did not harm reliability?

They rolled out changes to lower environments first, then production in stages, monitoring p95 runtime, retries, OOMs, memory spill, and disk spill, plus platform pressure like pod churn and etcd load. Only when all metrics remained acceptable did they advance the rollout.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Cost Optimization](https://yomu.fyi/topic/cost-optimization), [Data Engineering](https://yomu.fyi/topic/data-engineering), [Performance](https://yomu.fyi/topic/performance)

- Source: [Salesforce](https://engineering.salesforce.com/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day)
- Source URL: https://engineering.salesforce.com/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day
- Ingested by Yomu: 2026-10-08T18:00:14.716Z

[Read original post](https://engineering.salesforce.com/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day)