# Building Reliable Production AI with Durable Workflows

[Salesforce](https://yomu.fyi/company/salesforce) · Scott Nyberg · Jul 27, 2026

**Type:** Problem & solution

## Summary

AI prototypes rely on simple prompt-response interactions, but production AI systems handling thousands of records must manage distributed execution issues like worker crashes, rate limits, and transient failures. Rather than treating long-running operations as single large jobs or rebuilding state from logs, systems require durable workflows that preserve execution history across failures. In Agentforce Grid, execution is organized into a hierarchy where column runs divide into recoverable batches and rows composed of discrete activities. Each batch checkpoints after every successful row, retrying up to ten times with exponential backoff and resuming strictly at incomplete work to prevent duplicated model calls. Exposing progress across these hierarchical boundaries ensures that both users and engineers can distinguish completed, failing, and retryable units.

## Context

Production AI systems that span thousands of records or operations frequently break down when worker restarts, deployments, rate limits, or transient errors interrupt execution. When execution state is tied to ephemeral worker processes or reconstructed ad hoc from logs and database records, systems cannot reliably determine which tasks succeeded, risking costly duplicated model quota or full restarts.

## Approach / What changed

Agentforce Grid implements durable workflows that treat execution itself as persistent state outside worker lifecycles. Execution is modeled as a hierarchy where column runs split into recoverable rows and activities with defined retry policies, non-retryable errors, and exponential backoff. Batches checkpoint after each successful row, resuming execution at the first uncompleted row across up to ten retries while using idempotent writes.

## Takeaways

- The most effective retry boundary is the smallest meaningful unit of recoverable work, avoiding both full job restarts and excessive orchestration overhead.
- Agentforce Grid checkpoints after each successful row and retries up to ten times with exponential backoff, resuming at the first incomplete row to avoid consuming duplicate model quota.
- Progress visibility should mirror recovery boundaries across cells, rows, columns, and worksheets, allowing users and engineers to track active, failed, and retrying operations.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.salesforce.com/building-reliable-production-ai-with-durable-workflows)