Loading…
Building Reliable Production AI with Durable Workflows
SalesforceScott Nyberg
Summary
AI prototypes rely on simple prompt-response interactions, but production AI systems handling thousands of records must manage distributed execution issues like worker crashes, rate limits, and transient failures. Rather than treating long-running operations as single large jobs or rebuilding state from logs, systems require durable workflows that preserve execution history across failures. In Agentforce Grid, execution is organized into a hierarchy where column runs divide into recoverable batches and rows composed of discrete activities. Each batch checkpoints after every successful row, retrying up to ten times with exponential backoff and resuming strictly at incomplete work to prevent duplicated model calls. Exposing progress across these hierarchical boundaries ensures that both users and engineers can distinguish completed, failing, and retryable units.
Context
Production AI systems that span thousands of records or operations frequently break down when worker restarts, deployments, rate limits, or transient errors interrupt execution. When execution state is tied to ephemeral worker processes or reconstructed ad hoc from logs and database records, systems cannot reliably determine which tasks succeeded, risking costly duplicated model quota or full restarts.
Approach / What changed
Agentforce Grid implements durable workflows that treat execution itself as persistent state outside worker lifecycles. Execution is modeled as a hierarchy where column runs split into recoverable rows and activities with defined retry policies, non-retryable errors, and exponential backoff. Batches checkpoint after each successful row, resuming execution at the first uncompleted row across up to ten retries while using idempotent writes.
Takeaways
- The most effective retry boundary is the smallest meaningful unit of recoverable work, avoiding both full job restarts and excessive orchestration overhead.
- Agentforce Grid checkpoints after each successful row and retries up to ten times with exponential backoff, resuming at the first incomplete row to avoid consuming duplicate model quota.
- Progress visibility should mirror recovery boundaries across cells, rows, columns, and worksheets, allowing users and engineers to track active, failed, and retrying operations.
Related reading
How Salesforce Built an Agentic Engineering Enablement Strategy for Thousands of Software Engineers
Salesforce addressed the challenge of scaling agentic engineering across thousands of software developers by focusing on organizational learning rather than tooling. While individual engineers initially developed disparate, uncoordinated workflows, the Technology, People, Innovation, and Learning (TPIL) team unified these approaches into the Proficiency Level (PL) Framework. The framework maps developer growth across four mindset-driven stages: AI-Assisted, AI-Validating, AI-Orchestrating, and AI-Native. To facilitate progress across these stages, TPIL introduced global AI camps, weekly working sessions, and manager coaching guides. Rather than tracking course completions or telemetry scores, Salesforce measured transformation through behavioral indicators, such as shifts toward validating agent output, designing multi-agent workflows, and asking higher-level questions regarding human judgment.
Scott NybergRemoving the Security Barrier to Agentforce Adoption
Enterprise AI adoption across regulated industries stalls when internal governance mandates private, dedicated network paths for sensitive data. To eliminate the need for manual VPN tunnels and complex proxy layers, Salesforce re-engineered Private Connect into a multi-cloud networking platform connecting customer environments to Hyperforce services. The Private Connect v2.0 architecture utilizes private links, transit gateways, and direct endpoint routing alongside an automated control plane for rapid Hyperforce regional rollouts. In addition to supporting cross-substrate interconnects for Azure and decoupling endpoint management, the platform incorporates a generic connector framework supporting tools like Snowflake, Databricks, and Kafka. Today, the system provisions secure connectivity in under 30 minutes while processing roughly 120 TB of data and 683 million requests monthly across 15 AWS regions.
Scott NybergWhy AI-Generated Code Is Easy but Engineering Trust Is Hard
At Salesforce, engineering teams discovered that AI coding agents could produce internally consistent code and passing test suites that still failed to meet actual requirements. To ensure correctness before writing code, the team developed an agentic Spec-Driven Development workflow centered on explicit specifications and gated verification. The system separates evidence-based repository lookups handled by agents from subjective judgment calls escalated to human engineers. Implementation plans must cite repository evidence reviewed by a Skeptic Agent, while a Compliance Matrix traces each success criterion directly to executable test proof. Finally, a multi-agent review system evaluates the implementation under an asymmetric rule allowing an independent judge to downgrade passing checks but never override failed gates.
Scott NybergHow AI Rebuilt Salesforce’s Decades-Old Localization Pipeline
Salesforce needed to deliver major releases across 34 languages as product localization volume surged by more than 35% under fixed budgets and unchanged delivery windows. To address these scaling constraints, the engineering team replaced their legacy translation workflows with an LLM-powered orchestration pipeline integrating prompt engineering, context engineering, and multi-stage validation. The architecture transforms existing style guides, glossaries, and terminology databases into structured context, routing interface strings through specialized prompt stages—up to 85 stages for some languages—to enforce product context, grammar, and branding rules. Combined with an automated validation loop spanning AI editing, AI validation, and human review, this system reduced localization costs by 50 to 90% while accelerating turnaround times across enterprise releases.
Scott Nyberg