What we’ve learned building cloud agents

Summary

Cloud agents have evolved from local-agent extensions into systems running on dedicated virtual machines, with isolated environments, dependencies, network access, parallel execution, and unattended long-running work. That shift makes the development environment a product: reliable execution increasingly depends on rebuilding tools, VM hibernation and resumption, image checkpointing and forking, integrations, and controlled access to secrets and networks. After a fragile work-stealing design produced roughly one 9 of reliability, the team migrated the agent loop to Temporal, whose retries and scheduling across failures helped exceed two 9s; Temporal now handles more than 50 million actions daily across over 7 million workflows. Decoupling agent-loop, machine, and conversation state enables independent pod lifecycles, asynchronous subagents, and retry-aware streaming, while shorter workflows simplify upgrades. The next goal is self-healing environments that can report and resolve missing secrets, blocked network access, or other obstacles.

Takeaways

1. 1

   Cloud agent output quality depends heavily on a complete development environment; missing tools or execution and verification capabilities may appear as subtle quality degradation rather than an explicit error.

2. 2

   Migrating the agent loop from work-stealing to Temporal enabled retries, scheduling across machines, and durability across failures, taking reliability from roughly one 9 to beyond two 9s.

3. 3

   Separating agent, machine, and conversation state allows independent pod lifecycles and retry-aware clients that can rewind streamed output after a failed step is retried.