DatabricksPavithra Rao, Jeanne Choo, Kyra Wulffert, Alex BaurAnnouncement
The Big Book of AgentOps presents AgentOps as an operating discipline for building, deploying and improving production AI agents, combining architecture, evaluation, observability, governance, security and cost management. It addresses the operational risks of tool calls, enterprise-data retrieval, API actions, retries and multi-step execution, where quality, permissions and spend can become difficult to control. The six-chapter book covers four agent architectures, four deployment patterns, a seven-phase lifecycle, DevOps practices, operationalization planning and stakeholder alignment, with Unity Catalog, Unity Gateway and MLflow central to its Databricks-based deployment guidance. Its practices include building evaluation datasets from real traces, calibrating automated judges with subject-matter-expert feedback, designing tracing and access controls into systems, and using RACI matrices to clarify ownership. The announcement cites customer examples including FactSet’s 44% accuracy improvement, DXC Technology’s 30% total-cost-of-ownership reduction, and ICE’s 77% syntactic accuracy and 96% execution matches.
The book responds to the challenge of moving AI-agent pilots into dependable enterprise production systems. Agents can retrieve enterprise data, call APIs, use tools, and execute multi-step tasks, creating operational questions around quality, traceability, permissions, failure handling, readiness, and cost.
The book codifies AgentOps practices across six chapters covering agent architectures, deployment patterns, a seven-phase lifecycle, DevOps principles, operational planning, and stakeholder management. It recommends narrow use cases, trace-based evaluation, automated judges calibrated against subject-matter-expert feedback, unified observability and governance, least-privilege controls, and explicit ownership.
The book distinguishes four agent architectures and four deployment patterns, with Unity Catalog, Unity Gateway, and MLflow supporting the Databricks deployment architecture guidance.
Evaluation should begin with subject-matter experts reviewing real traces, then use representative datasets and calibrated automated judges to support safer iteration.
A single request can trigger multiple model calls, retries, sub-agents, and guardrail checks, so usage attribution, spending limits, and accountability are required.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
AgentOps is the operational discipline of building, deploying, and monitoring production AI agents. Learn how tracing, evaluation, and governance turn raw agent behavior into reliable, cost-controlled systems.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
