How we built OCA, our AI on-call assistant

Summary

Ramp built OCA, an AI on-call assistant that investigates incidents, shares findings in Slack, answers follow-up questions, and can ask Inspect to prepare code-fix pull requests for human review. In the past four weeks, it assisted with 575 of 1,220 incident-related merged PRs; among incidents with merged fixes, OCA-assisted fixes averaged 37% fewer engineers and about 50% less estimated engineer time, based on Slack activity. Investigations run in a monorepo through Claude Code or Codex, with Temporal managing workflow, while prompts, custom tools, skills, and six one-shot hooks provide successive evidence and reasoning checks. Session-driven testing exposed errors such as misleading one-hour log windows, leading to just-in-time instructions and interim updates that balance accuracy with response speed. OCA remains human-in-the-loop: it has broad read-only access, narrowly scoped writes, and cannot merge changes, although the team is building toward end-to-end routine incident handling.

Takeaways

1. 1

   OCA assisted with 575 of 1,220 PRs merged to address incidents over four weeks; responders rated it helpful 90% of the time.

2. 2

   Six one-shot hooks add staged cross-checks, reasoning-trap checks, sanity checks, Slack-rule reviews, and remediation-plan checks before OCA posts findings.

3. 3

   OCA’s memory system had no discernible effect on responder ratings but appeared to speed up investigations, according to the team’s evaluation.