Loading post…
TwilioMaria BermudezExplainer
How we AI-updated 1,138 Error-code Docs Pages
Summary
Twilio used AI to update 1,138 error-code documentation pages over about 12 weeks, with a human reviewing every change. The team first scored roughly 2,692 pages with a deterministic script covering structure, content, and resources, then routed low-scoring pages to an LLM pipeline while sending borderline pages to human review. That pipeline used OpenAI's gpt-5.4 via the Responses API, generated full-file responses, sanitized them, converted them into git apply patches, linted results, and rejected any page whose code snippet fingerprint changed. Small-batch pull requests, tracking files, local-modification checks, and named reviewers limited duplicate work and kept approval in the workflow. Across about 27 pull requests from March to May 2026, the process marked 1,138 error codes complete, while the team acknowledged that human review was the quality bar and that future scaling would benefit from an automated evaluation harness and diff-based generation.
Context
Error-code pages are needed when users are already encountering failures, but Twilio’s catalog was growing as new products and codes shipped. The post says the catalog contained roughly 2,692 codes during the audit and more than 3,200 by publication, while earlier documentation work was associated with error-rate reductions from 23% to 8.8%.
Approach / What changed
A deterministic scorer evaluated structure, content, and resources before any model call. Pages scoring below 70 entered a pipeline that selected candidates, generated updates with gpt-5.4 or smaller variants, sanitized output, created a single-hunk git apply patch, linted the result, and opened a pull request. Humans gated every phase, while tracking files, duplicate checks, local-change checks, and code-snippet fingerprints prevented unsafe or repeated updates.
Takeaways
A plain script scored pages out of 100: pages below 70 entered the pipeline, pages scoring 70–84 received human review, and pages scoring 85 or higher were left alone.
The pipeline compared fingerprints of code snippets before and after generation; any change rejected the page, preventing the update from being committed or opened as a pull request.
The project completed 1,138 error-code updates across about 27 pull requests in 12 weeks, but had no automated evaluation harness and relied on human review as its quality bar.
FAQ
How did Twilio decide which pages should be sent to the model?
A deterministic script scored every page on structure, content, and resources. Pages below 70 entered the pipeline, those scoring 70 to 84 received a human look, and pages scoring 85 or higher were left alone.
How were existing code samples protected from model changes?
The pipeline took a fingerprint of every code snippet before updating a page and compared it afterward. If any snippet changed, the page was rejected, left uncommitted, and returned to the queue.
What limitations did the team identify?
There was no automated evaluation harness, so human review remained the quality bar rather than a reproducible score. The model also returned whole files instead of only changed lines, and merge-status reconciliation stayed partly manual.
Source and freshness
- Source
- twilio.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.