The AI Tool Trade-off: Rented Speed vs. Owned Control
The AI tool trade-off: where AI compresses bounded work, what verification and governance really cost, and the condition that decides whether to rent capability or own the stack.
MetaBy Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu, Eric LiuAnnouncement
Meta has deployed Catching JiTTests, a just-in-time test generation system that uses LLMs to catch bugs before they land in production. The system runs overnight on spare capacity and targets risky diffs, using two workflows: a dodgy diff workflow treating each diff as a mutant, and an intent-aware workflow inferring the change’s intention and risks. False positives are filtered through an LLM-as-Judge ensemble and a rule-based assessor called RubFake, with a human-on-the-loop approach where engineers set criteria rather than reviewing each output. In the first half of 2026, the system caught 74 serious privacy bugs, lowered privacy SEVs by 20 percent, and engaged 589 engineers with a 60 percent true-positive rate. This approach catches classes of bugs that traditional LLM-based code reviews might miss, demonstrating that AI-generated testing can scale to reduce organizational risk.
Meta’s codebase of hundreds of millions of lines evolves constantly, making traditional testing insufficient to catch bugs before they reach production.
Catching JiTTests uses two tandem workflows: a dodgy diff workflow that treats each diff as a mutant to generate weak catches, and an intent-aware workflow that infers the change’s intention and risks to construct targeted tests. False positives are filtered via an LLM-as-Judge ensemble of multiple state-of-the-art models and a rule-based assessor called RubFake. The system then employs a human-on-the-loop approach where engineers set criteria and review aggregate outcomes rather than individual test results.
Catching JiTTests generates executable tests that demonstrate concrete behavioral differences introduced by a code change, catching classes of bugs that LLM-based code reviews might miss.
The system employs a human-on-the-loop approach, reducing friction by having engineers set criteria and review aggregate outcomes rather than supervising every output.
In the first half of 2026, Catching JiTTests caught 74 serious privacy bugs at Meta, lowered overall privacy SEVs by 20 percent, and achieved a 60 percent true-positive rate.
Catching JiTTests generates executable tests that demonstrate actual behavioral differences introduced by a change, rather than reviewing code for potential issues. It uses a dodgy diff workflow treating each diff as a mutant and an intent-aware workflow that infers the change's intention and risks. This allows it to catch classes of bugs that traditional LLM-based reviews might not.
Instead of human-in-the-loop where engineers review every output, human-on-the-loop lets engineers set criteria and review outcomes at an aggregate level. The system filters false positives via LLM-as-Judge and RubFake, then only reaches out to engineers for high-ranked catches, reducing load.
In the first half of 2026, the system caught 74 serious privacy bugs at the diff stage before they reached production, reducing overall privacy SEVs by 20 percent. It had a 60 percent true-positive rate and engaged 589 engineers at Meta.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
The AI tool trade-off: where AI compresses bounded work, what verification and governance really cost, and the condition that decides whether to rent capability or own the stack.
Air is a workspace layer that turns AI coding agents into inspectable sessions. This explainer maps its four parts: the agent, the session workspace, the review loop, and the cost signal.
A trade-offs guide to agentic AI: use agents for open-ended analysis and repair, but keep execution deterministic, context governed, and human checkpoints explicit.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.