---
title: "Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale"
description: "Meta has deployed Catching JiTTests, a just-in-time test generation system that uses LLMs to catch bugs before they land in production. The system runs overnight on spare capacity and targets risky diffs, using two workflows: a dodgy diff workflow treating each diff as a mutant, and an intent-aware workflow inferring the change’s intention and risks. False positives are filtered through an LLM-as-Judge ensemble and a rule-based assessor called RubFake, with a human-on-the-loop approach where engineers set criteria rather than reviewing each output. In the first half of 2026, the system caught 74 serious privacy bugs, lowered privacy SEVs by 20 percent, and engaged 589 engineers with a 60 percent true-positive rate. This approach catches classes of bugs that traditional LLM-based code reviews might miss, demonstrating that AI-generated testing can scale to reduce organizational risk."
---

# Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

[Meta](https://yomu.fyi/company/meta) · By Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu, Eric Liu · Oct 29, 2026

**Type:** Announcement

## Summary

Meta has deployed Catching JiTTests, a just-in-time test generation system that uses LLMs to catch bugs before they land in production. The system runs overnight on spare capacity and targets risky diffs, using two workflows: a dodgy diff workflow treating each diff as a mutant, and an intent-aware workflow inferring the change’s intention and risks. False positives are filtered through an LLM-as-Judge ensemble and a rule-based assessor called RubFake, with a human-on-the-loop approach where engineers set criteria rather than reviewing each output. In the first half of 2026, the system caught 74 serious privacy bugs, lowered privacy SEVs by 20 percent, and engaged 589 engineers with a 60 percent true-positive rate. This approach catches classes of bugs that traditional LLM-based code reviews might miss, demonstrating that AI-generated testing can scale to reduce organizational risk.

## Context

Meta’s codebase of hundreds of millions of lines evolves constantly, making traditional testing insufficient to catch bugs before they reach production.

## Approach / What changed

Catching JiTTests uses two tandem workflows: a dodgy diff workflow that treats each diff as a mutant to generate weak catches, and an intent-aware workflow that infers the change’s intention and risks to construct targeted tests. False positives are filtered via an LLM-as-Judge ensemble of multiple state-of-the-art models and a rule-based assessor called RubFake. The system then employs a human-on-the-loop approach where engineers set criteria and review aggregate outcomes rather than individual test results.

## Takeaways

- Catching JiTTests generates executable tests that demonstrate concrete behavioral differences introduced by a code change, catching classes of bugs that LLM-based code reviews might miss.
- The system employs a human-on-the-loop approach, reducing friction by having engineers set criteria and review aggregate outcomes rather than supervising every output.
- In the first half of 2026, Catching JiTTests caught 74 serious privacy bugs at Meta, lowered overall privacy SEVs by 20 percent, and achieved a 60 percent true-positive rate.

## FAQ

### How does Catching JiTTests differ from traditional LLM-based code review?

Catching JiTTests generates executable tests that demonstrate actual behavioral differences introduced by a change, rather than reviewing code for potential issues. It uses a dodgy diff workflow treating each diff as a mutant and an intent-aware workflow that infers the change's intention and risks. This allows it to catch classes of bugs that traditional LLM-based reviews might not.

### What is the human-on-the-loop approach used in JiTTests?

Instead of human-in-the-loop where engineers review every output, human-on-the-loop lets engineers set criteria and review outcomes at an aggregate level. The system filters false positives via LLM-as-Judge and RubFake, then only reaches out to engineers for high-ranked catches, reducing load.

### What results did Catching JiTTests achieve in 2026?

In the first half of 2026, the system caught 74 serious privacy bugs at the diff stage before they reached production, reducing overall privacy SEVs by 20 percent. It had a 60 percent true-positive rate and engaged 589 engineers at Meta.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Privacy](https://yomu.fyi/topic/privacy), [Testing](https://yomu.fyi/topic/testing)

- Source: [Meta](https://engineering.fb.com/2026/10/29/security/catching-bugs-before-they-land-jittestings-human-on-the-loop-approach-at-full-scale)
- Source URL: https://engineering.fb.com/2026/10/29/security/catching-bugs-before-they-land-jittestings-human-on-the-loop-approach-at-full-scale
- Ingested by Yomu: 2026-10-29T18:01:07.683Z

[Read original post](https://engineering.fb.com/2026/10/29/security/catching-bugs-before-they-land-jittestings-human-on-the-loop-approach-at-full-scale)
