---
title: "AutoSynthData: Generating Training Data for Enterprise Agents"
description: "AutoSynthData, built by ServiceNow CoreAI, generates training data for enterprise agents by targeting capability gaps from evaluation, using a stronger teacher such as Qwen3.8-27B to demonstrate successes. It uses a target model’s failures and a stronger teacher’s successes to decide what to learn next. The system produces tasks composing a system specification, user prompt, and verifier, ensuring feasibility, realism, and difficulty against the target environment. After sample-level checks and batch-level review, the dataset grows via target and multiply phases, generating 2,000 samples in 18 hours for the Hybrid domain. In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM, showing the approach works across domains."
---

# AutoSynthData: Generating Training Data for Enterprise Agents

[Hugging Face](https://yomu.fyi/company/hugging-face) · Esakkivel Esakkiraja, Shruthan Radhakrishna, Denis Akhiyarov, Sagar Davasam · Oct 2, 2026

**Type:** Problem & solution

## Summary

AutoSynthData, built by ServiceNow CoreAI, generates training data for enterprise agents by targeting capability gaps from evaluation, using a stronger teacher such as Qwen3.8-27B to demonstrate successes. It uses a target model’s failures and a stronger teacher’s successes to decide what to learn next. The system produces tasks composing a system specification, user prompt, and verifier, ensuring feasibility, realism, and difficulty against the target environment. After sample-level checks and batch-level review, the dataset grows via target and multiply phases, generating 2,000 samples in 18 hours for the Hybrid domain. In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM, showing the approach works across domains.

## Context

Enterprise agents struggle with environment-specific workflows and rules. Generating useful training data that targets these weaknesses is difficult because tasks must be feasible, realistic, and solvable with a reliable verifier. AutoSynthData uses the model’s evaluation failures and a stronger teacher to create such data.

## Approach / What changed

AutoSynthData first evaluates the target model on diagnostic tasks to identify capability gaps. A stronger teacher characterizes solvable tasks and successful trajectories. Capability specification cards are created, omitting original prompts and details. The system then generates tasks in two phases: target (core samples) and multiply (variants). Each candidate undergoes validation, execution, solver evaluation, and repair. The accepted samples are used for supervised fine-tuning. After training, the updated model is evaluated, and remaining gaps guide the next generation round.

## Takeaways

- AutoSynthData uses a target model’s failures and a stronger teacher’s successes to define what the model should learn next, creating a curriculum that adapts as the model improves.
- Generated tasks must pass validation for feasibility (possible in environment), realism (plausible user request), and difficulty (not already solved), with a verifier that is consistent, sound, and complete.
- In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 percentage points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM.

## FAQ

### How does AutoSynthData ensure generated tasks are useful?

It validates each task for feasibility, realism, and difficulty against the target environment. It also verifies that the task has a consistent, sound, and complete verifier. Sample-level checks and batch-level review further improve quality.

### What are the two phases of data generation?

The target phase creates core training samples from capability specifications. The multiply phase expands them into novel variants with different states, prompts, and solutions, anchored to the vetted target set to limit drift.

### What results were achieved in EnterpriseOps Gym?

In the Hybrid domain, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) and verifier success from 63.01% to 68.55%. In ITSM, Pass@1 rose from 18.77% to 27.18%.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Fine-Tuning](https://yomu.fyi/topic/fine-tuning), [Synthetic Data](https://yomu.fyi/topic/synthetic-data)

- Source: [Hugging Face](https://huggingface.co/blog/ServiceNow-AI/autosynthdata)
- Source URL: https://huggingface.co/blog/ServiceNow-AI/autosynthdata
- Ingested by Yomu: 2026-10-02T06:00:08.895Z

[Read original post](https://huggingface.co/blog/ServiceNow-AI/autosynthdata)