How to Build on BlueField with DOCA Agent Skills
A practical how-to for using NVIDIA DOCA Agent Skills to generate BlueField code with verified APIs and hardware checks, reducing correction cycles and failed builds.
Loading post…
Hugging FaceEsakkivel Esakkiraja, Shruthan Radhakrishna, Denis Akhiyarov, Sagar DavasamProblem & solution
AutoSynthData: Generating Training Data for Enterprise Agents
Summary
AutoSynthData, built by ServiceNow CoreAI, generates training data for enterprise agents by targeting capability gaps from evaluation, using a stronger teacher such as Qwen3.8-27B to demonstrate successes. It uses a target model’s failures and a stronger teacher’s successes to decide what to learn next. The system produces tasks composing a system specification, user prompt, and verifier, ensuring feasibility, realism, and difficulty against the target environment. After sample-level checks and batch-level review, the dataset grows via target and multiply phases, generating 2,000 samples in 18 hours for the Hybrid domain. In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM, showing the approach works across domains.
Context
Enterprise agents struggle with environment-specific workflows and rules. Generating useful training data that targets these weaknesses is difficult because tasks must be feasible, realistic, and solvable with a reliable verifier. AutoSynthData uses the model’s evaluation failures and a stronger teacher to create such data.
Approach / What changed
AutoSynthData first evaluates the target model on diagnostic tasks to identify capability gaps. A stronger teacher characterizes solvable tasks and successful trajectories. Capability specification cards are created, omitting original prompts and details. The system then generates tasks in two phases: target (core samples) and multiply (variants). Each candidate undergoes validation, execution, solver evaluation, and repair. The accepted samples are used for supervised fine-tuning. After training, the updated model is evaluated, and remaining gaps guide the next generation round.
Takeaways
AutoSynthData uses a target model’s failures and a stronger teacher’s successes to define what the model should learn next, creating a curriculum that adapts as the model improves.
Generated tasks must pass validation for feasibility (possible in environment), realism (plausible user request), and difficulty (not already solved), with a verifier that is consistent, sound, and complete.
In EnterpriseOps Gym, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 percentage points (35% relative) in Hybrid and from 18.77% to 27.18% in ITSM.
FAQ
How does AutoSynthData ensure generated tasks are useful?
It validates each task for feasibility, realism, and difficulty against the target environment. It also verifies that the task has a consistent, sound, and complete verifier. Sample-level checks and batch-level review further improve quality.
What are the two phases of data generation?
The target phase creates core training samples from capability specifications. The multiply phase expands them into novel variants with different states, prompts, and solutions, anchored to the vetted target set to limit drift.
What results were achieved in EnterpriseOps Gym?
In the Hybrid domain, synthetic SFT on Gemma-4-26B-A4B-it improved mean Pass@1 by 7.2 points (35% relative) and verifier success from 63.01% to 68.55%. In ITSM, Pass@1 rose from 18.77% to 27.18%.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.