Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

Summary

This guide presents a low-cost recipe for improving structured-output compliance in Liquid AI’s LFM2.5-350M, a task where parseable, schema-adherent responses determine whether an LLM can feed downstream systems. It fine-tunes the model with Group Relative Policy Optimization (GRPO) through TRL, using about 500 samples, 100 steps, three weighted rewards for format, field count, and JSON Schema validation, and a LoRA adapter training roughly 6M parameters. Prompt augmentation covers fenced-code-block instructions and top-level-array tasks, while evaluation uses IFStruct with llama.cpp on an OpenAI-compatible local server. On 2,000 IFStruct samples, the BF16 base model passes 22.6%, rising to 29.7% after tuning. JSON improves from 18.0% to 31.9%, bare-list compliance from 16.6% to 29.7%, while YAML changes little.

Takeaways

1. 1

   The three rewards use weights [1.0, 0.5, 2.0]: format correctness, expected top-level field count, and JSON Schema validation, respectively.

2. 2

   Training runs for 100 steps with eight generations per prompt group, a per-device batch size of four, gradient accumulation of eight, and a maximum completion length of 1,024.

3. 3

   Average latency increased from 1,453 ms for the base model to 1,518 ms for the GRPO-tuned model in the reported IFStruct evaluations.