ai_extract: Spot Fixes vs. a Measured Eval Loop
A side-by-side comparison of two ways to improve ai_extract document extraction: patching one field at a time, or versioning prompts and judging them against fixed ground truth.
Red HatAditi Saluja, Mustafa EyceozExplainer
Atos completed a proof of value for a German public sector customer, using Red Hat OpenShift AI to fine-tune small language models for structured data extraction and violence detection from social media comments. The challenge required self-hosted models for data sovereignty, extracting 14 features into valid JSON, and accurately identifying violence triggers. The solution combined synthetic data generation with GPT-4.1 as teacher and supervised fine-tuning on three student models (Phi-4 Mini 3.8B, Qwen 2.5 1.5B, Qwen 2.5 7B) trained on only 3 NVIDIA L40 GPUs. A custom 34-point rubric evaluated JSON validity, entity extraction, and violence detection against domain-specific eval data. The fine-tuned Phi-4 Mini scored 30 points, outperforming GPT-4.1’s 29 points, with JSON validity reaching 99% from a base of 1%, while the Qwen 2.5 7B scored 30.8 points.
German public sector customer required self-hosted models for data sovereignty and needed to perform structured analysis on unstructured German social media comments, extracting 14 features into valid JSON and detecting violence calls to action.
Synthetic data generation using GPT-4.1 as teacher model to create training data, then supervised fine-tuning (SFT) and orthogonal subspace fine-tuning (OSFT) on student models (Phi-4 Mini 3.8B, Qwen 2.5 1.5B, Qwen 2.5 7B) using Red Hat OpenShift AI with 3 NVIDIA L40 GPUs. Evaluation used a custom 34-point rubric measuring JSON validity, entity extraction, and violence detection.
A 1-billion parameter model, when fine-tuned, can outperform GPT-4.1 in domain-specific tasks like structured extraction from German social media.
Custom evaluation frameworks with domain-specific rubrics and eval data are essential for measuring true production readiness, as generic benchmarks miss nuanced requirements.
High-performance SLM training is achievable with modest hardware: only 3 NVIDIA L40 GPUs were used for fine-tuning, making customization accessible.
Three student models were fine-tuned: Phi-4 Mini 3.8B, Qwen 2.5 1.5B, and Qwen 2.5 7B, using synthetic data from GPT-4.1 as teacher.
Phi-4 Mini base model produced valid JSON only 1% of the time, but after fine-tuning it reached 99% validity, demonstrating huge improvement.
Training used only 3 NVIDIA L40 GPUs on Red Hat OpenShift AI; no GPU was needed for data generation as the teacher model was accessed via API.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A side-by-side comparison of two ways to improve ai_extract document extraction: patching one field at a time, or versioning prompts and judging them against fixed ground truth.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.