Model vs. Harness: Where AI Agent Controls Belong
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
A side-by-side comparison of two ways to improve ai_extract document extraction: patching one field at a time, or versioning prompts and judging them against fixed ground truth.
ai_extract turns messy PDFs into structured fields, and the hard part is knowing whether a change actually helped. This comparison sets ad-hoc field patching against a measured loop: candidates live in a prompt registry with aliases, field rules sit in JSON schema descriptions while document-level reasoning goes in a global prompt, and each candidate is scored against fixed ground truth over several runs. A BETTER, WORSE, or INCONCLUSIVE verdict filters run-to-run noise before promotion, which is why one team cut total errors by 32% and revision errors by 64% instead of trading one regression for another.
Run-to-run variation can make an unchanged prompt look better or worse, so a single measurement is not evidence. Scoring every candidate against fixed ground truth over three runs lets the team return a BETTER, WORSE, or INCONCLUSIVE verdict and filter noise before promotion.
Field-specific rules belong in JSON schema descriptions, while document-level reasoning goes in a global prompt so it is not repeated per field. Precision mode handles complex structures, and citations are used to diagnose why a specific field failed.
In one public-sector proof of value, a fine-tuned Phi-4 Mini scored 30 on a 34-point rubric against GPT-4.1's 29, with JSON validity rising from 1% to 99%. Self-hosted small models also keep data inside the environment, which matters when extraction input cannot leave.
The posts behind it
Document extraction across dozens of PDF layouts is unreliable when fields are tuned in isolation; fixing one field can cause hidden regressions and apparent improvements may stem from normal run-to-run variation. The project extracting roughly 40 structured fields from partner-submitted order documents used human reviewers who corrected and approved extracted values, providing ground truth for evaluation. Field-specific rules go in JSON schema descriptions, document-level reasoning in a global prompt, precision mode handles complex structures, and citations diagnose failures. Candidates are versioned in MLflow Prompt Registry with aliases and evaluated against fixed ground truth using three runs to filter noise before promotion, outputting a BETTER, WORSE, or INCONCLUSIVE verdict. This approach reduced total errors by 32% and revision errors by 64%, creating a repeatable and regression-safe extraction workflow now automated by Genie Code.
Jev is a System One model that returns typed answers with calibrated probabilities in 70–500 ms without generating text, used for table ranking and failure labeling. On the Agentic Analytics Benchmark, Jev labeled 277 failures across 4 models in under a second, revealing wrong data selection as the largest bucket, mostly entity resolution errors like wrong identifiers or unresolved named entities. Jev also ranked 18 warehouse tables per question at nDCG@10 0.951 in 0.4 seconds with no fine-tuning or gold data. The labeler flagged 18 likely grader errors under a 0.65 threshold, traced to a comparator that returns not equal on row-count differences before linking columns, now tracked as a bug. However, feeding these rankings as schema hints left accuracy flat across 20 models, and development-set savings did not transfer to the held-out board.
The post presents a study by Ai2 on the reliability of LLMs as novelty judges for scientific research ideas, using six models including GPT-5.4 and Claude Opus 4.6. The researchers mined ICLR 2026 peer reviews to create a validated pool of 154 high-novelty and 145 lower-novelty human ideas, then paired them with 154 simple AI-generated ideas. Testing revealed that small prompt changes could shift accuracy by over 50 percentage points on AI-generated ideas, such as GPT-5.4 dropping from 93.5% to 40.9% with a simpler prompt. Additional search and reasoning did not consistently improve judgments; a simple Claude Opus 4.6 baseline achieved 0.854 macro-F1 at $2.59, while AI Scientist scored 0.772 at $82.05. The findings indicate that LLM-based novelty evaluation remains unreliable, particularly for AI-generated ideas, and the authors release benchmark data and an automated pipeline for future research.
Atos completed a proof of value for a German public sector customer, using Red Hat OpenShift AI to fine-tune small language models for structured data extraction and violence detection from social media comments. The challenge required self-hosted models for data sovereignty, extracting 14 features into valid JSON, and accurately identifying violence triggers. The solution combined synthetic data generation with GPT-4.1 as teacher and supervised fine-tuning on three student models (Phi-4 Mini 3.8B, Qwen 2.5 1.5B, Qwen 2.5 7B) trained on only 3 NVIDIA L40 GPUs. A custom 34-point rubric evaluated JSON validity, entity extraction, and violence detection against domain-specific eval data. The fine-tuned Phi-4 Mini scored 30 points, outperforming GPT-4.1’s 29 points, with JSON validity reaching 99% from a base of 1%, while the Qwen 2.5 7B scored 30.8 points.
More like this
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
Compare two places to keep Claude's working state: curated files on your own machine versus a durable session store or isolated cloud computer, and learn which one fits your deployment.
CIMD uses an HTTPS URL as an OAuth client identifier that points to a metadata document. This comparison shows when to pre-register clients manually and when to let dynamic discovery handle agent onboarding.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.