AI for Science Is a Governed Loop, Not a Single Model
AI for Science is a governed workflow that combines domain skills, context, adapted models, and evaluation to turn scientific questions into reproducible evidence.
DatabricksKristen Drexinger, Seshendranath BallaProblem & solution
Document extraction across dozens of PDF layouts is unreliable when fields are tuned in isolation; fixing one field can cause hidden regressions and apparent improvements may stem from normal run-to-run variation. The project extracting roughly 40 structured fields from partner-submitted order documents used human reviewers who corrected and approved extracted values, providing ground truth for evaluation. Field-specific rules go in JSON schema descriptions, document-level reasoning in a global prompt, precision mode handles complex structures, and citations diagnose failures. Candidates are versioned in MLflow Prompt Registry with aliases and evaluated against fixed ground truth using three runs to filter noise before promotion, outputting a BETTER, WORSE, or INCONCLUSIVE verdict. This approach reduced total errors by 32% and revision errors by 64%, creating a repeatable and regression-safe extraction workflow now automated by Genie Code.
A project extracting roughly 40 structured fields from partner-submitted order documents across dozens of PDF layouts faced unreliable results from ad-hoc prompt tuning: fixing one field often introduced regressions elsewhere, and changes could appear beneficial purely due to normal run-to-run variation. Human reviewers corrected and approved extracted values before they moved downstream, providing a ground-truth evaluation dataset that was not yet being used systematically.
Build a repeatable extraction workflow: (1) Determine which ai_extract lever to use — schema descriptions for field-specific rules, global instructions for document-level reasoning, precision mode for structural problems, and citations to diagnose failures. (2) Version prompts and schemas in MLflow Prompt Registry with aliases (@prod) for deployment and rollback without pipeline code changes. (3) Evaluate candidates against fixed human-reviewed ground truth using three runs per candidate, track accuracy, coverage, and noise floor, and classify as BETTER, WORSE, or INCONCLUSIVE before promoting to production.
Place field-specific rules in schema descriptions rather than the global prompt to keep the extraction contract inspectable and prevent the global prompt from accumulating unrelated exceptions.
Use precision mode in ai_extract 2.1 for documents with large schemas, repeated line items, or complex tables; it reduces hallucination and over-extraction even when prompt and schema are held constant.
Evaluate every extraction change against a fixed set of human-reviewed documents, running each candidate three times to filter out noise, before promoting it to production.
If the rule defines behavior for a single field, put it in the schema description. If the rule defines how to interpret the document as a whole (e.g., source priority, cross-field consistency), put it in the global prompt.
Precision mode helps most on repeated structures like tables and line items, reducing over-extraction where the model returns a value when the correct answer is blank. It is complementary to schema rules: use precision mode when the model understands field definitions but struggles with data structure.
Run each candidate three times on a fixed human-reviewed evaluation set, compare metrics against the baseline, and calculate a noise floor from the baseline runs. Only promote changes classified as BETTER; reject WORSE changes and investigate INCONCLUSIVE ones before making a human decision.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
AI for Science is a governed workflow that combines domain skills, context, adapted models, and evaluation to turn scientific questions into reproducible evidence.
Agents CLI is the command-line surface where you run a coding agent, score its output against a dataset, and trace what happened, so quality becomes a loop instead of a guess.
Agent tracing records prompts, tool calls, and outputs as one span-linked run, then turns that record into eval datasets, alerts, and regression checks. Here is the five-stage flow from capture to decision.
A side-by-side comparison of two ways to improve ai_extract document extraction: patching one field at a time, or versioning prompts and judging them against fixed ground truth.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.