---
title: "From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry"
description: "Document extraction across dozens of PDF layouts is unreliable when fields are tuned in isolation; fixing one field can cause hidden regressions and apparent improvements may stem from normal run-to-run variation. The project extracting roughly 40 structured fields from partner-submitted order documents used human reviewers who corrected and approved extracted values, providing ground truth for evaluation. Field-specific rules go in JSON schema descriptions, document-level reasoning in a global prompt, precision mode handles complex structures, and citations diagnose failures. Candidates are versioned in MLflow Prompt Registry with aliases and evaluated against fixed ground truth using three runs to filter noise before promotion, outputting a BETTER, WORSE, or INCONCLUSIVE verdict. This approach reduced total errors by 32% and revision errors by 64%, creating a repeatable and regression-safe extraction workflow now automated by Genie Code."
---

# From spot fixes to measured gains: improving document extraction accuracy with ai\_extract and MLflow Prompt Registry

[Databricks](https://yomu.fyi/company/databricks) · Kristen Drexinger, Seshendranath Balla · Oct 27, 2026

**Type:** Problem & solution

## Summary

Document extraction across dozens of PDF layouts is unreliable when fields are tuned in isolation; fixing one field can cause hidden regressions and apparent improvements may stem from normal run-to-run variation. The project extracting roughly 40 structured fields from partner-submitted order documents used human reviewers who corrected and approved extracted values, providing ground truth for evaluation. Field-specific rules go in JSON schema descriptions, document-level reasoning in a global prompt, precision mode handles complex structures, and citations diagnose failures. Candidates are versioned in MLflow Prompt Registry with aliases and evaluated against fixed ground truth using three runs to filter noise before promotion, outputting a BETTER, WORSE, or INCONCLUSIVE verdict. This approach reduced total errors by 32% and revision errors by 64%, creating a repeatable and regression-safe extraction workflow now automated by Genie Code.

## Context

A project extracting roughly 40 structured fields from partner-submitted order documents across dozens of PDF layouts faced unreliable results from ad-hoc prompt tuning: fixing one field often introduced regressions elsewhere, and changes could appear beneficial purely due to normal run-to-run variation. Human reviewers corrected and approved extracted values before they moved downstream, providing a ground-truth evaluation dataset that was not yet being used systematically.

## Approach / What changed

Build a repeatable extraction workflow: (1) Determine which ai\_extract lever to use — schema descriptions for field-specific rules, global instructions for document-level reasoning, precision mode for structural problems, and citations to diagnose failures. (2) Version prompts and schemas in MLflow Prompt Registry with aliases (@prod) for deployment and rollback without pipeline code changes. (3) Evaluate candidates against fixed human-reviewed ground truth using three runs per candidate, track accuracy, coverage, and noise floor, and classify as BETTER, WORSE, or INCONCLUSIVE before promoting to production.

## Takeaways

- Place field-specific rules in schema descriptions rather than the global prompt to keep the extraction contract inspectable and prevent the global prompt from accumulating unrelated exceptions.
- Use precision mode in ai\_extract 2.1 for documents with large schemas, repeated line items, or complex tables; it reduces hallucination and over-extraction even when prompt and schema are held constant.
- Evaluate every extraction change against a fixed set of human-reviewed documents, running each candidate three times to filter out noise, before promoting it to production.

## FAQ

### How should I decide whether to put a rule in the schema or the global prompt?

If the rule defines behavior for a single field, put it in the schema description. If the rule defines how to interpret the document as a whole (e.g., source priority, cross-field consistency), put it in the global prompt.

### When is precision mode most effective?

Precision mode helps most on repeated structures like tables and line items, reducing over-extraction where the model returns a value when the correct answer is blank. It is complementary to schema rules: use precision mode when the model understands field definitions but struggles with data structure.

### How do I avoid promoting a change that causes regressions?

Run each candidate three times on a fixed human-reviewed evaluation set, compare metrics against the baseline, and calculate a noise floor from the baseline runs. Only promote changes classified as BETTER; reject WORSE changes and investigate INCONCLUSIVE ones before making a human decision.

**Tags:** [ai\_extract](https://yomu.fyi/topic/ai-extract), [Document Extraction](https://yomu.fyi/topic/document-extraction), [MLflow](https://yomu.fyi/topic/mlflow), [Prompt Registry](https://yomu.fyi/topic/prompt-registry)

- Source: [Databricks](https://www.databricks.com/blog/spot-fixes-measured-gains-improving-document-extraction-accuracy-aiextract-and-mlflow-prompt)
- Source URL: https://www.databricks.com/blog/spot-fixes-measured-gains-improving-document-extraction-accuracy-aiextract-and-mlflow-prompt
- Ingested by Yomu: 2026-10-27T18:00:12.517Z

[Read original post](https://www.databricks.com/blog/spot-fixes-measured-gains-improving-document-extraction-accuracy-aiextract-and-mlflow-prompt)
