Direct Preference Optimization Beyond Chatbots

Summary

DharmaOCR applies Direct Preference Optimization (DPO) beyond chat alignment to reduce text degeneration in structured OCR, where outputs are either acceptable transcriptions or failure loops. In a benchmark of Brazilian Portuguese structured-document OCR, vanilla degeneration rates among open-source model families ranged from below 1% to above 33%, while supervised fine-tuning (SFT) did not consistently reach production-acceptable levels. After SFT, the pipeline generated multiple candidates for 23,726 documents, used an automated LLM judge to score them, and retained the model’s degenerate outputs as rejected examples paired with chosen outputs. DPO reduced degeneration in every tested family, by 59.4% on average and up to 87.6%, without reducing extraction quality according to the benchmark; applicability is presented as dependent on failures being categorically identifiable, reliably scoreable, and sufficiently numerous.

Takeaways

1. 1

   SFT optimizes token-level likelihood and does not explicitly penalize completion-level repetition loops, so its effect on degeneration was inconsistent.

2. 2

   The DPO dataset used multiple SFT-generated candidates from 23,726 documents and an automated LLM judge; degenerate outputs were preserved as rejected examples.

3. 3

   DPO reduced degeneration in every tested model family, with a 59.4% average reduction and an 87.6% best case; one reported rate fell from 1.61% to 0.20%.