How we built a custom vision LLM to improve document processing at Grab

Summary

Grab developed a specialized Vision LLM for extracting text and key information from Southeast Asian identity cards, driver’s licenses, and registration certificates, where varied languages and layouts challenged traditional OCR and existing models. It selected Qwen2-VL 2B for its efficient size, SEA language tokenizer coverage, and dynamic native-resolution image handling, while synthetic Common Crawl text images and Documint supplied reviewed training labels for detection, orientation, OCR, and KIE. LoRA fine-tuning improved Latin-script documents but struggled with Thai, Vietnamese, and dense unstructured layouts, leading to two-stage full-parameter training with synthetic SEA text images followed by task-specific document fine-tuning. The fully fine-tuned 2B model raised Thai accuracy by 70 percentage points and Vietnamese accuracy by 40 points over baseline, while a custom roughly 1B-parameter model combining Qwen2-VL 2B’s vision encoder with Qwen2.5 0.5B’s decoder stayed within a 3-point accuracy gap across most document types and achieved lower latency.

Takeaways

1. 1

   Full-parameter fine-tuning substantially outperformed LoRA for specialized documents containing non-Latin Southeast Asian scripts.

2. 2

   The fully fine-tuned Qwen2-VL 2B model improved Thai document accuracy by 70 percentage points and Vietnamese document accuracy by 40 percentage points over baseline.

3. 3

   The custom roughly 1B-parameter model remained within a 3-point accuracy gap of the 2B model across most document types while achieving lower latency.