Enhancing Agent Retrieval with Structured Chart Extraction

Summary

Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai_parse_document. Its pipeline uses ai_prep_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.

Takeaways

1. 1

   Structured chart JSON improves both answer correctness and retrieval metrics across the ViDoRe V3 subset and Chart-RAG dataset.

2. 2

   Adding the top three retrieved images at answer time improves correctness by 4 percentage points on Chart-RAG and 2.6 percentage points on the ViDoRe V3 subset, without changing retrieval.

3. 3

   The chart-JSON setup with three images reaches 75.9% correctness on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four tested multimodal embedding baselines.