Enhancing Agent Retrieval with Structured Chart Extraction

Summary

Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai_parse_document. Its pipeline uses ai_prep_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.

Context

Agents and text-based retrieval systems struggle with chart-based questions because important values and labels are embedded in figures. Captions can describe a chart’s subject but may omit the data needed for fine-grained questions, causing systems to retrieve the wrong page or lack enough information to answer correctly.

Approach / What changed

The pipeline uses ai_parse_document to represent charts as structured JSON, ai_prep_search to create retrieval-ready chunks, a 300-million-parameter BGE text embedding model for indexing, and ai_search connected to Genie for retrieval and answering. The evaluation compares caption-only and JSON-enriched indexes, then measures the additional effect of providing the top three retrieved images at answer time.

Takeaways

• Structured chart JSON improves both answer correctness and retrieval metrics across the ViDoRe V3 subset and Chart-RAG dataset.
• Adding the top three retrieved images at answer time improves correctness by 4 percentage points on Chart-RAG and 2.6 percentage points on the ViDoRe V3 subset, without changing retrieval.
• The chart-JSON setup with three images reaches 75.9% correctness on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four tested multimodal embedding baselines.

Related reading