---
title: "Enhancing Agent Retrieval with Structured Chart Extraction"
description: "Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai_parse_document. Its pipeline uses ai_prep_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines."
---

# Enhancing Agent Retrieval with Structured Chart Extraction

[Databricks](https://yomu.fyi/company/databricks) · The Databricks AI Research Team · Aug 27, 2026

**Type:** Benchmark

## Summary

Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai\_parse\_document. Its pipeline uses ai\_prep\_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.

## Context

Agents and text-based retrieval systems struggle with chart-based questions because important values and labels are embedded in figures. Captions can describe a chart’s subject but may omit the data needed for fine-grained questions, causing systems to retrieve the wrong page or lack enough information to answer correctly.

## Approach / What changed

The pipeline uses ai\_parse\_document to represent charts as structured JSON, ai\_prep\_search to create retrieval-ready chunks, a 300-million-parameter BGE text embedding model for indexing, and ai\_search connected to Genie for retrieval and answering. The evaluation compares caption-only and JSON-enriched indexes, then measures the additional effect of providing the top three retrieved images at answer time.

## Takeaways

- Structured chart JSON improves both answer correctness and retrieval metrics across the ViDoRe V3 subset and Chart-RAG dataset.
- Adding the top three retrieved images at answer time improves correctness by 4 percentage points on Chart-RAG and 2.6 percentage points on the ViDoRe V3 subset, without changing retrieval.
- The chart-JSON setup with three images reaches 75.9% correctness on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four tested multimodal embedding baselines.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Search](https://yomu.fyi/topic/search)

- Source: [Databricks](https://www.databricks.com/blog/enhancing-agent-retrieval-structured-chart-extraction)
- Source URL: https://www.databricks.com/blog/enhancing-agent-retrieval-structured-chart-extraction
- Ingested by Yomu: 2026-08-30T12:35:09.645Z

[Read original post](https://www.databricks.com/blog/enhancing-agent-retrieval-structured-chart-extraction)