---
title: "Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering"
description: "Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance."
---

# Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

[Apple](https://yomu.fyi/company/apple) · Sep 11, 2026

**Type:** Benchmark

## Summary

Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance.

## Context

Existing video-caption evaluation metrics mainly match generated captions against ground-truth references. This can penalize high-quality captions because video description has a one-to-many nature, and these metrics provide limited fine-grained analysis of caption quality.

## Approach / What changed

CapQuiz evaluates captions through their utility in answering human-verified, fine-grained multiple-choice questions derived from videos. It covers 10 question types in Descriptive and Inferential categories across 24 video domains, while CapF1 combines CapP for factuality and CapR for coverage.

## Takeaways

- CapQuiz is reference-free and evaluates caption information fidelity through multiple-choice questions derived from the associated video.
- Its taxonomy contains 10 question types spanning Descriptive and Inferential categories across 24 diverse video domains.
- CapF1 combines CapP, which measures factuality, with CapR, which measures coverage; experiments report significantly better correlation with human judgments than existing metrics.

**Tags:** [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Multimodal](https://yomu.fyi/topic/multimodal), [Video](https://yomu.fyi/topic/video)

- Source: [Apple](https://machinelearning.apple.com/research/video-caption-quality)
- Source URL: https://machinelearning.apple.com/research/video-caption-quality
- Ingested by Yomu: 2026-09-11T16:03:37.086Z

[Read original post](https://machinelearning.apple.com/research/video-caption-quality)