Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

Summary

Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance.

Takeaways

1. 1

   CapQuiz is reference-free and evaluates caption information fidelity through multiple-choice questions derived from the associated video.

2. 2

   Its taxonomy contains 10 question types spanning Descriptive and Inferential categories across 24 diverse video domains.

3. 3

   CapF1 combines CapP, which measures factuality, with CapR, which measures coverage; experiments report significantly better correlation with human judgments than existing metrics.