Loading post…
AppleBenchmark
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Summary
The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.
Context
MT evaluation spans many language pairs, domains, and levels of annotation granularity, so LLMs must be thoroughly evaluated across these dimensions before they can be reliably used as alternatives to human evaluation.
Approach / What changed
The authors evaluated LLMs on two MT quality annotation schemes, Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), by measuring agreement with human annotators. They used a long-context test set covering 70 language pairs plus the public WMT23 and WMT25 data, analyzing both score agreement and error span annotation agreement across language pairs and domains.
Takeaways
MT evaluation spans a large number of language pairs, domains, and levels of annotation granularity, which the authors argue LLMs must be evaluated across before being used as alternatives to human evaluation.
LLM agreement with human annotators can exceed human-human agreement for some evaluation tasks, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings.
Human annotators are most challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation.
FAQ
Which MT quality annotation schemes did the study evaluate?
The study evaluated Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), comparing LLM agreement with human annotators on both. It investigated both score agreement and error span annotation agreement.
What data did the evaluation cover?
The evaluation used a long-context test set of 70 language pairs plus the publicly available WMT23 and WMT25 data. It covered a variety of language pairs and domains, examining both score and error span annotation agreement.
How do the authors propose improving annotation reliability?
The authors suggest human-LLM collaborative annotation pipelines that address the reliability issues identified in the work. They note the results highlight potential for improvement in both human and LLM annotation performance.
Source and freshness
- Source
- machinelearning.apple.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.