---
title: "Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged"
description: "The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work."
---

# Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

[Apple](https://yomu.fyi/company/apple) · Oct 12, 2026

**Type:** Benchmark

## Summary

The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.

## Context

MT evaluation spans many language pairs, domains, and levels of annotation granularity, so LLMs must be thoroughly evaluated across these dimensions before they can be reliably used as alternatives to human evaluation.

## Approach / What changed

The authors evaluated LLMs on two MT quality annotation schemes, Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), by measuring agreement with human annotators. They used a long-context test set covering 70 language pairs plus the public WMT23 and WMT25 data, analyzing both score agreement and error span annotation agreement across language pairs and domains.

## Takeaways

- MT evaluation spans a large number of language pairs, domains, and levels of annotation granularity, which the authors argue LLMs must be evaluated across before being used as alternatives to human evaluation.
- LLM agreement with human annotators can exceed human-human agreement for some evaluation tasks, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings.
- Human annotators are most challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation.

## FAQ

### Which MT quality annotation schemes did the study evaluate?

The study evaluated Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), comparing LLM agreement with human annotators on both. It investigated both score agreement and error span annotation agreement.

### What data did the evaluation cover?

The evaluation used a long-context test set of 70 language pairs plus the publicly available WMT23 and WMT25 data. It covered a variety of language pairs and domains, examining both score and error span annotation agreement.

### How do the authors propose improving annotation reliability?

The authors suggest human-LLM collaborative annotation pipelines that address the reliability issues identified in the work. They note the results highlight potential for improvement in both human and LLM annotation performance.

**Tags:** [Benchmark](https://yomu.fyi/topic/benchmark), [Evaluation](https://yomu.fyi/topic/evaluation), [LLMs](https://yomu.fyi/topic/llm), [Machine Translation](https://yomu.fyi/topic/machine-translation)

- Source: [Apple](https://machinelearning.apple.com/research/llm-translation-quality-annotation)
- Source URL: https://machinelearning.apple.com/research/llm-translation-quality-annotation
- Ingested by Yomu: 2026-10-13T00:00:23.175Z

[Read original post](https://machinelearning.apple.com/research/llm-translation-quality-annotation)