Loading…
Topic
1 post about Machine Translation. Every summary links to the original.
See the idea
Compare two ways to adapt a model to a new language or domain: generate a broad synthetic corpus like AI4Bharat's, or fine-tune on one narrow task and evaluate it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.
The best engineering reads, once a week.
