---
title: "When LLM judges agree, should we believe them?"
description: "Multijudge panels in LLM-as-a-judge evaluations often suffer from correlated errors when models share prompt templates, training lineages, or evaluation rubrics. To account for these interdependencies without human reference labels, a method presented at ICML uses Ising models to simultaneously infer judge reliability and pairwise correlation. The unsupervised algorithm iteratively updates the soft probability of each item's true latent label while estimating both individual judge skill and pairwise dependence structures. Evaluated across retrieval relevance, toxicity classification, and summarization benchmarks with ten zero-temperature judges, the approach achieved test accuracies of 0.912, 0.792, and 0.806. These dependence-aware aggregation scores consistently outperformed weighted majority voting baselines by 9% to 14% on standard metrics across all three tasks."
---

# When LLM judges agree, should we believe them?

amazon.science · Krishna Balasubramanian, Sasha Podkopaev · Aug 26, 2026

**Type:** Problem & solution

## Summary

Multijudge panels in LLM-as-a-judge evaluations often suffer from correlated errors when models share prompt templates, training lineages, or evaluation rubrics. To account for these interdependencies without human reference labels, a method presented at ICML uses Ising models to simultaneously infer judge reliability and pairwise correlation. The unsupervised algorithm iteratively updates the soft probability of each item's true latent label while estimating both individual judge skill and pairwise dependence structures. Evaluated across retrieval relevance, toxicity classification, and summarization benchmarks with ten zero-temperature judges, the approach achieved test accuracies of 0.912, 0.792, and 0.806. These dependence-aware aggregation scores consistently outperformed weighted majority voting baselines by 9% to 14% on standard metrics across all three tasks.

## Context

Standard aggregation methods like majority vote and weighted majority vote assume that judge models make errors independently. In LLM-as-a-judge pipelines, this assumption fails when judges share prompt templates, training lineages, model families, or evaluation rubrics, causing multijudge panels to reinforce shared blind spots.

## Approach / What changed

The proposed method aggregates labels in an unsupervised setting by modeling the panel as an Ising network. The algorithm treats the true label as a latent variable, iteratively updating soft label probabilities while estimating individual judge reliability profiles and pairwise correlation parameters directly from evaluation logs.

## Takeaways

- Dependence-aware aggregation uses Ising models in an unsupervised loop, alternating between estimating latent label probabilities and updating judge reliability with pairwise correlation parameters.
- In benchmarks with 10 zero-temperature judges, the method achieved accuracies of 0.912 on relevance, 0.792 on toxicity, and 0.806 on summarization, outperforming historical-accuracy baselines by 9% to 14%.
- A class-dependent variant allows the pairwise relationship patterns between judges to change depending on the label, capturing setups where judges agree broadly on clear items but form clusters on ambiguous ones.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

- Source: [amazon.science](https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them)
- Source URL: https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them
- Ingested by Yomu: 2026-08-27T15:24:22.420Z

[Read original post](https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them)