LLM-as-judge

Deep dive

How it works

Parts:

• Rubric — criteria, defined scale points, 1-3 few-shot anchors; fixed text, cacheable.
• Judge — model call: rubric + item(s) in, reasoning then verdict out.
• Verdict schema — structured output (reason, then winner), enforced by constrained decoding.
• Order controller — shows each pair in both orders, merges.
• Aggregator — majority over samples; Bradley-Terry fit (or Elo) over outcomes.
• Calibration set — human-labelled items for agreement and Cohen's kappa, sized so the confidence interval on kappa is narrow enough for the decision.

Operations:

1. Write the rubric, one criterion per call; reasoning first, verdict last (chain of thought; G-Eval adds a form-filling format).
2. Pointwise: score one output. Pairwise: A vs B. Reference-guided: put the reference answer in the prompt; helps most on maths and facts.
3. Order swap: call (A, B) and (B, A). Same winner both times is a win; a flip means order decided, so record a tie.
4. Repeat s times at temperature > 0, or use several judge models; take the majority (self-consistency).
5. Fit Bradley-Terry strengths on win counts (ties count ½ each).
6. Calibrate: raw agreement and kappa against human labels; accept the judge above a threshold fixed beforehand.
7. Version the judge model, rubric and prompt; when any changes, re-run the calibration set and an anchor set of old outputs to measure drift.

 item(s) ─► [rubric prefix | A | B] ─► judge ─► {reason, winner}
                 ▲ cached              (B, A) swapped, same call
        consistent? ─ yes ─► win ──┐
                 └── no ──► tie ───┴─► win matrix ─► Bradley-Terry ─► ranking
 human labels ─► agreement, kappa ◄─ judge verdicts      (gate before use)

Code

Aggregation only. The judge is a trait; the fake has a configurable position bias.

trait Judge {
    fn first_wins(&mut self, first: usize, second: usize) -> bool;
}

// fake judge: P(first wins) = sigmoid(skill diff + position_bias)
struct FakeJudge { skill: Vec<f64>, position_bias: f64, rng: u64 }

impl FakeJudge {
    fn uniform(&mut self) -> f64 {
        self.rng ^= self.rng << 13; self.rng ^= self.rng >> 7; self.rng ^= self.rng << 17;
        (self.rng >> 11) as f64 / (1u64 << 53) as f64
    }
}
impl Judge for FakeJudge {
    fn first_wins(&mut self, a: usize, b: usize) -> bool {
        let z = self.skill[a] - self.skill[b] + self.position_bias;
        self.uniform() < 1.0 / (1.0 + (-z).exp())
    }
}

// w[i][j] = wins of i over j; with both_orders a verdict that flips with the order is a tie
fn collect<J: Judge>(j: &mut J, n: usize, reps: usize, both_orders: bool) -> Vec<Vec<f64>> {
    let mut w = vec![vec![0.0; n]; n];
    for a in 0..n {
        for b in a + 1..n {
            for _ in 0..reps {
                let ab = j.first_wins(a, b);
                if !both_orders {
                    if ab { w[a][b] += 1.0 } else { w[b][a] += 1.0 }
                    continue;
                }
                let ba = j.first_wins(b, a);
                match (ab, ba) {
                    (true, false) => w[a][b] += 1.0,               // a wins in both orders
                    (false, true) => w[b][a] += 1.0,               // b wins in both orders
                    _ => { w[a][b] += 0.5; w[b][a] += 0.5 }        // order decided it: tie
                }
            }
        }
    }
    w
}

// Bradley-Terry by MM (Zermelo): p_i <- W_i / sum_j n_ij / (p_i + p_j)
fn bradley_terry(w: &[Vec<f64>], iters: usize) -> Vec<f64> {
    let n = w.len();
    let mut p = vec![1.0; n];
    for _ in 0..iters {
        let mut next = vec![0.0; n];
        for i in 0..n {
            let wins: f64 = w[i].iter().sum();
            let denom: f64 = (0..n).filter(|&j| j != i)
                .map(|j| (w[i][j] + w[j][i]) / (p[i] + p[j])).sum();
            next[i] = wins.max(1e-9) / denom;                      // guard: zero wins -> p -> 0
        }
        let g = next.iter().map(|x| x.ln()).sum::<f64>() / n as f64;
        p = next.iter().map(|x| x / g.exp()).collect();            // fix the scale (geometric mean 1)
    }
    p
}

fn cohens_kappa(a: &[usize], b: &[usize], k: usize) -> f64 {
    let n = a.len() as f64;
    let (mut ca, mut cb, mut agree) = (vec![0.0; k], vec![0.0; k], 0.0);
    for (&x, &y) in a.iter().zip(b) {
        ca[x] += 1.0; cb[y] += 1.0;
        if x == y { agree += 1.0 }
    }
    let po = agree / n;
    let pe: f64 = (0..k).map(|c| ca[c] / n * cb[c] / n).sum();    // agreement expected by chance
    (po - pe) / (1.0 - pe)
}

fn rank_order(p: &[f64]) -> Vec<usize> {
    let mut idx: Vec<usize> = (0..p.len()).collect();
    idx.sort_by(|&a, &b| p[b].partial_cmp(&p[a]).unwrap());
    idx
}

fn main() {
    // hand example: yes/yes 20, no/no 15, yes/no 5, no/yes 10
    let mut a = vec![1; 20]; a.extend(vec![0; 15]); a.extend(vec![1; 5]); a.extend(vec![0; 10]);
    let mut b = vec![1; 20]; b.extend(vec![0; 15]); b.extend(vec![0; 5]); b.extend(vec![1; 10]);
    let kappa = cohens_kappa(&a, &b, 2);
    assert!((kappa - 0.4).abs() < 1e-12);                          // po = 0.7, pe = 0.5

    let n = 8;
    let skill: Vec<f64> = (0..n).map(|i| 0.4 * i as f64).collect(); // true order: 7 > 6 > ... > 0
    let truth: Vec<usize> = (0..n).rev().collect();
    for bias in [0.0, 1.5] {
        let mut single = FakeJudge { skill: skill.clone(), position_bias: bias, rng: 42 };
        let mut both = FakeJudge { skill: skill.clone(), position_bias: bias, rng: 42 };
        let r1 = rank_order(&bradley_terry(&collect(&mut single, n, 200, false), 200));
        let r2 = rank_order(&bradley_terry(&collect(&mut both, n, 200, true), 200));
        println!("bias {bias}: single order {r1:?}   both orders {r2:?}");
        if bias > 0.0 {
            assert_ne!(r1, truth);                                 // lower index always shown first: ranking is wrong
            assert_eq!(r2, truth);                                 // order-swap cancels the bias
        }
    }
    // judge vs gold, alternating presentation order; label 1 = "first is better"
    for bias in [0.0, 1.5] {
        let mut j = FakeJudge { skill: skill.clone(), position_bias: bias, rng: 7 };
        let (mut gold, mut got) = (vec![], vec![]);
        for r in 0..20 { for a in 0..n { for b in a + 1..n {
            let (f, s) = if (a + b + r) % 2 == 0 { (a, b) } else { (b, a) };
            gold.push((skill[f] > skill[s]) as usize);
            got.push(j.first_wins(f, s) as usize);
        } } }
        println!("bias {bias}: kappa vs gold = {:.3}", cohens_kappa(&gold, &got, 2));
    }
}

Measured by running it. Without bias both protocols return the true order. With position bias 1.5 logits, skill step 0.4 per item and 200 repeats per pair, single-order judging returned [6, 4, 7, 5, 0, 3, 1, 2]; both orders returned [7, 6, 5, 4, 3, 2, 1, 0]. The single-order failure is built in: the weaker index is always shown first, so the bias helps it. Judge-vs-gold kappa on single pairs (alternating order) fell from 0.529 to 0.368 with the bias. The fake says nothing about real models.

Complexity and constants

• Calls: pointwise items × criteria × samples; all pairs n(n−1)/2 × 2 × samples (the run above: 28 × 2 × 200 = 11,200 fake calls).
• Bradley-Terry MM is O(n²) per iteration; 100-200 iterations suffice for small n.
• Decoded reasoning tokens dominate cost; the rubric prefix is identical prefill every call.

Efficiency

• Cost: one judgement = prefill of rubric + item (compute-bound) + decode of reasoning and verdict (memory-bandwidth-bound, one pass per token). Tokens ≈ (prefix + item + output) × items × samples × orders; money scales with it, wall-clock with decode speed and rate limits. No hardware figures: they depend on model and provider.
• Levers:
  • Cache the rubric prefix — shared part prefilled once (prompt caching; gain depends on provider) — rubric placed before the item.
  • Sample pairs, not all pairs — about n log n comparisons instead of n² — ranking many candidates; the comparison graph must stay connected.
  • Cheap judge first, strong judge on near-ties — tokens go where the verdict is uncertain — large sets.
  • Short reasoning budget — fewer decoded tokens — only if calibration agreement holds (measure it).
  • Stop sampling when votes are unanimous — fewer samples on easy items — clear winners.
• Naive → standard → state of the art:
  • One order, no reasoning, no calibration · cheapest, biased, unmeasured
  • Rubric + reasoning, both orders, s-sample majority, kappa on human set · about 2·s× the naive tokens
  • Cached prefix, sampled pairs, adaptive sampling, drift anchors · fewer calls, same decisions

Implementation notes

• Log which slot each candidate occupied, so bias is measurable.
• Say in the rubric that length is not a criterion; compare winner and loser lengths in logs.
• Use a judge from a different model family than the system under test.
• Keep reasoning before the verdict in the schema.
• Store every raw judgement (prompt hash, model id, reasoning, verdict) for audits.

Pitfalls

• Fixed order: the first slot wins more than it deserves (position bias).
• Verbosity bias: longer answers win.
• Self-preference: Zheng et al. call it self-enhancement; G-Eval flags a bias toward LLM-generated text.
• Raw agreement without chance correction: skewed labels give 90 % agreement at kappa near 0.
• Silent judge updates shift scores under the same model name: pin the version, re-run anchors.
• Tuning the system against one judge exploits its habits: refresh human labels.
• Weak reasoning on maths and code: use a reference or an executable check.

Variants and alternatives

In production

MT-Bench and Chatbot Arena (FastChat llm_judge), OpenAI evals model-graded evals, Inspect model_graded_qa, promptfoo rubric assertions, DeepEval G-Eval.

Implementations

Sources

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. paper · code
• Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, 2023. paper
• Bradley and Terry, Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, Biometrika 39(3/4), 1952. paper
• Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement 20(1), 1960.