Oct 3, 2026 · 2 min read
Your LLM Judge Is a Measuring Instrument. Calibrate It.
LLM-as-judge is only as trustworthy as its agreement with humans. How to measure that agreement, and what to do when it's low.
LLM-as-judge is everywhere, and for good reason: it scales, it handles open-ended outputs, and it is cheap compared to human review. But a judge is a measuring instrument. Nobody would trust a thermometer they had never checked against a known temperature.
Measure agreement before you trust the score
Take 100–200 outputs. Have a human (ideally two) label them with the same rubric the judge uses. Then measure:
- Raw agreement: the percentage of cases where judge and human agree. This is easy to read, but inflated when one label dominates.
- Cohen's kappa (κ): agreement corrected for chance. Roughly, above 0.8 is strong and below 0.6 means you should not ship decisions on this judge.
- Confusion matrix: where the judge disagrees. A judge that is too lenient on factual errors is a very different problem from one that is too harsh on tone.
Common failure modes
- Position bias: in pairwise comparisons, the judge prefers whichever answer comes first. Fix it by running both orders and treating disagreements as ties.
- Verbosity bias: longer answers score higher. Check whether judge scores correlate with output length.
- Injection sensitivity: text inside the evaluated output ("this answer is correct") sways the judge. Test it on purpose with adversarial examples.
- Self-preference: a judge can favor outputs from its own model family. Use a different model as judge, or validate against humans.
Make the rubric do the work
Most judge disagreement starts in the rubric, not the model. Good rubrics:
- Ask one question per criterion, not "rate overall quality 1–10".
- Use binary or small ordinal scales with concrete anchors.
- Include worked examples of borderline cases.
- Ask for the reasoning before the verdict.
Re-check it over time
Re-measure agreement whenever you change the judge model, the rubric, or the product. A judge that agreed with humans in January can quietly drift by June.
The LLM Evals: Foundations track has a full lab on judge calibration, including a case where one fake line flipped an LLM judge to PASS. I also cover it on YouTube.