← Back to writing

AI Product Judgment

Who Checks the LLM Judge?

✦6 min

Once a team decides some part of its AI output needs judgment, the usual next step is an LLM judge: a second model that reads each output and says whether it is good. It solves a scale problem, but the judge is a model too, and it can be wrong in the same ways.

Picture an AI that grades job-interview practice answers. A user answers "Tell me about a time you disagreed with your manager," and the grader checks whether the answer gave a clear situation, a specific action and a result. Here the judge isn't an internal tool. Its grade is the product. If it praises a vague answer or fails a strong one, the user learns the wrong lesson. I've built a small practice tool like this myself.

Practitioners agree on the starting point: before you trust a judge, check it against a person.

Agreement with one expert

Hamel Husain's guide to LLM judges treats a domain expert's labels as the ground truth. The expert marks a set of outputs pass or fail, the judge grades the same set, and you compare. (Hamel Husain, Using LLM-as-a-Judge for Evaluation)

The comparison should not be a single accuracy number. Husain recommends reporting the judge's true positive rate and true negative rate separately. (LLM judge guide) If 90% of practice answers are fine, a judge that passes everything is 90% accurate and useless. TPR tells you how often the judge agrees on the good answers. TNR tells you how often it catches the bad ones, and that second number is usually the one at risk. Eugene Yan cites a study where judges caught only 30 to 60% of inconsistent summaries while scoring above 95% on consistent ones. (Eugene Yan, Evaluating LLM-Evaluators)

How many labels? Husain suggests about 100 per failure mode, with enough passes and fails, and warns that below about 60 the confidence intervals get too wide. His FAQ with Shreya Shankar suggests splitting 100 to 200 labeled examples three ways: a train set for few-shot examples in the prompt, a dev set for tuning the prompt, and a test set you touch once at the end. (LLM judge guide; AI Evals FAQ)

What counts as good enough varies. Aman Khan suggests aiming for at least 90% agreement with human labels before trusting a judge. (Aman Khan, Beyond vibe checks) Yan adds a warning about the metric itself: in one case, Cohen's kappa came out at 0.3 to 0.5 while rank correlations on the same judge were 0.8 to 0.9, because kappa corrects for chance agreement and the others don't. (Eugene Yan)

Critique shadowing

Husain's method for building that label set is called critique shadowing. Pick one principal domain expert. Build a varied set of outputs. The expert gives each one a pass or fail plus a written critique, detailed enough to use as a few-shot example. Then revise the judge prompt until it agrees with the expert. In his Honeycomb example, agreement passed 90% after three rounds. (LLM judge guide)

For the interview grader, the expert would be someone who has run a lot of interviews. Their critiques ("no result stated," "the action is the team's, not yours") become the judge's examples.

One judge per failure, and pass or fail

Husain and Shankar recommend one focused judge per failure mode instead of one judge for "overall quality." (AI Evals FAQ) For the interview grader, that means separate checks for "names a specific situation," "describes the user's own action" and "states a result," not a single "how good is this answer" score.

They also recommend binary answers. Adjacent points on a 1 to 5 scale are hard to tell apart, annotators drift to the middle, and a 3 doesn't say what to fix. (LLM judge guide; FAQ on binary vs Likert) Yan and OpenAI lean the same way, with pairwise comparison for subjective things like tone. (Eugene Yan; OpenAI, Evaluation best practices)

Not everyone agrees. Anthropic's documentation uses 1 to 5 scales for tone, and a 2026 study found human and model ratings aligned best on a 0 to 5 scale. (Anthropic docs, Develop tests; arXiv 2601.03444) The common middle ground is to keep a scale only when every level has a written definition, or to break the scale into binary checks and add them up.

Biases to test for

Judges have known habits. Yan's summary of MT-Bench and related studies lists three. Position bias: Claude-v1 favored one slot about 70% of the time in one setup. Verbosity bias: Claude-v1 and GPT-3.5 preferred the longer answer over 90% of the time, even when the extra length added nothing. Self-preference: GPT-4's win rate on its own outputs rose about 10 points, and Claude-v1's about 25. (Eugene Yan)

Verbosity matters most for the interview grader, since good interviewers mark rambling answers down. Common fixes: swap the order in pairwise tests and count a win only if it holds both ways, keep compared answers similar in length, and ask the judge to reason before it decides. (Eugene Yan, Patterns; OpenAI)

Self-preference raises a practical question: should the judge be a different model from the one being judged? Anthropic's documentation calls a different model "generally best practice," and Yan warns about self-enhancement. Husain's FAQ says the same model is usually fine, because what matters is whether the judge agrees with human labels. (Anthropic docs; Eugene Yan, Patterns; AI Evals FAQ)

Non-English judges

If users answer in another language, expect the judge to be less reliable. A Findings of EMNLP 2025 study across 25 languages and five models found an average Fleiss' kappa of about 0.3 for cross-language consistency. Bigger models and multilingual training did not fix it. Majority voting across several models helped. (Fu & Liu, How Reliable is Multilingual LLM-as-a-Judge?) A judge validated in English needs its own labeled set in each language it grades.

A small validation plan

For a team with one grader and no eval budget, the plan fits on a page:

  1. Collect real outputs (practice answers, support replies) and have one expert mark each pass or fail with a short critique.
  2. Write one binary judge per failure mode found in those critiques.
  3. Split the labels into train, dev and test. Tune on dev, report TPR and TNR on test.
  4. Add a verbosity check: pad a few failing answers and see if they start passing.
  5. Repeat steps 1 to 3 for each language you support.

Once TPR and TNR are known, they can even correct the judge's failure rate on unlabeled production data. (AI Evals FAQ) If you have a judge in production today, start with step 1: get one expert to label the next 100 outputs, and find out how often the judge catches the bad ones.

let's build something. ✦

Happy to talk about products, AI and the calls behind them.

Say hello