← Back to writing

AI Product Judgment

Most of Your AI Evals Should Be Boring Code

✦5 min

When a team starts evaluating AI output, the conversation usually jumps straight to using a second model as the judge. Have one model grade what another model wrote, put a score on a dashboard, done. It feels right: the output came from a model, so a model should check it.

People who run evals for a living mostly say the opposite. Start with plain code, and bring in a model judge only for the questions code can't answer.

Three kinds of grader

Anthropic's guide to agent evals sorts graders into three types. (Anthropic, Demystifying evals for AI agents)

  • Code-based graders are fast, cheap and give the same answer every time. Their weakness is that they can fail a correct answer that is phrased in a way the rule didn't expect.
  • Model-based graders can handle open-ended output, but they need calibrating against human judgment, and they don't always give the same answer twice.
  • Human review is the gold standard. It is also the slowest and most expensive, so it is mostly used to check the other two.

Hamel Husain and Shreya Shankar, who teach a widely followed evals course for engineers and PMs, turn this into a rule: if code can detect the failure, use code. Save the LLM judge for calls that need judgment, and give each judge one specific failure to look for instead of an "overall quality" score. (AI Evals FAQ)

What code can catch

Picture an AI feature that writes product descriptions for an online store. Before anyone asks whether a description is persuasive, a long list of failures can be caught with simple checks:

  • The output isn't valid JSON, or a required field is missing.
  • The description is longer than the page allows.
  • A price or size in the text doesn't match the product data.
  • A banned claim shows up, such as "cures" or "guaranteed".
  • A link points to a page that doesn't exist.
  • The text is in the wrong language.

Each of these is a yes or no question. A rule answers it in milliseconds, for free, on every output, and gives the same answer tomorrow. When a check fails, it says exactly what to fix.

Teams usually find that a large share of real failures look like this. They are boring, and they are also the failures users notice first. On a travel site I run, the most common problems were broken formatting and pages that rendered an empty card, not bad writing.

Where the rules come from

The best rules don't come from brainstorming. They come from reading real outputs.

Husain and Shankar call this error analysis and describe it as the most important part of evals: read a sample of real outputs, write a short note on what went wrong in each, then group the notes into categories. (AI Evals FAQ) Many of those categories turn out to be things a rule can catch. The rest become the job of a model judge or a human reviewer.

Shankar's research explains why you can't write the full list up front. People need criteria to grade outputs, but grading outputs is how they discover the criteria. Some failures only become obvious after you see them. (Shankar et al., Who Validates the Validators?) So the rule list grows over time: every time a new kind of broken output shows up, someone writes a check so nobody has to look for it again.

Regression checks and capability checks

Anthropic separates two kinds of eval. Capability evals start with a low pass rate and give the team something to improve. Regression evals should pass close to 100% of the time, and their job is to protect what already works. When a capability is reliably solved, its tests move into the regression set. (Anthropic)

Code checks are a natural fit for the regression set. They are cheap enough to run on every change, and because they are deterministic, a failure means something changed, not that the grader had a bad day. Husain's FAQ recommends the same thing for evals that run in CI: keep them small, fast and mostly made of code assertions, and add each real production failure as a new case. (AI Evals FAQ)

Some teams go a step further and make the checks a gate. If an output fails, it doesn't ship. For content, that can mean the build stops until the article is fixed.

Pass or fail beats a score

Code checks also push a team toward binary results. A check either passes or fails, and it says why.

Husain argues for this even when a model is the grader: a pass or fail decision with a short written reason is easier to act on than a 1 to 5 score, because nobody can say what separates a 3 from a 4. (AI Evals FAQ) Not everyone agrees. Anthropic's own documentation shows 1 to 5 scales for things like tone, and a 2026 study found model and human ratings agreed most on a 0 to 5 scale. (Anthropic docs, Develop tests; arXiv 2601.03444) A middle path is to keep a scale only where each point is clearly defined, and use pass or fail everywhere else.

Where code stops

A rule can tell you a paragraph is too long. It can't tell you the paragraph is wrong.

Whether an answer is accurate, whether the tone fits the brand, whether a summary leaves out the one thing that mattered: these need judgment. That is the job of a model judge, checked against human labels, or of a person.

Code can still help at that edge. It can't confirm a fact, but it can record when the fact was last checked and flag it when it gets old, so a person knows where to look.

The order that comes out of this

  1. Code for anything a rule can detect: format, length, required fields, numbers that must match, banned words, links, dates.
  2. A model judge for questions of judgment, each judge with one clear yes or no question and checked against human labels.
  3. People for the cases that need the real world or carry real risk, guided by what the first two layers flag.

The first layer is the cheapest to build and run, and it often catches most of the problems. It is also the one teams skip, because a rule about JSON fields looks less impressive than an AI grader. If you are setting up evals for a new AI feature, start by collecting the last few dozen bad outputs and writing a rule for every failure a rule can catch.

let's build something. ✦

Happy to talk about products, AI and the calls behind them.

Say hello