AI Product Judgment
The PM Owns the Error Log
Ask a product manager "how do you think about evals?" and the answer is often a tool name. Or a metric: "we track helpfulness and groundedness." Neither answer is wrong, but neither says much about the product.
The people who teach evals point somewhere duller. The most useful eval work a PM does is deciding what counts as a failure, and making sure each real failure turns into a test. That work lives in an error log, and someone has to own it.
The weak answer
A tool name tells you where scores are stored. A generic metric describes a quality bar the team never defined. Husain and Shankar's FAQ calls metrics like "helpfulness" and "coherence" "generally not useful" as quality measures for most LLM apps, though they can help pick out traces worth reading. (AI Evals FAQ)
A stronger answer starts with data: we read real conversations, wrote down what went wrong, and built tests for the failures that mattered. That is error analysis.
Error analysis in plain words
Husain and Shankar call error analysis "the most important activity in evals." (AI Evals FAQ) The loop:
- Collect real traces. A trace is the full record of one interaction.
- Open coding. For each trace, write a short free-text note on what went wrong. No categories yet.
- Axial coding. Group the notes into categories. That list is your failure taxonomy.
- Stop at saturation. Keep reading until new traces stop producing new categories.
The FAQ suggests annotating at least 30 traces yourself before letting an LLM propose categories, from a pool of about 100 varied traces. (AI Evals FAQ)
Picture a support chatbot for a bank. A rubric written on day one might say "accurate, polite, helpful." After a hundred conversations, the notes might say something else: it quotes the savings rate from an old page, it answers a card dispute when it should hand off to a person, it asks for the account number twice. None of those is "helpfulness." Each one is something an engineer can fix and a test can check.
One person decides what fails
Husain and Shankar recommend a "benevolent dictator": one domain expert who makes the final quality calls. Add annotators only when needed, and measure their agreement. Don't outsource the core error analysis. (AI Evals FAQ; Lenny's Podcast) The expert has to be the person whose judgment matters, such as a psychologist for a mental health app. Developers and managers are poor stand-ins. (Husain, LLM judge guide)
For the bank bot, that might be a senior support lead. The PM may not be that person. The PM's job is to name them, give them time, and make reading easy. As Husain puts it, "You must remove all friction from the process of looking at data." (Your AI Product Needs Evals)
Why you can't write every eval up front
Shankar's research names the problem criteria drift. People need criteria to grade outputs, but grading outputs is how they figure out the criteria. (Shankar et al., Who Validates the Validators?) The bank bot's missed handoff probably wasn't on anyone's list until someone read the transcript.
Experts disagree on what follows. Eugene Yan argues for eval-driven development: "First, write some evals; then, build systems that pass those evals." (Eugene Yan) Husain and Shankar call that approach "generally not recommended" and say to write evaluators for failures you have actually observed. (AI Evals FAQ)
I think both are right at different moments. Before launch, you can set success criteria. Anthropic suggests making them specific, such as "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity" in place of "safe outputs." (Anthropic docs, Develop tests) The detailed evaluators come after you read what the product actually does.
Are evals the new PRD?
Braintrust argues the eval is now the spec, the acceptance criteria, the roadmap and the launch review in one, and quotes OpenAI's Kevin Weil: "Writing evals is the most important thing a PM can do in the AI era." (Braintrust) Braintrust sells eval tooling. Husain and Shankar make a softer claim: well-written judge prompts act as living product requirements. (Lenny's Podcast)
The part that holds up: when output changes from run to run, a set of pass and fail examples says more about quality than a paragraph in a doc. The part that reads as vendor framing is replacement. An eval set doesn't explain who the user is, why this problem, or what the team chose not to build.
Anti-patterns
- Generic helpfulness scores. They score a quality nobody defined. (AI Evals FAQ)
- Dashboards of 1 to 5 averages. A 3.4 doesn't say what to fix. Husain: "If someone says you need to measure 8 things on a 1-5 scale, they don't know what they are looking for." (LLM judge guide) Anthropic's docs do use 1 to 5 scales for tone, so the fair middle is to keep a scale only where every point has a written definition. (Anthropic docs)
- Buying tooling before a review habit exists. Yan: "buying or building yet another evaluation tool won't save the product." (Eugene Yan)
- Letting an LLM do the first read. "Use LLMs to scale what you've learned, not to avoid looking at data." (AI Evals FAQ)
A failure list the product writes for you
One cheap source of failures is a log of every question the product could not answer. Each entry arrives already labeled: someone asked, the product had nothing. Reading it is open coding with half the work done. A chatbot I built logs these, and each one becomes a candidate fix.
The log has a blind spot. It only catches misses the product noticed, and a confident wrong answer never shows up there. That is why the FAQ suggests mixing sampling methods (random, outliers, user feedback) and always keeping some random traces. (AI Evals FAQ)
The PM's job list
- Own the failure definitions. Write down in plain words what counts as a fail, with real examples, and update it as the log grows.
- Label the data or appoint the labeler. Read traces yourself, or name the domain expert who will and protect their time.
- Feed each fixed miss back in as a regression case. The FAQ recommends adding representative production failures to the CI set. (AI Evals FAQ) Anthropic expects regression evals to pass close to 100% of the time, so a fixed bug that comes back gets caught. (Anthropic, Demystifying evals)
For a better answer to the interview question, start this week: pull 30 real conversations from your product, write one line on what went wrong in each, group the lines, and decide who gets the final say on pass or fail.