AI Product Judgment
Evaluating Agents: Outcome First, Then the Path
A single-turn AI feature gives you one output to check. An agent gives you a final answer plus everything it did to get there: which tools it called, what it passed them, what came back and what it decided next. When teams start evaluating agents, they often keep the single-turn habit and read only the last message. If it reads well, the test passes.
That habit breaks down fast. The last message is the agent's own report of what happened, and the agent can be wrong about that.
Why "the output looks fine" is a trap
Anthropic's guide to agent evals separates the transcript, the full record of a run, from the outcome, the final state of the world after the run. Its example is an agent that says "Your flight has been booked." What counts is whether the reservation exists in the database. (Anthropic, Demystifying evals for AI agents)
Picture a travel-booking agent that searches flights, picks one within the user's budget, books it and emails a confirmation. Any of those steps can go wrong while the final message still sounds right. The search can use the wrong date. The booking call can time out while the agent carries on. Read only the last message and you see a confident confirmation every time.
Grade the outcome or the path?
This is where experts split.
Anthropic's advice is to grade what the agent produced, not the path it took. Agents often find valid routes the eval author didn't expect, and a check that demands a fixed order of steps punishes them for it. The guide still keeps a few checks on the transcript, such as turn limits and verifying tool calls. (Anthropic)
Tooling vendors lean the other way. LangSmith's AgentEvals can compare an agent's tool-call sequence against a reference trajectory, or judge only the final output. (LangChain docs, Evals) Arize argues that a correct final answer can still hide a costly or risky path. (Arize, Agent trajectory evaluations) Both sell eval tooling, but the point holds. If the travel agent books the right flight after hitting the payment step twice, an outcome grader may be happy and the customer may not.
Hamel Husain and Shreya Shankar offer a middle path in two phases. First, judge end-to-end success as a black box. Then, for the runs that failed, look at the steps: tool choice, parameter extraction, error handling and context retention. (AI Evals FAQ) The outcome tells you whether there is a problem. The steps tell you where it is.
Find the first upstream failure
In the second phase, Husain and Shankar suggest recording the first upstream failure in each broken run, because later failures often cascade from it. (AI Evals FAQ)
If the search step pulled the wrong date, the booking, the price and the email are all wrong too. Count only the first broken step and it's clear the search needs fixing, not the email. The FAQ also describes a transition failure matrix: for each failed run, note the last step that worked and the first step that didn't, then look for the cells with the most runs. Those are your hotspots.
Check tool calls in pieces
A tool call can fail in more than one way, so the FAQ suggests checking it on four things separately: the tool name, the arguments, the result that came back and the state it left behind. (AI Evals FAQ)
The booking agent can call the right tool with a return date before the departure date. It can get an error and ignore it. Each needs a different fix, and most can be caught with plain code. OpenAI's guide makes the same point about architecture: once you move from a fixed workflow to an agent, tool selection and argument precision become things you have to evaluate. (OpenAI, Evaluation best practices)
Most of the time is not good enough
Agents don't behave the same way every run. Anthropic describes two ways to count repeated runs. pass@k is the chance that at least one of k tries succeeds, and it goes up as k grows. pass^k is the chance that all k tries succeed, and it goes down. At a 75% success rate per try, pass^3 is about 42%. (Anthropic)
pass@k is fine when one success is enough. A customer-facing agent is different: each customer gets one run, and the same request has to work every time. pass^k comes from Sierra's τ-bench, where GPT-4o-class agents succeeded on under 50% of tasks and pass^8 fell below 25% in the retail domain. (Yao et al., τ-bench; Sierra, Benchmarking AI agents)
Start with 20 to 50 real failures
Anthropic says "20-50 simple tasks drawn from real failures is a great start," and recommends including cases where a behavior should happen and cases where it shouldn't. (Anthropic) For the booking agent, that means trips it should book and trips where it should stop and ask, such as when nothing fits the budget.
Read the transcripts
Anthropic's guide says it plainly: "Read the transcripts!" That is how you tell a real agent mistake from a grader rejecting a valid answer. (Anthropic) Husain adds a warning about clean scores: "If you're passing 100% of your evals, you're likely not challenging your system enough." (AI Evals FAQ)
Know which engine answered
Many agents have a fallback. If the model call fails, a simpler path answers. If the fallback answers every request, the outputs can still pass, and the eval ends up scoring the fallback while the model sits broken. A fallback that answers silently looks exactly like a model that works.
The fix is tracing: log the full path of each response so you can split results by stage. Husain's guidance is to log the whole workflow, segment error analysis by step, and track how often a system hands work to a person or a backup. (AI Evals FAQ; Hamel Husain, Your AI Product Needs Evals) I wrote about a test of mine that passed for exactly this reason in When the Roadmap Meets an LLM.
The order that comes out of this
- Grade the outcome against the real end state, not the agent's summary of it.
- Diagnose the steps on failed runs and record the first upstream failure.
- Check tool calls on name, arguments, result and resulting state, in code where you can.
- Repeat each task several times and report pass^k for anything a customer touches.
- Tag every response with the path or engine that produced it.
If you are setting up agent evals this week, start with the last item. Add one field to every response that says what produced it, then pull 20 of your worst runs and read them end to end. You will likely find the first upstream failure before you write a single grader.