Scoring an agent on its final answer alone hides how it got there.
Two runs reaching the same result can differ by twenty tool calls and a corrupted file.
So evaluation covers three things: did it succeed, what did it cost, and what did it touch.
Trajectory review compares the steps taken against the steps the task actually needed.
The results are noisy, so a single run proves nothing and several runs per task are the minimum.
Evaluating a single model call is comparatively easy: one input, one output, one score.
An agent takes a variable number of steps, calls tools, and changes state as it goes.
Scoring only the final answer throws away almost everything that distinguishes a good run from a lucky one.
A useful evaluation covers outcome, efficiency and safety together.
Outcome is the end state rather than the text: was the ticket actually filed, do the tests pass.
Defining that checkable end state per task is most of the work in building an agent evaluation.
Efficiency is steps, tokens, wall-clock time and money, and it varies enormously between runs.
An agent that succeeds in forty calls where three would do is a production problem, not a success.
Safety is what it touched: files written, records changed, requests sent, permissions used.
Trajectory evaluation compares the actual sequence of steps against a reference or a rubric.
It is how you catch an agent that reached the right answer through an unacceptable route.
Agent behaviour is stochastic, so one run tells you almost nothing about reliability.
Running each task several times and reporting a success rate is the minimum honest treatment.
Failures should be classified rather than counted: wrong tool, bad arguments, gave up, or looped.
That classification is what turns an evaluation number into a list of things to fix.
One question on this card. Every option explains itself once you pick, right or wrong.
17 concepts in this module, then a graded test with every answer explained.