Before production: where the first 50 eval cases come from
- Agentic AI
- Production
Evaluation always opens as an argument about metrics. The first obstacle is not the metric: there is nothing to measure.
We tried writing scenarios from scratch. Slow and misleading both, because whoever writes them is guessing where the system will struggle. The guesses do not match reality. The system sailed through most of the cases we invented, then got stuck somewhere in production that had never occurred to us.
The examples were already there
On an automation platform the raw material for an eval set is sitting right there: closed incident records. Each one is an input with a verified outcome.
The most useful were incidents a person had intervened in. The agent produced a proposal and a human edited it instead of approving it. The correct answer is known and the point where the agent went off is on record - you will not find a better test case than that.
The second source was reopened incidents. Closed automatically, reopened shortly after. The agent said "resolved" and it was not. Where else are you going to find silent failures.
To those we added the rare-but-expensive ones: incidents that vanish into production averages because they are infrequent, and hurt when handled wrongly.
And we gave half the set to boring, typical incidents. That part we learned the hard way. A set built only from hard cases never shows you the regressions that happen on easy ones.
Fifty is enough
Expectations about set size always run high. Fifty well-chosen cases do more work than two hundred random ones.
The logic is simple. The bigger the set, the more expensive it is to run; the more expensive, the less often it runs; the less often it runs, the less it is worth. Fifty cases can run after every prompt edit - that is where the value actually is.
Instead of growing the set we fixed its distribution. Roughly ten hard cases, ten rare ones, thirty ordinary ones.
What we can measure without a model
Agent output is structured rather than free text, which makes most of the evaluation deterministic: was the right tool chosen, were the parameters right, did the chain stop at the right step.
Those three together cover most cases. Text-quality evaluation is only needed on summaries that reach a person, and even there, comparing against a human label gave more trustworthy results than putting a model in the judge's chair.
A frozen set stops working
A static eval set loses its value over time, because the system quietly starts shaping itself around it. Prompts get tuned until those fifty pass and behaviour outside the set stays invisible.
One rule prevented that: every new failure type that reaches a person in production goes into the set, and one of the oldest cases comes out. Size stays fixed, contents rotate.
What it caught most
We assumed evaluation was something you run when the model version changes. Where it caught the most was prompt edits.
A small improvement to a prompt can fix the case you aimed at and break a different one. That regression surfaces in production days later, usually when someone complains. In the eval set it surfaces in two minutes.
Tool definition changes belong in the same bracket. Clarifying one description can make the model start reaching for that tool in places you never expected.
The hard part of building an eval set is not statistics, it is archaeology. The system has already made its mistakes and they are sitting somewhere.