Every stalled pilot we have been asked to rescue had a working demo and no scoreboard. Without one, “is this better?” is a matter of opinion, and opinion does not get past a compliance review.
The harness is not overhead you add at the end. It is the first artefact, and it is what makes every later decision cheap.
What is a golden set?
A golden set is a fixed collection of real cases with the correct outcome attached, labelled by the people who do the job today. Fifty cases beats five hundred synthetic ones, because the distribution is real — including the awkward tail that breaks agents in production.
Build it from your own logs. Sample across the whole distribution, not just the clean middle, and keep every case that a human found genuinely hard.
Why write it before the agent?
Writing the harness first forces you to define what “correct” means while nobody is invested in a particular implementation. Once an agent exists, the definition of success quietly bends toward what that agent already does, and you lose the ability to compare approaches honestly.
It also gives you a cheap answer to the question that kills momentum in week six: is the new prompt actually better, or does it just look better on the three examples someone pasted into Slack?
What goes in a score?
Score the things that would cause a rollback: task success, citation or evidence accuracy where the answer must be traceable, escalation correctness, and cost per resolved task. A single blended number hides regressions — keep the components visible and gate on each.
For anything customer-facing, add a safety dimension scored by a separate rubric. It is the one that compliance reads.
How do you keep it honest over time?
Freeze the golden set and version it like code. When you add cases — and you should, from every incident — bump the version and re-baseline, rather than quietly editing the set until the numbers improve. Track score by version so a regression is a diff, not a debate.
Run it in CI on every prompt, model, and tool change. A change that drops the score does not merge, the same way a failing test does not merge.
The eval harness is what converts “the model seems better” into a number a compliance reviewer can sign.
How long does this take?
Two weeks for a first harness on a well-understood workflow, most of it spent labelling rather than writing code. That is the same two weeks a team usually spends arguing about whether the demo is good enough — the difference is that at the end of it you have an answer.
What is the exit criterion?
You are done when a scored golden set exists, its baseline is recorded, and CI fails on regression. Only then does agent code start. Every stage after this one has an exit criterion too — you never fund the next one on vibes.