Here is a workflow that runs in a great many teams, including good ones.
Someone changes the prompt. They run it against the handful of examples they always use to check things by hand. It reads better. It ships.
Nothing about that is lazy — it is fast, it feels responsive, and the person doing it is genuinely paying attention. It is also not evaluation, and the difference matters more than it first appears.
Why a handful of examples cannot tell you anything
The examples you check by hand are not a random sample. They are the ones you already had open, which means they are the ones that motivated the change in the first place. Of course the new version does better on them. That is what you were optimising against.
The deeper problem is that reading output is not a stable measurement. Read the same two answers in a different order, or after lunch, or knowing which one is the new version, and you will not always reach the same verdict. There is no way to re-run a judgement you made by eye three weeks ago, which means there is no way to know whether anything has drifted since.
So the failure mode is not that people ship bad prompts. It is that a prompt change improves the case you were looking at and quietly breaks two cases you were not, and nobody finds out until a user says something. By then the change is three weeks old and tangled up with four other changes, and the cost of working out which one did it is far higher than the cost of having measured in the first place.
The smallest thing that counts as evaluation
An evaluation set does not have to be sophisticated to be an enormous improvement over nothing. What it has to be is fixed, automatic and repeatable.
Twenty to fifty input-and-expected-output pairs is genuinely enough to start. They should be real inputs where you can get them, and realistic ones where you cannot — including the awkward cases, since a set made only of well-behaved examples measures nothing you were worried about. Run every candidate prompt against the whole set, not against the subset that inspired the change.
Then score it the same way every time. This is the part where most explanations go vague, so it is worth being concrete about the two cases:
Structured output is the easy one. If the model is returning JSON fields, a classification, a yes or no, or an extracted value, you can compare directly. Exact match, field by field. There is nothing to interpret.
Open-ended text is the case people give up on, and they should not. You will not get an exact match against a reference answer, but you can score how close the meaning is, or whether the claims in the answer are actually supported by the material the model was given. That second one — a grounding score — is usually the more useful measurement anyway, because it catches the failure you actually care about rather than penalising a correct answer for using different words.
Neither of these has to be a perfect judge of quality. It has to be a consistent judge, so that the number moving means something changed.
The part that pays for itself later
The reason to build an evaluation set is not the change you are making today. You could probably have made that call by hand.
The reason is that the set outlives the change. Six weeks from now, someone will make a small adjustment to the prompt to fix an unrelated complaint, and there will be no reason for anyone to suspect it affects the behaviour you carefully tuned today. With a set, that is a number that moves and a conversation that happens before release. Without one, it is a regression that ships and gets discovered by whoever it inconveniences.
This is the same argument as having tests, which is why the teams that build evaluation sets tend to be the teams that already write tests, and the teams that do not tend to describe both as things they will get to later.
It is also the argument for keeping the set when you change models, which is the single change most likely to move behaviour in ways nobody predicted. A model swap without a fixed set to re-run is a leap of faith dressed as an upgrade.
What it looks like when it is not hypothetical
In practice this is far less impressive than it sounds. It is a file — a spreadsheet or some JSON — with a column for the input, a column for the reference answer, and a column for each prompt version’s output and score. Twenty rows. Something that runs it and prints two numbers.
The moment you can point at a number instead of an impression, you have changed the conversation from “I think this is better” to “this is better on eighteen of twenty cases and worse on two, and here are the two”. That second sentence is what an engineering decision sounds like.
It is also, not incidentally, the exact answer to one of the more effective questions in an AI engineering interview — because the people asking it are not testing whether you know what an evaluation set is. They are testing whether you have been burned by not having one.