Write the eval before the feature

There is an old discipline in software engineering that says: write the test first, watch it fail, then write the code that makes it pass. Reasonable people disagree about how universally to apply it. But in AI engineering, the argument is over before it starts, because there is no other way to know if you are done.

Consider what "done" means for a feature that summarizes support tickets. Done cannot mean "the code runs," because the code always runs. It cannot mean "the summaries look good," because look good to whom, on which tickets, compared to what? The only workable definition is something like: on a representative set of 100 real tickets, the summary captures the customer's actual issue in at least 95 cases, never invents facts, and stays under 80 words. That sentence is an eval set. Writing it down is the spec work. Automating it is an afternoon.

Here is what changes once it exists.

Prompt changes become measurable. LLM development involves dozens of micro-decisions: system prompt phrasing, output format, temperature, which model, how much context. Without an eval, each decision is a debate. With one, it is a number. We have watched hour-long prompt discussions collapse into a two-minute run more times than we can count.

Regressions become visible. Models get swapped, providers update behavior, someone "improves" the prompt on a Friday. Without evals these changes land silently and quality drifts until a customer complains. With evals in CI, the drift shows up in a failing check, the same way a broken unit test does.

Scope conversations become honest. When the score plateaus at 82% and the remaining failures are all tickets written in fragmented English with three interleaved issues, you can look at the actual cases and decide: is this worth two more weeks, or do we route these to a human? That is a product decision made on evidence. Without the eval, the same conversation is two people trading anecdotes.

The objection is always time. Building an eval set feels like a detour when the feature seems two days from working. But the feature is not two days from working; it is two days from demoing, which is a different thing entirely, and the weeks that follow will be spent debugging quality by anecdote. The eval set is not overhead on the work. It is the work, front-loaded, in its cheapest form.

A practical starting shape, for teams that want one: collect 50 to 100 real inputs, not synthetic ones. Write expected outcomes or scoring criteria for each. Score with exact checks where possible and a judge model where necessary, and spot-check the judge. Run it on every change. Grow it every time production surprises you.

None of this is novel. It is test-driven development wearing a new jacket. The teams that resist it re-learn, expensively, why the old discipline existed.