Evals Are the Unit Tests of the AI Era

If you’ve written normal software, you know the feeling a test suite gives you: permission to change things. You refactor, you run the tests, they go green, you ship. The tests are a net under the trapeze. Build with AI for the first time and you’ll notice that net is gone — and the vertigo is real. You tweak a prompt to fix one bad case, and you have no idea what else you just broke. That missing net is what an eval is for.

Start with what an eval actually is, because the word gets thrown around loosely. An eval is a set of example inputs, paired with a way of judging whether the model’s output on each one is good. That’s it. It’s a test — but a test built for a component that doesn’t behave like code. A function returns the same answer every time; a language model returns a distribution of answers, some great, some wrong, and the same input can land differently on two runs. So an eval can’t check output == expected the way a unit test does. It has to ask a fuzzier question: is this output good enough, often enough?

That fuzziness is why the grader is the whole game. For each example, something has to decide pass or fail, and you’ve got a ladder of options. At the bottom, exact-match and keyword checks — cheap, fast, brittle. In the middle, code-based rules: did the JSON parse, is the number in range, did it call the right tool. At the top, the increasingly common move of using another model as the judge — an “LLM-as-judge” — to score things like “is this answer helpful and grounded in the source.” That last one is powerful and genuinely fraught: you’re now trusting one fallible model to grade another, and the judge has its own biases (it tends to like longer answers, its own phrasing, the first option you show it). A judge you never audit is just a second thing that can be confidently wrong.

Now the part that matters for how you ship. Without evals, you’re steering an AI product by vibes. You try a prompt, eyeball a handful of outputs, decide it “feels better,” and ship. The problem is that a language model’s behavior is a vast surface, and your gut samples three points on it. You fix the one case a user complained about and silently regress five you weren’t looking at — the classic whack-a-mole where every fix spawns a new failure just out of view. With no eval, you don’t even find out until someone else does. That’s the literal meaning of flying blind: you’re making changes to a system whose behavior you can’t observe at the scale it operates.

Evals change the shape of the work. Building with AI stops being “write the perfect prompt” and becomes something closer to experimental science. You assemble a dataset of cases that matter — especially the ugly edge cases and the exact failures users hit — and you make a number that says how well the system handles them. Then every change gets an honest verdict: did this go up? You can compare two models on your actual task instead of a leaderboard. You can catch a regression before it ships instead of after. You can hand a teammate a prompt change and a green eval instead of a shrug. The single most useful habit is turning every real-world failure into a new test case, so your system can fail that way exactly once.

The honest caveat: evals are not free, and a bad eval is worse than none, because it hands you false confidence. Your dataset can be unrepresentative. Your judge can be miscalibrated. A green score on the wrong cases just means you’re now confidently blind. So evals aren’t a one-time setup — they’re a thing you maintain, distrust, and keep sharpening against reality, the same way you’d distrust a test suite that never fails.

But the core trade is the same one that made unit tests non-negotiable in normal software. You spend real effort building the net, and in return you get the thing that effort buys: the ability to change your system quickly and know whether you made it better. In a field moving as fast as this one, that isn’t overhead. It’s the only thing that lets you move fast without eventually moving off a cliff.

The science, to look up: the machine-learning practice of train/validation/test splits and held-out evaluation; “LLM-as-a-judge” and its known biases (position bias, verbosity bias, self-preference); benchmark contamination and overfitting to a test set (a version of Goodhart’s Law — when a measure becomes a target, it stops being a good measure).