Evaluating LLM Outputs: Building an Eval Harness That Actually Catches Regressions

Traditional software has unit tests because a function either returns the right value or it does not. LLM output does not work that way, which is exactly why so many teams skip testing it altogether and rely on a developer eyeballing a handful of responses before every deploy. That approach catches obvious breakage and misses everything subtle — which is most regressions.

The three layers of a real eval harness

• Golden set regression — a fixed set of real inputs with known-good outputs or acceptance criteria, run on every prompt or model change, diffed automatically.
• LLM-as-judge scoring — a second model scores each output against a rubric (correctness, groundedness, tone, format) for cases too varied for exact-match testing.
• Production sampling — a rolling sample of real traffic reviewed on a cadence, because your golden set will drift from what users actually ask within weeks.

The golden set is the one teams skip because it is tedious to build. It is also the one that actually catches regressions before a customer does — a prompt tweak that improves nine cases and quietly breaks a tenth will slip past any amount of manual spot-checking but shows up immediately as a red row in a regression diff.

Make the rubric concrete

"Is this a good answer?" is not a rubric an LLM judge can score consistently. "Does the answer cite a source for every factual claim, avoid recommending a discontinued product, and stay under 150 words?" is. The specificity that feels tedious to write is exactly what makes the judge model's scores reproducible enough to trust in a CI pipeline.

Wire it into the deploy pipeline, not a side script

An eval suite that a developer has to remember to run gets skipped the week of a deadline — which is exactly the week a regression is most likely to ship. We wire the golden-set regression into CI the same way a unit test suite runs, with a required pass before a prompt change can merge. It slows down the fun part of prompt iteration slightly. It has caught more production incidents for us than any other single practice in this list.