Agent Evals Are the New Unit Tests — And Yours Are Flaky

Your eval suite is not measuring your agent. Most days it is measuring the weather.

I have a folder in one of my projects called evals/. For about two months it was the most reassuring directory in the repo. Green checkmarks, a score in the high eighties, a little table printed at the end of every run. I shipped against that number. I quoted that number in a meeting.

Then I ran the suite three times in a row on the exact same commit and got 84, 79, and 88.

Nothing changed. Not the prompt, not the code, not the model version. The only thing that moved was the sampler, the wall clock, and my confidence.

That is the moment I started treating evals as a piece of engineering rather than a report card. Because an eval suite that swings nine points on an unchanged commit is not a quality signal. It is a random number generator with a nice CLI.

Why we ended up here

Unit tests earned their authority through a boring property: determinism. Same input, same code, same output, every time. Once a test is deterministic, a failure carries information. Something you changed broke something you did not intend to break. That single property is what makes a red build worth stopping for.

Agents violate that property at every layer. Sampling is stochastic. Tool calls hit real systems with real latency and real rate limits. Retrieval depends on an index that drifts. The model behind the API is a moving target, and unless you pin a snapshot, it can change under you on a Tuesday with a blog post as the only changelog.

So we did the natural thing and wrote tests anyway, then quietly learned to ignore them. Every team I have talked to about this has some version of the same ritual: run the evals, see a couple of failures, shrug, re-run, ship. We rebuilt flaky tests, except this time we decided the flakiness was philosophically inevitable.

It mostly is not. A large share of eval noise comes from things we control.

The four sources of noise, roughly in order of how much they cost me

Sampling. The obvious one, and the one people fix first by setting temperature to zero. Worth doing, but it buys less than you think: with tool calls, batching, and non-deterministic kernels on the provider side, temperature zero is "less random," not "deterministic."

The judge. If an LLM grades your output, you now have two stochastic systems in series and only one of them is the thing you are testing. I have watched a judge flip its verdict on identical text because the previous item in the batch was a failure. The grader had picked up a mood.

The environment. Live APIs, a database with today's data in it, a vector index that got re-embedded last week. Half of my "regressions" were the fixture drifting, not the agent degrading.

The task itself. Some tasks genuinely have many correct answers, and any single-string assertion against them is a coin flip dressed up as a test.

Only the last one is a real property of agents. The other three are ordinary engineering problems that we let slide because the word "AI" made them feel exotic.

Measure the noise floor before you measure anything else

This is the one habit that changed how I work. Before comparing prompt A to prompt B, I run the same prompt against itself.

// Five runs, same commit, same prompt. This is the baseline you compare against.
const runs = await Promise.all(Array.from({ length: 5 }, () => runSuite({ commit: 'HEAD' })));

const scores = runs.map(r => r.score);
const mean = scores.reduce((a, b) => a + b) / scores.length;
const spread = Math.max(...scores) - Math.min(...scores);

console.log(`mean ${mean.toFixed(1)} · spread ${spread.toFixed(1)}`);

If the spread is nine points, then your new prompt scoring four points higher means nothing at all. You have not improved the agent; you have observed the suite. I have thrown away two full afternoons of "optimization" that lived entirely inside the noise band, and I only found out because I finally ran the baseline.

Publish the spread next to the score. Always. A score without an error bar is a rumor.

Separate the two things you are actually testing

Once I stopped treating the suite as one number, it split cleanly into two kinds of check that deserve completely different treatment.

Contract checks are deterministic and belong in CI as blocking. Did it emit valid JSON against the schema? Did it call the tool with the right argument types? Did it refuse the out-of-scope request? Did it stay under the token budget? Did it stop instead of looping? None of these need a judge, and none of them should ever be allowed to be flaky. If a contract check fails intermittently, that is a bug in your agent, not noise in your eval.

Quality checks are statistical and do not belong in a blocking gate at all. Was the summary good? Was the plan sensible? These need many samples, a stable rubric, and a trend line. Treat them like performance benchmarks: track them, alert on sustained movement, never fail a PR because one run of one rubric dipped.

Most broken eval suites I have seen are broken because these two got mixed into a single average. A hard schema violation and a slightly weaker paragraph get folded into the same percentage, and now you cannot act on either.

Make the environment boring

For contract checks, record the world and replay it. Every tool call, every retrieval result, every API response gets captured once into a fixture and replayed on subsequent runs. This is not novel; it is VCR cassettes from the Rails era, applied to agents.

const tools =
  process.env.EVAL_MODE === 'record'
    ? withRecorder(liveTools, { out: 'fixtures/' })
    : replayFrom('fixtures/');

The pushback I get is always the same: "then you are not testing against reality." Correct. That is the point. Contract checks test your agent against a frozen world so failures are attributable. You test against reality separately, on a schedule, in a suite that is allowed to be noisy because nobody is blocked by it.

Pin the model snapshot too, and treat a version bump as a code change with its own PR and its own baseline re-run. A model upgrade that silently lands mid-week will make three weeks of your trend line meaningless.

Fix the judge or drop it

If you keep an LLM judge, it needs the same discipline you would apply to a human grader you did not fully trust:

• Grade one item per call. No batches, no shared context, no mood carried over from item three.
• Give it a rubric with explicit failure conditions, not a vibe scale from one to ten.
• Ask for the reason before the verdict, so the verdict is conditioned on something.
• Calibrate against a set of examples you graded by hand. If the judge disagrees with you on those, it is not going to be right on the ones you did not check.
• Track judge-vs-human agreement over time. When it drifts, the eval is broken even if the agent is fine.

And be honest about when a judge is unnecessary. A surprising number of things I once graded with a model can be checked with a regex, a schema, or an exact match against a known answer. Those checks are free, instant, and never have a bad day.

What I actually run now

A short, brutal contract suite on every commit, fully replayed, zero tolerance for flakiness. If it goes red, something is genuinely broken and I stop.

A larger quality suite nightly against live systems, five samples per task, reported as mean with spread, plotted over time. Nobody is blocked by it. It exists to show me slopes, not verdicts.

And a habit: every production failure becomes a contract check the same day, with the real inputs baked in as a fixture. That is the part that compounds. The suite gets sharper because reality keeps donating test cases.

Conclusion

The comparison to unit tests is right, but not in the flattering way people usually mean it. Evals are like unit tests in that they are the difference between shipping with confidence and shipping with hope — and also in that a flaky suite is worse than no suite, because it teaches the team to ignore red.

The good news is that most eval flakiness is not a deep property of language models. It is unpinned versions, live fixtures, a moody grader, and a single average smearing together things that should never have been averaged. Those are all fixable with ordinary engineering.

Measure your noise floor. Split contracts from quality. Freeze the world for the checks that must be deterministic. Then, and only then, start believing your numbers.

Diego Vallejo, August 2026