Skip to Content

PART 3 OF 3 – HOW DO YOU REGRESSION TEST A SYSTEM THAT ANSWERS DIFFERENTLY EVERY TIME?

November 16, 2026
Sebastian Keck

Final part of a 3-part series on the shift of the bottleneck from “writing code” to “trusting code” in agentic development.

Assertions assume repeatability, and LLM backed systems do not offer it. A practical field guide to invariants, golden sets, tolerance bands, judge models and where evals belong in the pipeline.

The first time it happened, I spent forty minutes looking for a bug that didn’t exist.

The suite was red, on the same commit and the same input that had passed the run before. Nothing had changed. I re-ran it: it passed. Four more runs, one more failure.

My instinct, trained over years, said flakiness: a bad wait, shared state, test pollution. It was none of those. The system had a component that simply doesn’t return the same thing twice, and my assertion assumed it would.

That someone was me.

Every assertion we write carries a hidden assumption: that the same input produces the same output. Once an LLM sits anywhere in the call path, that assumption is gone, and most of our tooling has no answer for it.

This is the last part of a short series on the verification gap in agentic delivery. Part one argued verification is now the bottleneck. Part two described the failure modes. This part is about the mechanics.

Start by shrinking the problem

Before any technique, one architectural move is worth more than all of them combined.

Most of your system is still deterministic, and should stay that way: the database, the routing, the authorisation check. What’s nondeterministic is usually one component, sometimes one function call.

So push it to a boundary and put an interface in front of it. The deterministic side gets tested as before, with a stub standing in for the model; only the component behind the interface needs the expensive treatment.

I’ve seen teams treat an entire application as nondeterministic because one feature calls a model, multiplying verification cost tenfold for no reason. Nondeterminism is a property of a component, not of a system, unless you let it leak.

Assert on invariants, not on strings

Once you’re inside the nondeterministic boundary, exact matching isn’t available, but everything that must hold true regardless of phrasing still is.

For a typical generation endpoint, that list is longer than people expect: valid JSON, schema conformance, required entities present, numeric fields within plausible bounds, no field resembling personal data, referenced identifiers that actually exist in the source data.

None of this cares about wording, yet together it catches a surprising share of real regressions.

def assert_invariants(response, source_record):
    payload = json.loads(response)              # parses
    validate(payload, SUMMARY_SCHEMA)           # conforms
    assert payload[“customer_id”] == source_record.id
    assert 20 <= len(payload[“summary”].split()) <= 120
    assert not PII_PATTERN.search(payload[“summary”])
    assert set(payload[“referenced_orders”]) <= set(source_record.order_ids)

That last line is my favourite: it asserts the system didn’t invent a reference, four tokens of Python catching a whole class of fabrication. Property-based testing extends this idea further, letting a framework like Hypothesis hunt for inputs that break a stated property instead of enumerating cases by hand.

Golden sets, and the discipline they need

You still need concrete cases: build a golden set, and be deliberate about where they come from.

The best sources are production traffic and past incidents: a case that caused a real problem once is worth fifty invented in a workshop. Sample deliberately for the awkward regions too, empty inputs and inputs adversarial by accident rather than design.

Then treat the set as a product: version it in the repository, give it an owner and a review cadence, and let every incident add a case. A golden set nobody has touched in six months describes a system that no longer exists.

Keep it small enough to run often. Two hundred well-chosen cases running nightly beat four thousand running quarterly that nobody trusts.

Replace pass and fail with a score and a band

This is the change people resist most, and the one that makes everything else work: a single nondeterministic case passing or failing tells you almost nothing, since it will occasionally fail for reasons that aren’t defects. Stop gating on individual cases, and start gating on the aggregate.

Run the set, score it per dimension, and compare against a baseline and a tolerance band: accuracy hasn’t dropped more than two points against the last release, no individual dimension has dropped more than five, and no case in the critical subset has failed at all.

Keep a small critical subset, usually safety, compliance and data boundaries, where the standard stays absolute; that clause matters most. You accept a distribution for quality, never for safety.

Also separate two things that look identical on a dashboard: variance, the system behaving within its nature, and flakiness, your harness being unreliable. Run the same case five times against a fixed version: spread is variance; spread that changes when you change the harness is flakiness.

On using a model as the judge

At some point you need to assess something no schema check can reach: is the summary faithful to the source, is the tone appropriate, is the answer actually responsive to the question.

Using a model as the evaluator works, but the caveats are easy to underestimate: it prefers longer answers, it favours output from its own model family, and its prompt drifts as people tweak it, so your measuring instrument changes while you’re using it to detect change.

Three rules I wouldn’t break: version the judge prompt and model in the same repository as the code, and treat any change to either as invalidating the baseline; calibrate against human labels periodically and publish the agreement rate; and never let the model under test judge itself.

Where this belongs in the pipeline

Cost and latency decide the tiers here. On every commit: the deterministic suite plus invariant checks, fast and cheap, catching most regressions. Nightly: the full golden set scored against the baseline, too slow for the commit path but too valuable to skip. On release candidates: judge-based evaluation on the dimensions that need it, plus the critical subset at full strength. In production: sampling and monitoring, since the real input distribution is the only honest one, an approach my colleagues have pushed further in Building Self-Healing Test Automation with Robot Framework MCP.

What I would tell my past self

Forty minutes lost to a bug that didn’t exist was cheap tuition.

The expensive version is a team that concludes their AI feature can’t be tested and stops trying, or one that keeps exact-match assertions and quietly raises the retry count until the suite goes green. I’ve seen both; the second is worse, because it looks like success.

Testing a nondeterministic system isn’t harder than testing a deterministic one; it’s a different question. We stopped asking whether the output is correct, and started asking whether the behaviour falls within an acceptable range, the way every engineering discipline outside software has handled tolerances for a century. We’re late to it, not ahead of it.

What does your eval gate look like today, and would it catch a two-point quality regression before a customer did?

About the author

Senior Test Automation Engineer | Germany
Sebastian Keck is a versatile Sogeti Germany consultant with broad expertise and a passion for new technologies, specializing in test automation, software testing, development, data engineering, data science, and project management.

Leave a Reply

Your email address will not be published. Required fields are marked *

Slide to submit