Testing After the Shape Changed
This was originally planned as Cycle 2, right after observability. It became Cycle 3 instead, after the multi-agent system. That reordering is the actual subject of this post, more than the suite itself.
Why this waited
A regression suite written against the single-agent shape, right before that shape changed underneath it, would have needed rewriting the moment the second agent landed. Testing a system mid-change protects against very little. So the suite waited for the dispatcher and the reconciliation check to exist first — and once they did, two failure modes appeared that hadn't been there before: the dispatcher can send a turn to the wrong agent, and a coordination bug between two agents' data can silently produce a wrong reconciliation number. Neither trips the approval gate from Cycle 1 — both are reads, not writes — and both now touch real numbers with real stakes: sales figures, not just yield.
There's a word for what this actually is: eval design. Not a general claim about testing philosophy — specifically, deciding what's worth checking now that two failure modes exist that didn't before, and building the smallest thing that checks them.
What twelve scenarios actually cover
The golden dataset is twelve scenarios. Six are new: three for dispatcher correctness — a dairy-only query routes to the dairy tool list, a vendor-only query routes to the vendor list, a reconciliation or mixed query routes to both — and two for reconciliation accuracy, checked against a 5% tolerance: a matched production/delivery window returns flagged: false, a deliberately mismatched one returns flagged: true, both verified by recomputing the digest against ground truth from the same seeded database rather than trusting the number the agent reports back. One more covers the vendor write gate: record_delivery pauses for approval the same way log_milking already did.
The other six are Cycle 1 cases, reused rather than re-derived: a chart read, a guard rejection against a nonexistent animal, the write-gate pause and its approved resume, the iteration cap, and model fallback. They already had a verification record from Cycle 1's own phases — re-testing them here would be duplicate coverage, not new protection. Worth being precise about that split, since "twelve scenarios" on its own overstates how much of this cycle is actually new ground.
What the assertions check, and what they don't
Every assertion is against the tool-call sequence and the finish-reason — which tools got called, with what arguments, in what order, and how the run ended. Not the assistant's exact wording. That's consistent with the rest of this project: the digest, not the raw text, has been the thing that mattered since Cycle 1.
The harness calls runAgentStream() directly — the same function POST /api/agent/run calls in production — so nothing here is a parallel implementation of the loop that could quietly drift out of sync with what's actually running.
Two things I decided against, for the same underlying reason. No LLM-as-judge: nothing here needs a second model scoring the first model's answer, because the questions being asked — did it call the right tool, did the reconciliation number match ground truth — have a mechanically checkable answer. And no hosted eval framework: Promptfoo, Braintrust, DeepEval all solve a prompt-regression-at-deploy-scale problem this project doesn't have yet. The harness is a plain node:test file, run via tsx --test — the same runner the server's existing unit tests already use. Zero new dependencies. Vitest exists in this repo, but only in web-angular; pulling it into the server for this would have meant a second test runner for one feature.
Running it against the real model
The suite runs against the actual Anthropic API, not a mocked one. That's the same standard the observability cycle set for itself — verify against the live app, not a simulation. The honest cost of that choice: real token spend on every run, and a model with genuine non-determinism, so a scenario could in principle need a retry rather than failing outright on a bad day.
In practice, across two full live runs: 12 out of 12, both times, zero flakes. That's a good result against flakiness, not proof it never surfaces — two clean runs is evidence, not a settled question. If it stops being quiet, the fallback is recording one known-good transcript per scenario and replaying its tool-call decisions instead of calling the model live. That's a bigger change than this cycle needed, so it stayed a documented option, not something built preemptively.
Two things worth naming from the build
The iteration-cap and model-fallback scenarios each ended up in their own test file — regression.cap.test.ts, regression.fallback.test.ts — separate from the main regression.test.ts. AGENT_MAX_ITERATIONS and ANTHROPIC_MODEL are both read once, at module load, not per call. Faking either one for a single scenario inside a shared process would leak into every other test running alongside it. Each of those two scenarios gets its own process instead, with the environment variable set before that process starts — the same per-invocation approach Cycle 1 already used to test these two things, just formalized into its own file now that there's a suite to fit it into.
One scenario didn't make it in: "unavailable API key," which Cycle 1 verified produced no trace at all. It's a route-level guard in index.ts — checked before runAgentStream() is ever called — not a behavior of the function this harness actually exercises. It doesn't fit the shape of what this harness tests, so it's left out, and noted as an open item rather than quietly dropped.
What's still open
Whether flakiness stays at zero past two runs — unknown until it's run more. Whether to eventually widen this into the fuller 15–20 scenario suite that was the original, broader plan — deferred, since the dispatcher and reconciliation logic should sit stable for a while first. And there's no path for a contributor without an API key beyond "the CI job skips" — fine for a project with one operator, worth reconsidering the day it isn't.
The suite is opt-in even for a normal local test run — RUN_REGRESSION=1 plus the key — so npm test stays free of token cost by default. CI is gated the same way, on the secret's presence: if it's not set, the step logs that it skipped and the rest of the pipeline stays green, the same convention this project already uses for a missing key elsewhere.
Tagged v0.6.0. Code's in the repo if you want to see the harness or the scenarios directly, same as always.
