Testing non-deterministic systems: a 26-conversation eval suite

The problem

You can't test a language model with a string assert. Ask the same thing twice and the wording changes, the suite goes red over nothing, and sooner or later someone switches it off. But wording that varies isn't the same as anything goes. The agent can say a price a hundred different ways, as long as the price exists in the catalog. It can confirm a slot in its own words, as long as the slot was actually booked. That line is what I turned into a test.

Constraints

• 01Deterministic enough to block a deploy. A suite that fails one run in five is worse than no suite: the team learns to ignore the red.
• 02Fast. It runs on every change that touches the agent, so half an hour of runtime means nobody runs it.
• 03Assert on behaviour, not on the sentence. "Did it quote R$ 54.90?" breaks for nothing; "did it quote a price that exists in the catalog?" is what I need to hold.
• 04Real agent, real tools. Mocking the model tests the mock. The suite drives the same runner that answers WhatsApp.

Architecture

The checks are the part that matters. provaSupervisor kills any price, total or product that isn't in the catalog. provaNomeInterno fails if the agent admits to being an "AI" instead of staying the specialist it is supposed to be. provaDiaDaSemana checks that the weekday it wrote actually matches the date. And provaPedido decodes the Pix code it sent — Pix being Brazil's instant-payment system — parsing the EMV payload, recomputing the CRC16 and checking the amount. If that code comes out wrong, the customer finds out at the bank, not me.

The hard part

The checks were easy to write, and I trusted them too much. I built a supervisor that flagged as a lie any number on screen the agent hadn't just fetched from a tool. First full run: 126 accusations. It looked thorough. It was almost all wrong — the agent had fetched those numbers, just not in the shape the guard expected.

Before letting that guard block a deploy, I measured it against the traces, looking at what the agent actually did that turn, and tuned it until the accusations matched reality. A guard is a hypothesis until someone measures it. And one that cries wolf 126 times gets switched off in the first week.

Trade-offs

• Scripted, not generative. I write the customer turns myself, so coverage only goes as far as my imagination. I chose depth on the failure modes that cost money.
• Behavioural asserts miss tone. The suite proves the agent didn't lie about a price; it doesn't prove it was warm. That one I read in the transcript.
• Five minutes is a deploy-gate budget, not a pre-commit one. It runs in CI, not on every keystroke.

What I'd do differently

There's one thing I'd invert. Instead of writing the conversation and asserting invariants on top of it, I'd let the model generate adversarial customers against a fixed set of invariants: a customer whose only goal is to get the agent to quote a price that doesn't exist. The script proves the bugs I imagined are gone; a generated adversary would find the ones I didn't. I know the shape of that harness now. I didn't when I started, and shipping the scripted version first was the right call.