The first time someone on the team asks: "How do we test the agent?", the room splits into two wrong answers.
One camp says you can't, it's non-deterministic, we'll watch it carefully in production.
The other camp mocks the LLM call and asserts on the mock, which produces a green test suite that verifies precisely nothing.
We ship agents at work, and we test them. Not the way you test a REST endpoint, since assertEquals dies the moment the same input can produce five valid answers.
But the discipline underneath testing survives contact with AI just fine. It ends up as a pyramid again, with different layers. Deterministic tests for everything around the model, LLM judges for the conversations, and evals in production that never stop running.
Share this post & I’ll send you some rewards for the referrals.
Run Your Auth With AI Agents (Partner)
Coding agents are here to stay, but vibe-coding auth is dangerous business. Connect your AI agents to the Descope MCP server instead!
This remote MCP server gives agents the ability to read documentation, manage users and tenants, configure auth flows, generate FGA schemas, review audit logs, and more…all through natural language.
(Thanks to Descope for partnering on this post.)
First, separate the model from the machine
The realization that unlocks agent testing: an agent is mostly not a model. It’s routing logic, tool implementations, schema validation, retry policies, state management, formatting. All deterministic, ordinary code that happens to sit around a probabilistic core.
Draw the boundary explicitly, then test each side on its own terms.
The machine (everything around the model) gets normal tests.
The model’s behavior (what it says and which tools it picks) gets judged scenarios.
The whole thing in the wild gets continuous evals.
Skip the separation and you end up writing slow, expensive judge tests for things a unit test covers in a millisecond, or skipping tests entirely because “it’s all non-deterministic anyway”. It isn’t. Most of it is boring, and boring is testable.
Layer 1: Deterministic tests for the machine
Tools are functions.
fetch_expense(id)returns the expense or a typed error. Unit test it against a real test database, and fake any external HTTP API at the transport level withrespxorpytest-httpx. No LLM anywhere.Schema validation is deterministic: given this malformed output, the boundary rejects it with this error.
Routing and state logic. If the graph should move from
awaiting_confirmationtoexecutingon approval, that's a reducer test.Failure policies. Retry budgets, fallbacks, timeouts: fake the model's failure, not its intelligence. A stub that throws a timeout is a legitimate mock; a stub that fakes a smart answer is not.
This layer should be the bulk of your suite, and it runs in seconds.
Layer 2: Scenario tests with an LLM judge
Now the genuinely new part. For the model’s actual behavior, we write scenario tests.
Each one is a multi-step conversation, and every step is a small piece of data.
step = {
"user": "I lost my card somewhere in the airport.",
# graded by a judge, because meaning has no regex
"criteria": [
"acknowledges the lost card",
"says the card will be blocked",
"asks at most one clarifying question",
],
# asserted against the mock's call log, exact match
"expected_calls": [], # empty on purpose: blocking a card before asking which one is a fail
"mocks": {"cards_api": "ok", "accounts_api": "ok"},
}Four fields, two kinds of assertion.
The agent runs its full loop against faked external APIs, real model, real prompts, real tool selection. Then the test compares the mock’s call log against expected_calls, a hard deterministic check with one right answer, and sends only the free-text reply to a second LLM call, the judge, which grades it against criteria and returns pass or fail with a reason.
Why a judge instead of string matching? Because “your card ending in 4821 has been blocked” and “I’ve blocked the card for you” are both correct, and no regex captures that space. Meaning is the one assertion a string comparison can’t make.
This suite is pytest as well, and it runs the real agent in-process. The agent picks its own tool calls; only the tool responses are faked, so there’s no queue, database, worker, or mock server to keep alive.
In our repo, each step lives in a YAML file that a Pydantic model validates on load, so a typo like expected_call fails at collection time instead of quietly checking nothing.
Evals are tests, and tests belong in the runner your team already uses.
Writing judges that don't lie to you
The judge is an LLM too, so it can be wrong, and a wrong judge doesn’t turn your suite red. It turns it green on bad replies. The short version of keeping it honest:
Binary criteria, three or four per step, each one checkable by a human in five seconds.
Never ask the judge what code already knows. Which tools ran is an exact answer, and
expected_callscovers it.Count thresholds; never ask for a score. Tally pass/fail across criteria, runs, and scenarios.
Calibrate the judge and gate on it. Known-good and known-bad replies run first, and a drifting judge skips the expensive suite.
Use a different model than the agent, and let calibration decide how big it needs to be.
Note: every one of those hides a trap, so the judge gets its own deep dive post.
Layer 3: Production is the eval that never ends
The uncomfortable truth about layers 1 and 2 is that they ran before the deploy.
The model’s behavior can still drift after it.
A provider-side model update, a data shift, users phrasing things nobody predicted.
So the last layer runs on live traffic, continuously, and it’s wired to alerts rather than a weekly report.
Split it by what each check costs.
Cheap checks on every trace. Schema-valid rate, tool-call errors, steps per run against the loop budget, latency, tokens per conversation. They’re free to compute at boundaries you already have, so run them on all of your traffic and page someone when one moves.
Refusals, on every trace too. A safety refusal comes back as a successful response carrying a refusal stop reason, so it never shows up in your error rate. Count it, or your first hint is a support ticket.
Judges on a sample, alerting on the rate. The same evaluators from layer 2 score a slice of real (redacted) conversations as they land. Judging everything means another model call per criterion on every conversation, and you don’t need all of it to see a trend. Say you handle 10,000 conversations a day: a 5% sample is 500 judged replies, plenty to catch a pass rate sliding from 95% to 85% the same day. Alert when a criterion’s rolling pass rate drops, never on one failed verdict. A single verdict is noise; a rate is signal.
The regression gate. Before any prompt tweak or model swap ships, the full scenario suite reruns against the change. Prompt changes are deploys. They get deploy discipline.
There’s one place where judging every conversation makes sense. It’s a flow where a single bad reply is worth acting on, like anything that moves money, and there the verdict routes that conversation to a human. At that point it’s a guardrail more than an eval.
📌 TL;DR
Most of your agent is normal code: tools, routing, parsing, validation. Normal unit tests, no model call, milliseconds each.
Layer 2 is judged scenarios: real model, mocked APIs, tool calls checked against the mock’s call log, and a judge that grades only the free-text reply.
Judge criteria are binary facts. “States the card will be blocked” is a criterion; “is helpful” is a vibe.
Thresholds are counted, not judged. Tally the binary checks (step → scenario → suite). A judge’s 0-100 rating drifts and never says what broke.
Calibrate the judge, and gate on it. Known-good and known-bad replies with expected labels, run before the scenarios, so a drifting judge skips the expensive suite.
Don’t let the agent’s model grade itself. Use a different judge model; calibration, not model size, tells you if it’s good enough.
Gate on a pass rate, not one green run. Four of five.
Layer 3 is production: schema-valid rate, refusal count, judged samples, and a regression run before any prompt or model change ships.
The mock-the-model anti-test verifies your mocks, not your agent. If a test can’t fail when the agent gets worse, it isn’t a test.
Follow me on LinkedIn | Twitter(X) | Threads
Thank you for supporting this newsletter.
Consider sharing this post with your friends and get rewards.
You are the best! 🙏
Press the Like button if you found this post helpful
Deterministic tool tests plus scenario judges plus prod monitoring is the right stack. One add from our side: keep a tiny eval set of failure cases you actually care about (wrong vendor, duplicate invoice, ambiguous ask). Happy path suites lie.