Deterministic tool tests plus scenario judges plus prod monitoring is the right stack. One add from our side: keep a tiny eval set of failure cases you actually care about (wrong vendor, duplicate invoice, ambiguous ask). Happy path suites lie.
That's a great addition! Indeed, happy path scenarios are not enough. I tend to follow how we used to do the other tests where we cover the happy path but also the edge cases and failure modes.
Deterministic tool tests plus scenario judges plus prod monitoring is the right stack. One add from our side: keep a tiny eval set of failure cases you actually care about (wrong vendor, duplicate invoice, ambiguous ask). Happy path suites lie.
That's a great addition! Indeed, happy path scenarios are not enough. I tend to follow how we used to do the other tests where we cover the happy path but also the edge cases and failure modes.