We've run this movie before.
A monolith gets uncomfortable, a conference talk says the future is lots of small services, everyone splits everything, and two years later half the industry is writing "we went back to the monolith" retrospectives.
That was microservices.
Multi-agent AI systems are speed-running the same arc right now. Same hype curve, same whiteboard-driven splits, same costs discovered in production.
I work on a system that runs 15+ specialized agents behind a supervisor, and I want to make two arguments that sound contradictory but aren't.
First, the split is genuinely worth it; our system couldn't work as one agent. Second, you almost certainly shouldn't start there, and the right moment to split is measurable rather than aesthetic.
Share this post & I’ll send you some rewards for the referrals.
Run Your Auth With AI Agents (Partner)
Coding agents are here to stay, but vibe-coding auth is dangerous business. Connect your AI agents to the Descope MCP server instead!
This remote MCP server gives agents the ability to read documentation, manage users and tenants, configure auth flows, generate FGA schemas, review audit logs, and more…all through natural language.
(Thanks to Descope for partnering on this post.)
Why one agent degrades
A single agent scales beautifully until it doesn't, and the failure is gradual enough that teams miss it.
Three forces compound.
1. Tool choice gets worse as the tool list grows.
With 5 tools, the model picks correctly nearly every time. With 25, the descriptions blur (update_expense vs modify_expense_details vs edit_submitted_expense) and it starts grabbing the almost-right one.
Which means the cheap fix comes before the split. Mark the long tail defer_loading: true, register a tool-search tool, and let the model pull in what this turn needs.
If the active list is still too wide after that, the domain is telling you something.
2. The system prompt becomes a legal document.
Instructions for expense approvals, travel booking, and knowledge lookup all live in one prompt, and they start conflicting. "Always confirm before any write" (payments needs this) fights "resolve the question in as few turns as possible" (support wants that).
The model doesn't crash on contradictions. It obeys them inconsistently, which is worse.
3. The blast radius becomes total.
Tune one sentence to fix the refund flow, and you've changed behavior everywhere.
Nobody can say what a prompt edit affects, so nobody wants to touch it. Same freeze that hits a scary legacy module, except the module is English prose.
None of this hurting yet?
Then you don't need multi-agent yet. One agent with eight tools and a coherent prompt outperforms a fleet of specialists connected by hope.
The supervisor pattern
Once the pressure is real, the split that works in production looks like this.
A supervisor sits at the front. Its only job is classification and routing. Read the intent, pick the specialist, hand off. It has no domain tools and does no domain work.
The moment your supervisor starts answering questions, you've rebuilt the mega-agent with extra steps.
Specialists own one domain each (payments, approvals, travel, procurement, knowledge lookup). Each gets three narrow things:
Its own prompt. A page of instructions about one job, with no contradictions, because there's nothing to contradict.
Its own toolset. Five to ten tools, all obviously distinct. Tool-choice accuracy comes back for free.
Its own state. The conversation slice and working memory for its task, checkpointed so a handoff or a crash doesn't lose the thread.
The counterintuitive part is that specialists get better by knowing less.
A payments agent that has never heard of travel booking can't be confused by it. The narrowing is the feature, the same reason a focused module beats a God class, transplanted into prompt space.
The costs nobody puts on the slide
Every one of these showed up for us. Budget for them before you split.
1. Routing becomes your biggest failure mode.
Every misroute is a guaranteed bad experience, because the travel agent cannot answer a payments question no matter how good it is.
You need routing accuracy in your evals, ambiguous-intent test cases, and a fallback ("ask a clarifying question") for genuinely unclear requests.
2. Every hop is latency, tokens, and a colder cache.
Supervisor classification is a model call before any real work starts. Handoffs are more. A three-hop flow can double your latency and your bill against a single-agent baseline. And prompt caching works on an exact prefix, so each agent now has its own cached prefix to warm instead of one shared one. Measure it, because users feel it.
3. Handoffs need a contract.
Say the approvals agent needs a payment executed mid-flow. What crosses the boundary? Pass raw conversation history, and specialists drown in irrelevant context. Pass too little and the user repeats themselves.
What works for us is a typed handoff, a structured summary of task, entities, and decisions so far, as a Pydantic model because our agents are Python:
from pydantic import BaseModel
class Handoff(BaseModel):
task: str # "execute the approved refund"
entities: dict[str, str] # {"expense_id": "exp_9x2", "user_id": "u_17"}
decisions: list[str] # ["manager approved 2026-07-01", "amount 4200 cents"]
open_questions: list[str] = [] # what the next agent still has to resolve
source_agent: str # "approvals_agent", for the traceBoundaries between agents are boundaries like any other.
3. Debugging spans agents now.
"The answer was wrong" might live in the supervisor's routing, the specialist's reasoning, or a tool result two handoffs back.
Without one trace tree per request across all hops, you're doing archaeology.
4. Each new agent carries infrastructure, unless you template it.
A specialist needs its graph compiled, its state checkpointed, its invocations logged, its errors handled, and none of that is domain work.
In the system I work on, every specialist extends one shared BaseAgent that owns all four, so a new agent declares a name, a state shape, and its nodes. The supervisor, state graph, and checkpointer stay shared infrastructure too.
The payoff that justifies all of it
Here's the operational win that made me a believer.
When each agent is a self-contained unit (prompt, tools, state), you can run two versions of one specialist side by side.
We literally keep supplier_management_agent and supplier_management_agent_v2 as siblings. The router sends a slice of traffic to v2, evals compare the two on live work, and rollback is a config flip.
Try that with a mega-agent. Rewriting one section of a 2,000-line prompt changes behavior globally, instantly, for everyone. There is no gradual rollout of a paragraph.
The same boundary buys you a second thing: model choice per specialist.
Swapping models mid-conversation throws away the cached prefix, so a single agent is stuck on one model for the whole run. A separate specialist can sit on a cheaper, faster model without touching the main loop's cache.
Agent boundaries are deployment boundaries, and deployment boundaries are what let you ship changes without holding your breath.
Conclusion
The microservices lesson transfers whole:
Split along domains, never along workflow steps.
A payments agent and a travel agent are real domains, with different tools, rules, and failure modes. "Agent 1 gathers info, agent 2 makes a plan, agent 3 executes" is not a domain split. It's one workflow sliced into three prompts that must all change together, a distributed monolith with extra model calls.
Two agents that can never be deployed or evaluated independently are one agent.
And if the supervisor keeps bouncing users between two specialists mid-conversation, those two are telling you they're one domain.
Multi-agent isn't hype. It's what a serious agent system grows into, the way serious codebases grow modules. Growth is the operative word. The mega-agent's pain arrives on a schedule (tool sprawl, prompt contradictions, frozen iteration), and the right response is the one that survived the microservices decade.
Start with one, try the cheap fixes, watch for the measurable signals, split along domains, and pay the routing-and-tracing tax with your eyes open.
One good agent beats five mediocre ones. Five focused agents beat one confused one. The skill is knowing which side of that line you're on, and that's a measurement rather than a vibe.
📌 TL;DR
Start with one agent. Split under measured pressure, not because the architecture diagram looks smarter with more boxes.
The splitting signals are concrete: a tool list you can't shrink even after deferring the long tail behind tool search, a system prompt that contradicts itself, and teams who can't change one behavior without breaking another.
The supervisor pattern: a router classifies intent and hands off to a specialist, each with its own narrow prompt, own small toolset, own state. Specialists get better by knowing less.
The costs are real: an extra model call before any work starts, misroutes (your new top failure mode), handoff contracts, traces that span agents, and a prompt cache per agent instead of one.
A shared
BaseAgentkeeps the per-agent cost flat. Graph compilation, checkpointing, logging, and error handling are inherited; a new specialist is a name, a state shape, and nodes.The killer operational win is versioned coexistence. Run
agent_v2next tov1, route a slice of traffic, roll back with a config flip. You can't A/B one paragraph of a 2,000-line mega-prompt.Split along domains (payments, travel, knowledge), never along "steps" of a workflow that always run together. A distributed monolith is still a monolith, even with agents.
Follow me on LinkedIn | Twitter(X) | Threads
Thank you for supporting this newsletter.
Consider sharing this post with your friends and get rewards.
You are the best! 🙏
Press the Like button if you found this post helpful
This is the same lesson distributed systems learned the hard way: every boundary you add is a place where state can drift and nobody's accountable for the seam. With multi-agent it's worse because the 'network call' between agents is a natural-language handoff, so failures are silent instead of throwing an exception. Curious whether you think there's an equivalent to a service mesh for agents, something that makes the handoffs observable, or if that's a sign the handoff shouldn't exist yet.
The growing-contract signal is the part I'll steal. It reframes the whole thing: you don't decide whether to split upfront, you let the schema tell you when a seam is really two prompts pretending to be one domain. And making the validated payload throw is exactly the move distributed systems eventually landed on too — turn silent drift into a loud exception at the boundary.