Picture a support agent handling a trip cancellation, 14 steps into a run. At step 3 it emailed the traveler. Then a deploy restarts the pod, the job queue retries the job, and the agent starts over from step 1. The traveler gets the same email twice.
The retry did its job. Re-running from the top is harmless for a request and a problem for an agent that already acted.
It also happens a lot. Say each model or tool call succeeds 98% of the time. Over 20 calls, only ≈ 67% of runs get through clean, so 1 run in 3 hits a failure. Even at 99%, it’s 1 in 5.
Timeout, retry, and circuit breaker from my earlier post all still apply. They just protect one call. This post is about protecting the whole run.
Share this post & I’ll send you some rewards for the referrals.
An agent isn't a request
Three things break the classic playbook:
It’s long. Minutes, not milliseconds. Plenty of time for a deploy to land mid-run.
It’s stateful. Step 14 builds on steps 1 to 13. Lose the process and you lose the run.
It acts. It sends emails and moves money, so replaying a step can do real damage.
Each classic pattern breaks on one of these properties. Here are the five adjustments that fix them.
1. Retry the step, not the run
You can retry the call, the step, or the whole run. Retry the step, because retrying the run pays again for the 13 steps that worked and replays their side effects.
The call level is already covered. The OpenAI and Anthropic SDKs retry connection errors, 429s, and 5xx twice by default, with backoff. What’s left for you is sorting step failures into three piles.
Time fixes it. A 429, a timeout, a 500, Anthropic’s 529 “overloaded”. Retry the step later.
The model fixes it. A booking ID that doesn’t exist. Send the error back as a tool result with
is_error: true, and the model usually corrects its call.Nothing fixes it. A revoked API key. Fail the step now.
Retries also multiply across layers. Three step attempts on top of the SDK’s own retries can mean nine model calls. Let one layer own retries. If it’s the step, set max_retries=0 and handle the Retry-After header yourself.
If you don’t want to handle this by yourself, Inngest helps with durable agentic workflows, so you don’t need to think about that. Each step.run gets its own retry counter, and the three piles become three except branches:
async def refund_booking(args, key):
try:
return await payments.refund(**args, idempotency_key=key)
except BookingNotFound as e: # the model fixes it
return {"error": str(e)}
except CredentialsRevoked as e: # nothing fixes it
raise inngest.NonRetriableError(str(e))
except RateLimited as e: # time fixes it
raise inngest.RetryAfterError("payments rate limit", e.retry_after_ms)Run it with ctx.step.run("refund-booking", refund_booking, args, key). Any other error gets Inngest's default retries. The key is what pattern 2 is about.
2. Make every tool safe to call twice
A retry is only safe when the thing being retried is idempotent.
Model calls are safe. Tools that write are the problem. create_ticket called twice creates two tickets.
The fix is the one that already protects payment APIs: an idempotency key. Stripe popularized this pattern and keeps keys for 24 hours; I've written a full guide to idempotency if you want the deep dive.
3. Checkpoint, so a crash means resume
Here's the pattern that single requests never needed.
A request that dies costs nothing. The client retries it from zero, and nothing is lost.
When a 20-minute run dies at step 14, "from zero" throws away 13 steps of work and replays every side effect along the way.
Save the state after each step instead. It's small, just the message history, the tool results, and a step counter.
async def run(run_id, task):
state = await load_state(run_id) or AgentState.new(task)
while not state.done:
state = await run_step(state)
await save_state(run_id, state) # after every step
return stateNow a crash costs one step, and pattern 2’s key makes re-running that step safe.
A checkpoint saves data, not execution, so something still has to notice a dead run and re-enter the loop, like a job queue or a durable-execution runtime. Runs also outlive deploys, so version your saved state like an API.
Inngest is that kind of runtime. It records each finished step, and after a crash it re-runs the function and skips the steps it already recorded. Here's the agent loop with one step per model call and one per tool call:
@inngest_client.create_function(
fn_id="agent-run",
trigger=inngest.TriggerEvent(event="agent/run.requested"),
)
async def agent_run(ctx: inngest.Context):
messages = [ctx.event.data["request"]]
for turn in range(20):
reply = await ctx.step.run(f"model-{turn}", call_model, messages)
if reply["done"]:
return reply
result = await ctx.step.run(f"tool-{turn}", call_tool, reply["tool"])
messages += [reply, result]
raise inngest.NonRetriableError("20 turns without an answer, needs a human")Say tool-3 issued the refund and the run dies in model-4. The retry reads everything up to tool-3 from Inngest’s record instead of calling it again, so the traveler isn’t refunded twice. If it dies inside tool-3, that step does run again, so call_tool passes the model’s tool-call ID as the idempotency key. The range(20) is the step cap from pattern 5, and hitting it fails the run instead of returning nothing.
One rule comes with this. Every model call, tool call, and database read goes inside a step, because code outside steps runs again each time the function resumes.
4. Fall back across failure domains, not just models
In a real provider outage, retries only make it worse. That’s where a circuit breaker and a fallback come in, with two agent rules.
Leave the failure domain. A sibling model at the same provider usually shares the outage. The cleanest fallback is often the same model on another cloud (Claude on Bedrock or Vertex AI, OpenAI models on Azure), so your prompts and evals still hold, as long as the features you use exist there. A gateway like LiteLLM or OpenRouter can do the routing.
Only switch to a model your evals passed. Models differ in how reliably they call tools, and 20 steps amplify small differences. Switch at a step boundary, which your checkpoint makes clean.
5. Bound the loop, then hand it to a human
The most expensive agent failure throws no error. The model loops on a tool that half-works, every call returns 200, and the run never ends.
So cap every run on steps, tokens, and wall-clock time, and stop it when a cap trips. Set a per-call timeout too. The OpenAI and Anthropic SDKs default to 10 minutes and retry a timeout twice, so one hung call can stall a run for half an hour.
A stopped run isn’t lost. Thanks to the checkpoint, it’s saved state plus a reason, which is exactly what a human needs to unstick it and resume. Runs that exhaust their retries belong there too.
Inngest can park the run while it waits for that human. Call this before any refund over your limit, and refund only if it comes back approved:
async def wait_for_approval(ctx: inngest.Context):
request_id = ctx.event.data["request_id"]
await ctx.step.run("ask-for-approval", request_approval, request_id)
for i in range(288): # check every 5 minutes, for 24 hours
decision = await ctx.step.run(f"check-{i}", read_decision, request_id)
if decision["state"] != "pending":
return decision # approved or rejected
await ctx.step.wait_for_event( # the event only wakes us early
f"wait-{i}",
event="refunds/decision-recorded",
if_exp="async.data.request_id == event.data.request_id",
timeout=datetime.timedelta(minutes=5),
)
return await ctx.step.run("expire", expire_if_pending, request_id)The saved decision is the source of truth, and the event only wakes the run early. wait_for_event only hears events sent after it starts listening, so a single 24-hour wait can miss a fast approver and expire an approved refund. Here a missed event costs five minutes. expire_if_pending only flips a record that’s still pending, so an approval and an expiry can’t both win.
The database does the rest. request_approval saves the request with its deadline under request_id, so a retry neither asks twice nor restarts the clock. The approval endpoint checks who's answering, accepts a decision only while the request is pending, saves it, then sends the event. expire_if_pending only flips a record that's still pending, so an approval and an expiry can't both win.
📌 TL;DR
At 98% per call, 1 in 3 runs of 20 calls hits a failure. Failure is the main path.
Retry the step, not the run. Retry what time fixes, send back what the model can fix, and fail fast on the rest. Let one layer own retries.
Give every tool that writes an idempotency key built from stable IDs, never one generated inside the step.
Checkpoint after every step, so a crash or a deploy means resume instead of restart.
Fall back to the same model on another cloud first, and only switch to models your evals passed.
Cap steps, tokens, and time; set per-call timeouts. The saved decision is the truth, and the approval event is only a wake-up.
Consider using Inngest for durable execution for workflows and AI
Follow me on LinkedIn | Twitter(X) | Threads
Thank you for supporting this newsletter.
Consider sharing this post with your friends and get rewards.
You are the best! 🙏
Press the Like button if you found this post helpful
the checkpoint pattern is the one i keep coming back to.
checkpoints save state. they don't save intent. resume a run at step 14 and you might be faithfully continuing a trajectory that went wrong at step 6. worth snapshotting the plan alongside the message history, so the resume check can flag drift too.