Fault Tolerance for AI Agent Pipelines: Retry and Recovery Patterns

Back with another one in the series where I break down the parts of building agents that nobody warns you about until production breaks. This one is about fault tolerance, and I picked it because my first "working" agent pipeline fell over the first time a real user touched it. The demo was flawless: the agent planned a task, called three tools, wrote a summary, and returned. Then I shipped it, a tool timed out on request forty, the whole run threw, and the user got a stack trace instead of an answer. I had built something that worked exactly once per happy path, not a fault tolerant ai agent pipeline with real error recovery, which is a very different thing.

If you are building on top of an orchestration layer, this pairs closely with the AI agent orchestration frameworks guide: the framework gives you the wiring, and fault tolerance is what keeps that wiring from catching fire.

Why AI Agent Pipelines Fail Differently

I spent years writing backend services, so I assumed agent reliability was the same problem with a new label, and it is not. A normal request is short, returning in a couple hundred milliseconds, whereas an agent run is long, making a dozen model calls and tool calls over thirty seconds or two minutes. Every one of those calls is a network hop to something that can be slow, rate-limited, or down, and none of them respond in real time the way a local function would, so each adds to the response time users feel. The math is unforgiving: if each of ten steps succeeds ninety-nine percent of the time, the whole run only succeeds about ninety percent of the time, so ten percent of your users see a failure, and they see it after waiting.

The steps are also not independent. Step six depends on the output of step five, and when step five returns garbage instead of failing outright, step six does something confidently wrong and the error surfaces three steps later where it is almost impossible to trace. Partial failure is therefore the normal case, not the exception, and any single tool that everything depends on becomes a single point of failure whose failure mode ripples outward. Non-determinism compounds all of this, because the same input can produce a slightly different plan each time, so the failure you saw once might not reproduce when you go looking for it. That is what makes building resilient ai agents genuinely hard: you are adding fault tolerance to a system that will not even fail the same way twice.

What Fault Tolerance Actually Means Here

Fault tolerance is the property of a fault tolerant system that keeps doing its job when parts of it fail, which does not mean nothing ever breaks; it means a broken part does not trigger a full system failure and the pipeline can continue operating on the rest. Good error recovery is the difference between a degraded answer and no answer at all. For an agent pipeline I think about it as four promises: retry things worth retrying, stop calling dependencies that are clearly down, resume from where it stopped instead of starting over, and when it truly cannot finish, fail in a way that is safe, visible, and recoverable rather than silent.

None of these is exotic; they are the same fault tolerance patterns distributed systems have relied on for years. If you want the broader version of this story, the piece on cascading failures in distributed systems covers how one slow dependency drags down everything upstream, because agent pipelines are just distributed systems where some of the nodes happen to be language models.

Retry Patterns That Actually Work

The first instinct everyone has is to wrap the flaky call in a retry loop, and while that instinct is right, the naive version of it is dangerous. Consider the trap: a tool times out, so you retry immediately, it times out again because it is overloaded, and your instant retries are now part of what is overloading it, producing a retry storm that makes the outage worse. Good ai agent retry patterns are built around two ideas, back off and add jitter. Exponential backoff means you wait longer after each failure, one second then two then four, while jitter adds a random offset so a thousand clients that failed at the same instant do not all retry at the same instant. Without jitter, backoff just synchronizes the stampede.

import random, time

def call_with_retry(fn, max_attempts=4, base=1.0, cap=30.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except TransientError as e:
            if attempt == max_attempts - 1:
                raise
            # exponential backoff with full jitter
            delay = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, delay))

The part that took me longest to internalize is that you should only retry idempotent work: retrying a read is free, but retrying "charge the customer" can do the thing twice, so before you retry a tool call you have to know whether running it again is safe, and if it is not, you make it safe by giving each operation an idempotency key so the downstream service recognizes a repeat and ignores it. The other half of retrying well is knowing what not to retry: a 500 or a timeout is worth another shot, but a 400 whose error message says your arguments are malformed is not, because that api call will fail identically every time and just burn your budget. Retry the transient, fail fast on the permanent.

Circuit Breakers: Stop Hammering a Dead Dependency

Retries handle the blip, not the outage. If a dependency has been down for two minutes, sending it a fresh retry every few seconds is pure waste that delays the moment your pipeline admits defeat and does something useful instead. The circuit breaker pattern fixes this by sitting in front of a dependency and counting failures: while things are healthy it stays closed and lets calls through, once failures cross a threshold it trips open and every call fails instantly for a cooldown window instead of waiting on a doomed request, and after the cooldown it goes half-open, lets a single trial call through, and either closes on success or re-trips on failure.

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at, self.state = 0, None, "closed"

    def call(self, fn):
        if self.state == "open":
            if time.time() - self.opened_at < self.cooldown:
                raise CircuitOpen("dependency is down, skipping call")
            self.state = "half_open"
        try:
            result = fn()
            self.failures, self.state = 0, "closed"
            return result
        except Exception:
            self.failures += 1
            if self.failures >= self.threshold:
                self.state, self.opened_at = "open", time.time()
            raise

What the breaker really buys you is speed of failure, turning a slow failure mode into a fast one. A pipeline that fails in fifty milliseconds because a breaker is open can fall back to a cached answer or a simpler model and continue operating, keeping response time low and still returning something, whereas a pipeline that waits thirty seconds on each dead call just makes the user wait longer for the same disappointment. Failing fast is a feature.

Checkpointing: Resume Instead of Restarting

This is the pattern that changed how my pipelines behave under failure, and it is the one people skip most often. Picture a run that gets through eight expensive steps and dies on the ninth because a tool hiccuped: the naive design throws everything away and restarts from step one, so you pay for those eight steps again, model calls included, and at scale the cost is real, never mind the latency. Checkpointing means you persist the pipeline's state after each meaningful step, so when a run fails and retries, it loads the last checkpoint and continues from step nine instead of step one. Most serious orchestration layers now ship this as durable state, which is one of the strongest reasons to build on a real framework instead of a hand-rolled loop that can spin into an infinite loop when a running agent loses its place.

def run_pipeline(run_id, steps, store):
    state = store.load(run_id) or {"index": 0, "data": {}}
    for i in range(state["index"], len(steps)):
        state["data"] = steps[i](state["data"])
        state["index"] = i + 1
        store.save(run_id, state)   # checkpoint after every step
    return state["data"]

Checkpointing also unlocks human-in-the-loop steps for free, because durable state lets you pause a run to wait for an approval, shut the process down, and resume the exact same run hours later. In a multi-agent setup this matters even more, since one agent stalling should not force a restart of the others. The guide to building multi-agent systems goes deeper on how supervisor and worker agents coordinate around shared state, and durable checkpoints are what make that coordination survive a crash.

Dead-Letter Queues and Graceful Degradation

Some work will exhaust every retry and still fail, and the wrong answer is to swallow the error and move on, because then you have silently dropped a user's request and nobody knows. A dead-letter queue is the standard move: when an operation fails past its retry budget, you push it with its full context onto a separate queue, so nothing is lost and you can inspect what failed, fix the underlying cause, and replay those items later. It turns a silent data-loss bug into a visible backlog you can actually work through.

Graceful degradation is the companion idea. When a step fails, ask whether the whole run needs to fail with it, because often it does not: if the enrichment tool is down you return the answer without the enrichment and flag it as partial, and if the premium model is rate-limited you fall back to a cheaper one, the same way load balancing shifts traffic off a hot node in the real world. A degraded answer that arrives beats a perfect answer that never does, and users forgive "here is most of what you asked for" far more readily than a spinner that ends in an error. The design rule I settled on is to decide, for every step, whether it is required or optional: required steps failing means the run fails cleanly with a clear message, while optional steps failing means the run continues with a noted gap, and writing that down per step turned a lot of my vague error handling into something predictable.

You Cannot Fix What You Cannot See

Every pattern above assumes you know a failure happened and where, which is a genuinely hard assumption in an agent pipeline because a run touches many services and the failure that matters is often three steps upstream of where the exception finally lands. This is why fault tolerance and observability are the same project: you need each run to carry a trace so you can see every step, its inputs, its outputs, its latency, and exactly where it broke, because without that you are guessing and non-determinism means your guesses will not reproduce. I wrote about the tooling side of this in LLM observability and tracing for agents in production, and I would not run any of the recovery patterns here without it, since retries, breakers, and dead-letter queues all emit signals, from a single error message to a real time latency spike, and observability is how you hear them.

Common Mistakes I Made

I retried non-idempotent calls and double-sent a batch of notifications to real users, which taught me idempotency keys are not optional the moment retries touch anything with a side effect. I also set my retry count high and my timeouts long, thinking patience meant resilience, when it actually meant a single failing dependency could hold a run hostage for two minutes before giving up; aggressive timeouts plus a circuit breaker recover faster than a long, hopeful wait. I logged errors but did not trace runs, so I could see that something failed but not why, and adding a trace ID that flowed through every step was the single highest-leverage change I made. Finally, I treated every step as required, so one optional formatting tool going down would kill an otherwise complete answer, and sorting steps into required and optional immediately cut my visible failure rate because most of what was "failing" did not actually need to.

Wrapping Up

Fault tolerance is not one feature you add at the end; it is a handful of small decisions spread across the pipeline: retry the transient, break on the sustained, checkpoint the expensive, degrade the optional, and trace everything so you can see what happened. Each one is boring on its own, but together they are the difference between a demo and something you can leave running while a real user leans on it.

I have been building and breaking these pipelines on Levelop while working through the agent engineering track, and this pattern came out of one too many 2 a.m. failures. If you want the rest of the series, the Levelop blog collects the other write-ups on orchestration, memory, and observability.