Hierarchical Agent Architecture: The Supervisor Agent Pattern

The first time I wired up multiple agents to work together, I let them all talk to each other. Any agent could call any other agent and share one message history. It worked in the demo. Then I gave it a real world task with a few branches and it turned into a group chat where nobody was in charge. Two agents ran the same search. One answered a question meant for another. The token bill closed the tab for me.

That mess pushed me toward a hierarchical agent architecture, and specifically the supervisor agent pattern. Instead of a flat crowd of peers, you put one agent in charge of routing work and a set of specialized agents underneath it. The supervisor decides who does what and when the job is done. The subagents do their one thing well.

What a hierarchical agent architecture is

A hierarchical agent architecture organizes agents into layers of responsibility. At the top sits a supervisor agent whose job is coordination, not execution. Below it sit subagents, each scoped to a narrow domain: one searches, one writes code, one queries knowledge bases, one drafts the answer. Work flows down, results flow back up, and the supervisor picks the next move.

The mental model that made it click is an org chart. A manager agent does not personally handle every complex task. It reads the request, figures out who is best suited, delegates, and checks the result. The individual agents are experts in narrow agent responsibilities and do not need the full plan. That single idea, separating coordination from execution, is the whole point. The supervisor holds the plan. The specialized agents hold the skills.

Why flat multi agent systems break down

Before the supervisor pattern, my agents were peers. Every one could see the full conversation and call any tool. That created three failures fast.

First, duplicated work. With no owner of the plan, two agents would independently decide a task needed a search, so both searched. Across a long run you pay for the same work many times.

Second, context bloat. When agents interact through one shared history, that history grows every step, and each agent processes the whole transcript on every call. The database agent does not need the messages about formatting the report, but in a flat design it reads them anyway. Anthropic's writeup on a multi agent research system makes the same point: subagents work best with their own focused context, not one thread that keeps expanding.

Third, unclear termination. Nobody owned the decision of when the work was finished, so agents kept handing tasks back and forth or stopped early.

A hierarchy fixes all three. One agent owns the plan, so work is not duplicated. Each subagent gets only the context it needs. And the supervisor is the single place where the "are we done" decision lives.

The supervisor agent pattern

The supervisor agent pattern is the simplest hierarchical design, and where I would start for almost any project. There is one supervisor and a flat set of subagents. The supervisor is the only agent that talks to the user and the only one that decides what happens next.

The supervisor reads the current state, decides which subagent acts next, and delegates. When the subagent reports back, it evaluates and either routes again or ends the run. It never does the work itself. Because its job is routing, its prompt is mostly a description of the team so it can match tasks to skills.

A subagent is a focused worker. It receives a specific task, uses its tool calls to complete it, and returns a result. It does not decide what happens next and does not know about the other agents. That isolation is the feature. A subagent with a small scope is easier to prompt, easier to test, and cheaper because its context stays lean. Need a new capability? Add a subagent and update the supervisor's list. You do not rewire the graph. This is the modularity that makes how to build multi agent systems tractable in the first place.

Designing the supervisor: routing and delegation

The heart of a supervisor is a routing decision: which subagent goes next, or is the task complete? The cleanest approach is to make the supervisor return a structured decision rather than free text. Free text means parsing the model's natural language to guess intent. Structured output makes routing unambiguous.

Here is stripped-down routing using a LangGraph-style state graph. The supervisor returns the next agent name, and a router sends state there.

from typing import Literal
from langgraph.graph import StateGraph, END

def supervisor(state):
    # Structured output forces one allowed name, not free text.
    decision = llm.with_structured_output(RouteDecision).invoke(
        [SUPERVISOR_PROMPT] + state["messages"]
    )
    return {"next": decision.next_agent}

graph = StateGraph(AgentState)
graph.add_node("supervisor", supervisor)
# Every worker reports back to the supervisor, never to each other.
for worker in ["researcher", "coder", "writer"]:
    graph.add_edge(worker, "supervisor")
graph.add_conditional_edges("supervisor", route, {
    "researcher": "researcher", "coder": "coder",
    "writer": "writer", "FINISH": END,
})

Two choices matter most. Every worker edge points back to the supervisor, never to another worker, which keeps the hierarchy intact so subagents cannot start freelancing. And FINISH is a first-class option, so the supervisor can end the run explicitly, solving the termination problem. Keep the supervisor prompt short: it runs on every hop, so a bloated one is slow and expensive in real time.

State and memory across the hierarchy

In a flat system there is one shared history you barely think about. In a hierarchy you decide, deliberately, what each layer sees. Give the supervisor a high level view: the plan, results so far, and routing history. Give each subagent a scoped view: its task plus the slice of context it needs, and nothing else. When it finishes, it returns a clean result, not its whole scratchpad.

This scoping is a cost and reliability lever, not just tidiness. A subagent that receives only its task runs on a small context window, which is faster and less likely to get distracted. Our AI agent orchestration frameworks guide walks through how the major frameworks handle state passing between coordinator and workers.

Error handling and recovery

A hierarchy gives you one natural place to handle failure. When a subagent fails, it reports up, and the supervisor decides what to do. There are three responses: retry the same subagent with a narrowed task, route to a different one, or return a partial result with an honest note. The supervisor is the right owner of that judgment because it has the full picture.

def supervisor(state):
    last = state["messages"][-1]
    if last.get("status") == "error":
        attempts = state.get("attempts", {})
        agent = last["from_agent"]
        attempts[agent] = attempts.get(agent, 0) + 1
        if attempts[agent] >= 2:      # stop looping on a failing worker
            return {"next": "writer", "attempts": attempts}
        return {"next": agent, "attempts": attempts}
    decision = llm.with_structured_output(RouteDecision).invoke(
        [SUPERVISOR_PROMPT] + state["messages"]
    )
    return {"next": decision.next_agent}

The attempt counter is the piece people skip and the one that saves you at 2 a.m. Without a cap, a subagent that keeps failing gets routed back to itself forever, burning tokens on an infinite loop. Counting attempts and forcing an escalation turns that into a bounded process. For anything running unattended, that bound is not optional. The same audit point is where you enforce access control on which agents may use sensitive tools.

When to go hierarchical, and when not to

A hierarchical agent architecture is not free. The supervisor adds a hop to every decision, so extra latency and tokens compared to a single agent. For a simple task one well-prompted agent can handle, a supervisor is over-engineering.

The pattern earns its keep when the work is varied and splits into distinct skills. If a request might need research, then code, then writing, and the mix changes per run, a supervisor routing to specialists beats one agent trying to be good at all three. It also helps when you need clear boundaries for safety and auditing.

My rough rule: if I can describe the task as "do this one thing," I use a single agent. If it is "figure out which of these to do, then do them in some order," that is a supervisor. And if even the supervisor feels too big, go one level deeper, where a top supervisor coordinates mid-level supervisors that each own a team. Reach for that nesting only once a single supervisor is clearly the bottleneck.

Common mistakes I made

I let subagents talk to each other "just this once." The writer needed one fact from the researcher, so I added a direct edge. Within a week the graph had side channels everywhere and I was back to a flat system in a supervisor costume. Keep the rule strict: workers report up, the supervisor routes down.

I overloaded the supervisor with domain logic, adding "if the task involves X, do Y" until it was doing half the reasoning. The supervisor should route, not solve. Moving that logic down into the subagents made routing faster and the subagents better.

I ignored termination until it bit me. Making "the task is complete" an explicit FINISH decision on every hop fixed more flaky agentic workflows than any other single change.

What to build next

Build it in stages. Start with one supervisor and two subagents, a researcher and a writer, and get the loop working. Add structured output so routing stops depending on parsing. Add the attempt counter and a FINISH path. Only then add a third subagent or nested supervisors. From there, the natural next topics are how state passes between coordinator and workers, and how you monitor a running hierarchy in production for ai systems at scale.

I have been working through agent design problems on Levelop's system design track, and the supervisor pattern kept coming back until I built it properly. If you are stuck in flat multi-agent chaos, the fix is almost always the same: stop letting everyone talk to everyone, and put one agent in charge of the plan.

Related reading

Start with the AI agent orchestration frameworks guide, then the hands-on walkthrough in how to build multi agent systems. More engineering breakdowns are on the Levelop blog.