Agent Orchestration Patterns

Interview answer (say this first). Orchestration is the decision about who does what, in what order, and who is allowed to stop the work. Start with a single agent; add a second only for a measured reason — specialisation, parallelism, or independent review. The common patterns are ReAct (reason and act in a loop), plan-and-execute (plan first, then run steps), router (classify then dispatch), supervisor (delegate to workers and aggregate), and evaluator or critic (check the work and send it back). Every added agent buys reliability with cost, latency, and a new class of coordination failures, so keep the control flow deterministic where you can and make only the uncertain step agentic.

Note:

Verified. Every runnable example on this page was executed offline in plain Python (Python 3.14). No model calls were made. Where a line represents model output, it is labelled illustrative.

Why this exists

The instinct is to build one powerful agent with every tool and a long prompt. It works in a demo and degrades in production for predictable reasons:

• Tool selection collapses. Accuracy drops as the tool list grows. With twenty tools, the model picks the wrong one often enough to matter.
• Context is polluted. A failed search, an old draft, and a user’s aside all stay in the window and influence later decisions.
• No parallelism. One loop is sequential by nature. Independent work waits in line.
• No separation of concerns. The prompt that plans, fetches, writes, and reviews is impossible to tune without breaking something else.
• No independent check. The same agent that wrote the answer grades it, and it is biased toward its own work.
• Debugging is guesswork. When the run fails, you have one blob of text, not named stages.

Orchestration splits the work so each piece is smaller, testable, and replaceable. The cost is coordination: more agents mean more prompts, more tokens, more latency, and more ways for the handoff between them to break.

Tip:

The one-sentence purpose. Orchestration is how you divide a task among agents to gain specialisation, parallelism, or independent review — and it is only worth the overhead when you can measure the gain.

Start from zero

Two distinctions do most of the work:

• Router vs supervisor. A router decides once, then gets out of the way. A supervisor stays in the loop, deciding repeatedly what to do next.
• Handoff vs agent-as-tool. A handoff replaces the active agent. Agent-as-tool keeps the caller in charge and just borrows the result.

The core idea

Think of a company instead of a single freelancer.

• A single agent is one very good generalist. Cheapest to start, best for small tasks, and the right default.
• A router is the front desk. It reads the request and sends it to billing, technical, or sales. One decision.
• A supervisor is a project manager. It keeps a task list, assigns each item to a specialist, collects results, and decides when the job is done.
• Workers are specialists. Each has a narrow brief and the few tools it needs.
• An evaluator is quality control. It checks the work against a checklist and rejects it or passes it.
• A critic is a code reviewer. It does not just reject; it explains what to change, and the author revises.

The classic topology looks like this:

flowchart TD
    U["User request"] --> R{"Router<br/>classify once"}
    R -->|"billing"| B["Billing agent"]
    R -->|"technical"| T["Technical agent"]
    R -->|"general"| G["General agent"]
    B --> A["Answer"]
    T --> A
    G --> A

A supervisor topology is a loop, because the supervisor keeps deciding:

flowchart TD
    U["Goal"] --> S["Supervisor<br/>decide next worker"]
    S --> W1["Worker: retrieve"]
    S --> W2["Worker: analyse"]
    S --> W3["Worker: write"]
    W1 --> S
    W2 --> S
    W3 --> S
    S -->|"enough"| A["Final answer"]
    S -.->|"cap steps"| L["Budget / turn limit"]

Choosing the smallest pattern that works is the whole skill:

How it works

1. Start with a single agent. Give it the tools and instructions for the task. Measure success rate, cost, and latency. This is your baseline. Every orchestration idea must beat it on a metric you care about.
2. ReAct runs a loop. Each iteration produces a thought, an action (a tool call), and an observation (the tool result). The loop ends when the agent emits a final answer or hits a step limit. It is the natural pattern when you cannot know the steps in advance.
3. Plan-and-execute separates planning from doing. A planner produces a list of steps. An executor runs them in order and reports which failed. If a step fails, the planner can revise the plan. This is better than ReAct when the whole shape of the task is knowable and you want the plan visible and auditable.
4. A router classifies once. It reads the request and outputs one label. Code then dispatches to the matching agent. The classifier can be a model, a keyword table, or a small trained model. Keep the routing decision loggable, because a misroute is expensive and invisible.
5. A supervisor delegates in a loop. It maintains the remaining work, picks a worker for the next step, calls that worker, records the result, and repeats until the goal is met or a limit fires. The supervisor must have a stopping rule, or it will delegate forever.
6. Workers are narrow. Each worker gets a small context and only the tools for its job. This is the main benefit: a worker cannot be confused by tools it does not have, and its mistakes stay local.
7. An evaluator judges against a rubric. It receives the output and returns a structured verdict plus reasons. The rubric must be explicit; “is this good?” is not a rubric. Pass or fail drives the next step: ship, retry, or escalate.
8. A critic improves rather than only judging. It returns specific feedback, the generator produces a revised draft, and the loop repeats until the score passes a threshold or a round limit fires. Without the round limit, critic loops can oscillate forever.
9. Choose single vs multi on evidence. Go multi-agent when the task genuinely spans domains, when independent work can run in parallel, when context must be isolated, or when you need an independent check. Stay single otherwise.
10. Put deterministic code around the agentic steps. Routing tables, validation, aggregation, budget checks, and the final approval are ordinary code. Only the genuinely uncertain decision should be a model call.
11. Account for the cost. Each agent has its own system prompt and history, so tokens grow roughly with the number of agents. Parallel workers cut latency but multiply peak cost.
12. Test the topology, not only the agents. A perfect worker in a broken chain still produces a broken run. Test each route end to end with a golden trace.

The syntax you will use

A ReAct step as data. Keeping the step explicit makes the loop testable and the trace readable.

from dataclasses import dataclass

@dataclass
class Step:
    thought: str
    action: str | None = None        # tool name, or None to answer
    action_input: str | None = None
    observation: str | None = None   # filled in after the tool runs

The ReAct loop. The policy stands in for the model; the tools are ordinary functions.

def react_loop(policy, tools, max_steps=8):
    history = []
    for _ in range(max_steps):
        step = policy(history)
        if step.action is None:
            history.append(step)
            return history, "answered"
        if step.action not in tools:
            step.observation = f"ERROR: unknown tool {step.action}"
        else:
            step.observation = tools[step.action](step.action_input or "")
        history.append(step)
    return history, "max_steps"        # a limit always exists

A router as a table. Rules first; replace with a model only when rules genuinely fail.

ROUTES = {"billing": ["invoice", "refund", "charge"],
          "technical": ["error", "crash", "timeout"],
          "sales": ["price", "plan", "quote"]}

def route(query: str) -> str:
    q = query.lower()
    for name, words in ROUTES.items():
        if any(w in q for w in words):
            return name
    return "general"

A plan-and-execute loop with a bounded replan. Run the plan, log every step, and let the planner revise at most max_replans times.

def plan_and_execute(planner, executor, goal, max_replans=2):
    plan = planner(goal)
    log = []
    for attempt in range(max_replans + 1):
        ok = True
        for step in plan:
            result = executor(step)
            log.append((attempt, step, result))
            if result is None:        # a step failed
                ok = False
                break
        if ok:
            return log, "done"
        plan = planner(goal + " (revised)")
    return log, "gave_up"             # the replan cap fired

A supervisor. Pick, run, record, repeat, and always cap the steps.

def supervise(goal, choose_worker, workers, max_steps=6):
    results = []
    for _ in range(max_steps):
        name = choose_worker(goal, results)
        if name is None:
            break
        results.append((name, workers[name](goal)))
    return results

Parallel workers. Independent work runs concurrently, and failures are handled per worker.

import asyncio

async def run_workers(workers, task):
    outcomes = await asyncio.gather(*(w(task) for w in workers), return_exceptions=True)
    return [r for r in outcomes if not isinstance(r, Exception)]   # keep the good ones

A critic loop with a threshold and a cap. Both limits matter.

def generate_then_critique(generate, critique, threshold=0.9, max_rounds=5):
    draft = generate(0)
    for i in range(max_rounds):
        score = critique(draft)
        if score >= threshold:
            return draft, "passed"
        draft = generate(i + 1)
    return draft, "max_rounds"

The SDK’s two coordination forms. In openai-agents, a handoff transfers control; Agent.as_tool() keeps the caller in control. Both were introspected from version 0.22.2.

from agents import Agent

writer = Agent(name="Writer", instructions="Draft the answer.")
reviewer = Agent(name="Reviewer", instructions="Review and finalise.",
                 handoffs=[writer])          # handoff: reviewer can hand control to writer

coordinator = Agent(name="Coordinator", instructions="Own the task.",
                    tools=[writer.as_tool(tool_name="draft", tool_description="Get a draft.")])
# agent-as-tool: coordinator stays in control and calls the writer like a function

Examples: simple to real

Example 1 — a ReAct loop that terminates.

A scripted policy and two tools. Verified output:

react status: answered
steps: [('search', 'docs about refund window'), ('calc', '42'), (None, None)]

The third step has no action, so the loop ends. An unknown tool is caught rather than crashing the loop, which is what a real agent must do:

react unknown tool: ERROR: unknown tool missing

Example 2 — a router that sends each request to one path.

Verified routing decisions:

"I need a refund"          -> billing
"app crash on start"       -> technical
"how much is the pro plan" -> sales
"hello"                    -> general

Four requests, four routes, one decision each. The danger is not the routing code; it is a misclassification that no one notices because the wrong agent still answers politely.

Example 3 — a supervisor with workers.

The supervisor selects workers by name and collects their results in order. Verified:

supervisor: {'retriever': 'docs for quarterly report',
             'analyzer': 'analysis of quarterly report'}

list(out) == ['retriever', 'analyzer']. The supervisor owns the order; the workers own the work. This is also the natural place to add a step cap and a cost check.

Example 4 — plan-and-execute with one replan.

The first execution fails at step-b, so the planner produces a revised plan and the executor succeeds. Verified:

plan-and-execute: done | entries: 4 | replans: {0, 1}

Attempt 0 and attempt 1 both appear in the log. That log is the audit trail: you can see exactly when the plan changed and why.

Example 5 — a critic loop that stops at the threshold.

Each revision improves the score. Verified:

critic loop: passed scores: [0.4, 0.7, 1.0]

It stopped after the third draft because 1.0 >= 0.9. Had the score never reached the threshold, max_rounds=5 would have stopped it. Always have both exits.

Example 6 — orchestration is not free.

Using an illustrative price of $3 per million input tokens and $15 per million output tokens:

single agent (2,000 in, 500 out):   $0.0135
4-agent pipeline (1,500 in, 300 out each): $0.0360
ratio: 2.67x

Latency behaves differently depending on topology. For three steps of 0.8s, 0.6s, and 0.5s:

sequential: 1.9s
parallel:   max(0.8, 0.6, 0.5) = 0.8s
saved:      1.1s

Parallelism wins only when the steps are independent. Dependent steps must stay sequential, whatever the topology looks like on a whiteboard.

In production

• Default to one agent. Every additional agent adds a prompt, a context, and an interface to test. Split only when a metric — success rate, latency, or review quality — justifies it.
• Isolate worker context. The main benefit of workers is that each sees less. Passing the full conversation to every worker throws that benefit away and multiplies cost.
• Always cap the loop. Supervisors and critics need a maximum number of steps and rounds. An uncapped supervisor is an unbounded spend.
• Make routing observable. Log the route, the confidence if you have one, and the fallback. A silent misroute is the most expensive orchestration bug because the answer still looks plausible.
• Validate the route. If the classifier is uncertain, ask a clarifying question or fall back to the general agent. Do not route on a coin flip.
• Parallel workers need idempotent tools. Two workers may retry the same side effect. Give each worker a stable operation key and deduplicate at the tool layer.
• Handle partial failure in fan-out. Decide up front whether one failed worker fails the run or the others proceed. asyncio.gather(..., return_exceptions=True) plus an explicit policy is clearer than an accidental crash.
• Give the critic a rubric and a cap. A critic with vague instructions produces vague feedback and can oscillate. A rubric plus max_rounds makes it a bounded improvement loop.
• Watch collusion and bias. An evaluator tends to approve work that resembles its own. Where it matters, use a different model or a deterministic check.
• Keep deterministic code in charge. The router table, the budget check, the aggregation, and the approval gate are code. Only the uncertain decision is a model call.
• Correlate traces across agents. Give the whole orchestration one run id and attach it to every sub-call, or you cannot reconstruct a failure.
• Test each route end to end. Unit-test the workers, then add a golden-trace test per route. A change that fixes the billing route can silently break the technical route.

Interview questions

1. When do you move from a single agent to multiple agents?

Answer. When you can name the reason and measure the gain. The valid reasons are domain specialisation (one prompt cannot hold all the instructions), context isolation (workers must not see each other’s noise), parallelism (independent work must run at once), and independent review (you need a check the generator did not perform). If none of those applies, a single agent with better tools and a tighter prompt is simpler and usually better.

Follow-up: “What is the first thing you try before splitting?” Improve the tool descriptions, remove unused tools, and tighten the instructions. Bad tool selection is often a description problem, not a reason to add an agent.

Trap. Splitting agents to fix a prompt bug. You now have two prompts with the same bug and a new interface that can also fail.

2. What is ReAct, and why is it still the default loop?

Answer. ReAct interleaves reasoning and acting: the agent thinks, calls a tool, reads the observation, and repeats until it can answer. It is the default because it needs no advance plan and adapts to what the tools return. It is the right shape when the steps are genuinely unknown, such as debugging or open-ended research.

Follow-up: “What is its weakness?” It has no global view. It can loop, repeat a failed action, or drift from the goal, so it needs a step limit and a way to detect repeated states.

Trap. Calling ReAct a framework. It is a loop pattern; you can implement it in twenty lines with a while loop and a tool registry.

3. Router versus supervisor — what is the difference?

Answer. A router makes one classification and dispatches; it is not involved after that. A supervisor stays in control and makes repeated delegation decisions until the task is done. Routers are cheap, fast, and ideal for distinct request types. Supervisors handle broad tasks that need several skills and an ongoing plan. A supervisor may contain a router as its first step.

Follow-up: “When is a router enough?” When each request belongs to exactly one specialist and the specialist can finish alone. If the task needs several specialists or a result must be assembled from several workers, you need a supervisor or a pipeline.

Trap. Building a supervisor when a router suffices. The extra loop adds cost and a failure mode for no benefit.

4. What is the difference between a handoff and an agent-as-tool?

Answer. A handoff transfers control: the new agent replaces the old one and owns the rest of the run. Agent-as-tool calls another agent for a bounded job and returns its output to the caller, which keeps control. Use a handoff when the specialist should finish the task; use agent-as-tool when the caller must combine the result with other work.

Follow-up: “What happens to context in each?” A handoff can carry the conversation to the new agent, so use an input filter to limit it. Agent-as-tool gives the sub-agent only the input the caller passes, which is better isolation by default.

Trap. Using a handoff and then expecting the original agent to post-process the result. After a handoff, control has moved.

5. When is plan-and-execute better than ReAct?

Answer. When the task has a knowable structure and you want it visible and auditable. Plan-and-execute produces a step list up front, so a human or a policy can approve it, and execution can be retried per step. ReAct is better when the path only becomes clear as you go and planning upfront would be wasted work.

Follow-up: “How does it handle a failing step?” The executor reports the failure and the planner revises the remaining plan. The replan is the interesting part, and it must be bounded, or the agent will replan forever.

Trap. Planning at a level too coarse to execute. “Research the market” is not a step; “collect the top five competitor prices” is.

6. How do you control the cost and latency of orchestration?

Answer. Measure both per run. Reduce agents, because each adds its own system prompt and history. Isolate context so workers do not receive the whole conversation. Run independent workers in parallel and keep dependent ones sequential. Cap turns, rounds, and total spend. Cache retrieval and repeated tool calls. Choose a smaller model for routing and evaluation, and reserve the strongest model for the hard reasoning step.

Follow-up: “When is a multi-agent system cheaper than a single agent?” When context isolation lets each worker use a much smaller prompt, or when a cheap router avoids an expensive generalist. Rarely, but real.

Trap. Assuming parallelism always lowers latency. It lowers wall-clock time only for independent steps, and it raises peak cost because all workers run at once.

7. Evaluator versus critic — are they the same?

Answer. An evaluator judges and returns a verdict; a critic judges and produces feedback that drives a revision. Evaluator is used as a gate (“does this pass?”). Critic is used as an improvement loop (“here is what to change”). Both need an explicit rubric, and both need a round limit.

Follow-up: “Why can a critic loop fail?” The critic and generator can oscillate between two states, the critic can be too vague to act on, or the score can plateau below the threshold. Cap the rounds and record the scores so a plateau is visible.

Trap. Letting the generator grade itself. Use a separate call, a separate prompt, or a deterministic check, and accept that some self-evaluation bias is unavoidable.

8. How do you combine a deterministic workflow with agentic steps?

Answer. Make the workflow the skeleton and the agent the muscle. Code owns the sequence, the routing table, validation, budget checks, retries, and the final approval. The model owns only the step that genuinely needs judgement, such as classification, drafting, or extraction. The model’s output is validated as structured data before it is allowed to influence control flow.

Follow-up: “Why not let the model control the whole workflow?” Because control flow should be testable and reproducible. A model-chosen branch is hard to reason about, and an invalid branch can skip a safety check. Deterministic code is cheap, fast, and debuggable.

Trap. Putting the agent in charge and the code in a supporting role. Then the failure modes of the model become the failure modes of the system, including skipping the checks you wrote.

Remember this

• Start single-agent. Add an agent only for specialisation, isolation, parallelism, or independent review, and prove the gain with a metric.
• ReAct is a loop; plan-and-execute is a plan. Use the first when the path is unknown and the second when the shape is knowable.
• A router decides once; a supervisor decides repeatedly. Do not build a supervisor when a router suffices.
• Handoff transfers control; agent-as-tool borrows a result.
• Every loop needs a limit, and every model-driven decision needs a deterministic check around it.