· 10 min read
How we structure multi-agent systems in LangGraph
Why we moved from single agents to multi-agent graphs, and how we handle state, error recovery and tool routing after 18 months in production.
Where single agents break down
When we started building LLM applications at MiraiTech, a single ReAct agent looked like enough: give it tools and a system prompt and let it reason. That stopped working once production traffic arrived.
A single agent hits a complexity ceiling. Past 5-7 tools, its accuracy drops. The context window fills with irrelevant tool output, and error recovery mostly means hoping the next attempt works. So we moved to multi-agent orchestration.
LangGraph lets us build these systems as stateful, observable graphs with explicit control flow.
Architecture patterns
Supervisor architecture
A central supervisor agent delegates work to specialized worker agents. Of the patterns here, it is the easiest to reason about and to debug.
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.types import Command
from langchain_openai import ChatOpenAI
from typing import Literal
llm = ChatOpenAI(model="gpt-4o")
def supervisor(state: MessagesState) -> Command[Literal["researcher", "writer", "reviewer", END]]:
"""Central orchestrator that routes to specialized agents."""
system_prompt = """You are a supervisor managing three workers:
- researcher: finds and validates technical information
- writer: produces polished content from research
- reviewer: checks accuracy and suggests improvements
Based on the conversation, decide which worker should act next.
If the task is complete, respond with FINISH."""
response = llm.with_structured_output(RouterOutput).invoke(
[{"role": "system", "content": system_prompt}] + state["messages"]
)
if response.next == "FINISH":
return Command(goto=END)
return Command(goto=response.next)
def make_agent(name: str, tools: list, system_prompt: str):
"""Factory for building worker agents with consistent structure."""
agent = create_react_agent(llm, tools, prompt=system_prompt)
def node(state: MessagesState) -> Command[Literal["supervisor"]]:
result = agent.invoke(state)
return Command(
update={"messages": result["messages"]},
goto="supervisor",
)
return node
graph = StateGraph(MessagesState)
graph.add_node("supervisor", supervisor)
graph.add_node("researcher", make_agent("researcher", research_tools, research_prompt))
graph.add_node("writer", make_agent("writer", writing_tools, writer_prompt))
graph.add_node("reviewer", make_agent("reviewer", review_tools, reviewer_prompt))
graph.add_edge(START, "supervisor")
app = graph.compile()Peer-to-peer architecture
Agents talk to each other directly, and each one decides who to hand off to next. This suits dynamic workflows where the best path cannot be predicted.
Choosing a pattern
| Pattern | Best for | Downsides |
|---|---|---|
| Supervisor | Predictable workflows, clear task decomposition | Bottleneck at supervisor, higher latency |
| Peer-to-peer | Dynamic exploration, creative tasks | Harder to debug, loop risk |
| Hierarchical | Large systems (10+ agents) | Complex state management, slow cold starts |
| Map-reduce | Parallelizable subtasks (e.g., batch analysis) | Fan-out cost, result merging complexity |
State management
State design is the most underrated part of a multi-agent system. We learned this the hard way: pass too much state and you overflow the context window; pass too little and agents make bad decisions.
Typed state with reducers
from langgraph.graph import MessagesState
from typing import TypedDict, Annotated
from langgraph.graph.message import add_messages
import operator
class ResearchState(TypedDict):
messages: Annotated[list, add_messages]
research_notes: Annotated[list[str], operator.add]
current_phase: str
confidence_score: float
sources: Annotated[list[dict], operator.add]
error_log: Annotated[list[str], operator.add]
iteration_count: int
def researcher(state: ResearchState) -> dict:
"""Researcher agent that accumulates findings into typed state."""
notes = state.get("research_notes", [])
confidence = state.get("confidence_score", 0.0)
result = research_agent.invoke({
"messages": state["messages"],
"existing_research": "\n".join(notes),
})
new_notes = extract_notes(result)
new_confidence = evaluate_confidence(notes + new_notes)
return {
"messages": result["messages"],
"research_notes": new_notes,
"confidence_score": new_confidence,
"iteration_count": state.get("iteration_count", 0) + 1,
}
def should_continue_research(state: ResearchState) -> str:
if state["confidence_score"] >= 0.85:
return "writer"
if state["iteration_count"] >= 3:
return "writer" # move on with what we have
return "researcher"The Annotated type with operator.add is what makes this work. It tells LangGraph to append to these lists instead of replacing them, so each agent's contributions accumulate.
Error recovery
In production, agents fail all the time. Tools return errors, models hallucinate tool calls and rate limits hit at the worst moment. We handle this with three patterns.
1. Retry with backoff at the node level
Wrap individual agent nodes in retry logic instead of restarting the whole graph. With LangGraph's checkpointing, you can resume from the last successful node.
2. Fallback chains
When a primary tool fails, route to an alternative. Each agent has a tool priority list: if vector search fails, it falls back to keyword search, then to cached results.
3. Human-in-the-loop escalation
For high-stakes decisions, we use LangGraph's interrupt to pause execution and wait for human input.
Tool routing and specialization
A common mistake is giving every agent access to every tool. Models struggle to choose from large tool sets, and performance drops significantly. We scope each agent's tools tightly to its role.
Our rule of thumb: no agent should have more than 5 tools. If an agent needs more, it is doing too much and should be split.
Dynamic tool selection
Agents that work across several domains get their tools per task instead of all upfront. A lightweight classifier looks at the user query and attaches only the relevant tools before the agent runs.
Running in production
Observability
Every agent invocation gets a trace. We instrument with LangSmith and tag each trace with the graph name, the run ID and the node that produced it. When something goes wrong, we can replay the exact sequence of agent decisions.
Latency budgets
Multi-agent systems are slower than single agents. We set a latency budget per node and stream output so users see progress while the agents work. A typical supervisor graph with three workers takes 8-15 seconds end to end. That is acceptable for async workflows and painful for chat.
Cost control
Every agent call costs money. We track token usage per node and alert when a single graph run goes over a cost threshold. Map-reduce graphs are the riskiest here: fanning out to 20 parallel agents can burn through a budget quickly.
| Metric | Our target | Why it matters |
|---|---|---|
| P95 latency | < 20s | User patience threshold for async tasks |
| Cost per run | < $0.15 | Sustainable at 10k daily runs |
| Success rate | > 95% | Below this, users lose trust |
| Max steps | 25 | Prevents runaway loops |
| Tool error rate | < 5% | Signal to fix tool reliability |
Lessons from 18 months in production
We have run multi-agent systems on several client projects. The main lessons:
- Start with a single agent and split only when you hit a wall. Most tasks do not need multi-agent orchestration.
- The state schema is your most important design decision. Get it wrong and every agent suffers.
- Deterministic edges beat LLM routing for known workflows. When the logic is clear, use conditional edges based on state instead of another LLM call.
- Test agents in isolation before composing them. Each worker should have its own eval suite.
Written by the MiraiTech team in Tokyo
If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.