Skip to content
MiraiTech
All writing
AI engineering

· 10 min read

How we structure multi-agent systems in LangGraph

Why we moved from single agents to multi-agent graphs, and how we handle state, error recovery and tool routing after 18 months in production.

Where single agents break down

When we started building LLM applications at MiraiTech, a single ReAct agent looked like enough: give it tools and a system prompt and let it reason. That stopped working once production traffic arrived.

A single agent hits a complexity ceiling. Past 5-7 tools, its accuracy drops. The context window fills with irrelevant tool output, and error recovery mostly means hoping the next attempt works. So we moved to multi-agent orchestration.

LangGraph lets us build these systems as stateful, observable graphs with explicit control flow.

Architecture patterns

Supervisor architecture

A central supervisor agent delegates work to specialized worker agents. Of the patterns here, it is the easiest to reason about and to debug.

supervisor.py
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.types import Command
from langchain_openai import ChatOpenAI
from typing import Literal
 
llm = ChatOpenAI(model="gpt-4o")
 
def supervisor(state: MessagesState) -> Command[Literal["researcher", "writer", "reviewer", END]]:
    """Central orchestrator that routes to specialized agents."""
    system_prompt = """You are a supervisor managing three workers:
    - researcher: finds and validates technical information
    - writer: produces polished content from research
    - reviewer: checks accuracy and suggests improvements
    
    Based on the conversation, decide which worker should act next.
    If the task is complete, respond with FINISH."""
    
    response = llm.with_structured_output(RouterOutput).invoke(
        [{"role": "system", "content": system_prompt}] + state["messages"]
    )
    
    if response.next == "FINISH":
        return Command(goto=END)
    
    return Command(goto=response.next)
 
def make_agent(name: str, tools: list, system_prompt: str):
    """Factory for building worker agents with consistent structure."""
    agent = create_react_agent(llm, tools, prompt=system_prompt)
    
    def node(state: MessagesState) -> Command[Literal["supervisor"]]:
        result = agent.invoke(state)
        return Command(
            update={"messages": result["messages"]},
            goto="supervisor",
        )
    return node
 
graph = StateGraph(MessagesState)
graph.add_node("supervisor", supervisor)
graph.add_node("researcher", make_agent("researcher", research_tools, research_prompt))
graph.add_node("writer", make_agent("writer", writing_tools, writer_prompt))
graph.add_node("reviewer", make_agent("reviewer", review_tools, reviewer_prompt))
graph.add_edge(START, "supervisor")
 
app = graph.compile()

Peer-to-peer architecture

Agents talk to each other directly, and each one decides who to hand off to next. This suits dynamic workflows where the best path cannot be predicted.

Choosing a pattern

PatternBest forDownsides
SupervisorPredictable workflows, clear task decompositionBottleneck at supervisor, higher latency
Peer-to-peerDynamic exploration, creative tasksHarder to debug, loop risk
HierarchicalLarge systems (10+ agents)Complex state management, slow cold starts
Map-reduceParallelizable subtasks (e.g., batch analysis)Fan-out cost, result merging complexity

State management

State design is the most underrated part of a multi-agent system. We learned this the hard way: pass too much state and you overflow the context window; pass too little and agents make bad decisions.

Typed state with reducers

state.py
from langgraph.graph import MessagesState
from typing import TypedDict, Annotated
from langgraph.graph.message import add_messages
import operator
 
class ResearchState(TypedDict):
    messages: Annotated[list, add_messages]
    research_notes: Annotated[list[str], operator.add]
    current_phase: str
    confidence_score: float
    sources: Annotated[list[dict], operator.add]
    error_log: Annotated[list[str], operator.add]
    iteration_count: int
 
def researcher(state: ResearchState) -> dict:
    """Researcher agent that accumulates findings into typed state."""
    notes = state.get("research_notes", [])
    confidence = state.get("confidence_score", 0.0)
    
    result = research_agent.invoke({
        "messages": state["messages"],
        "existing_research": "\n".join(notes),
    })
    
    new_notes = extract_notes(result)
    new_confidence = evaluate_confidence(notes + new_notes)
    
    return {
        "messages": result["messages"],
        "research_notes": new_notes,
        "confidence_score": new_confidence,
        "iteration_count": state.get("iteration_count", 0) + 1,
    }
 
def should_continue_research(state: ResearchState) -> str:
    if state["confidence_score"] >= 0.85:
        return "writer"
    if state["iteration_count"] >= 3:
        return "writer"  # move on with what we have
    return "researcher"

The Annotated type with operator.add is what makes this work. It tells LangGraph to append to these lists instead of replacing them, so each agent's contributions accumulate.

Error recovery

In production, agents fail all the time. Tools return errors, models hallucinate tool calls and rate limits hit at the worst moment. We handle this with three patterns.

1. Retry with backoff at the node level

Wrap individual agent nodes in retry logic instead of restarting the whole graph. With LangGraph's checkpointing, you can resume from the last successful node.

2. Fallback chains

When a primary tool fails, route to an alternative. Each agent has a tool priority list: if vector search fails, it falls back to keyword search, then to cached results.

3. Human-in-the-loop escalation

For high-stakes decisions, we use LangGraph's interrupt to pause execution and wait for human input.

Tool routing and specialization

A common mistake is giving every agent access to every tool. Models struggle to choose from large tool sets, and performance drops significantly. We scope each agent's tools tightly to its role.

Our rule of thumb: no agent should have more than 5 tools. If an agent needs more, it is doing too much and should be split.

Dynamic tool selection

Agents that work across several domains get their tools per task instead of all upfront. A lightweight classifier looks at the user query and attaches only the relevant tools before the agent runs.

Running in production

Observability

Every agent invocation gets a trace. We instrument with LangSmith and tag each trace with the graph name, the run ID and the node that produced it. When something goes wrong, we can replay the exact sequence of agent decisions.

Latency budgets

Multi-agent systems are slower than single agents. We set a latency budget per node and stream output so users see progress while the agents work. A typical supervisor graph with three workers takes 8-15 seconds end to end. That is acceptable for async workflows and painful for chat.

Cost control

Every agent call costs money. We track token usage per node and alert when a single graph run goes over a cost threshold. Map-reduce graphs are the riskiest here: fanning out to 20 parallel agents can burn through a budget quickly.

MetricOur targetWhy it matters
P95 latency< 20sUser patience threshold for async tasks
Cost per run< $0.15Sustainable at 10k daily runs
Success rate> 95%Below this, users lose trust
Max steps25Prevents runaway loops
Tool error rate< 5%Signal to fix tool reliability

Lessons from 18 months in production

We have run multi-agent systems on several client projects. The main lessons:

  1. Start with a single agent and split only when you hit a wall. Most tasks do not need multi-agent orchestration.
  2. The state schema is your most important design decision. Get it wrong and every agent suffers.
  3. Deterministic edges beat LLM routing for known workflows. When the logic is clear, use conditional edges based on state instead of another LLM call.
  4. Test agents in isolation before composing them. Each worker should have its own eval suite.

Written by the MiraiTech team in Tokyo

If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.