Case study · AI
Bilingual support agent
The agent replies in whichever language the customer writes in. It answers from the retailer's policy and product documents and cites them. When it cannot resolve a case, it hands over to a person with a structured summary instead of a raw transcript.
- Year
- 2025
- Duration
- 4 months
- Type
- AI
- Status
- In production

Results
- 71% of tier-1 tickets resolved without staff
- 1.9 s median time to first token
- Handoffs in either language with nothing retyped
The problem
A Japanese retailer with a growing overseas customer base had one small team answering LINE and web chat in two languages. Tier-1 questions (order status, returns, size guides, store hours) made up about 70% of volume, and every one still needed a person. English tickets waited hours longer than Japanese ones. Earlier chatbots had failed because they answered confidently from stale FAQ pages and lost the context when a ticket was handed to staff.
What we built
We built an agent that detects the language of each message, answers from a versioned knowledge base of policies and product data with citations, and takes real actions such as order lookups and returns through typed tools. When confidence is low or the customer asks for a person, it hands off to Zendesk with the language, the intent, what has already been checked and a suggested reply. Staff never have to ask the customer to repeat themselves.
- Language detection per message
- Each message is classified as JP or EN on its own, so customers who switch languages get a reply in the one they just used. Tone presets cover Japanese keigo and casual English.
- Answers with citations
- Hybrid pgvector and BM25 retrieval over policy documents and the product catalog. Every answer links to its source paragraph, and a nightly freshness check flags stale documents.
- Typed action tools
- Order status, returns and store hours run as Pydantic-validated tools against the retailer's APIs. Idempotency keys make sure a retried message never creates a second return.
- Handoff to staff
- Escalations arrive in Zendesk with the language, intent, verified facts and a draft reply. Staff accept, edit or reject the draft, and rejections go into the eval set.
- Agent console
- A Next.js console shows live conversations, retrieval traces and confidence scores. The support lead can pause the agent on any topic with one click.
How it works
One support agent for every channel. It picks the language per message, cites its sources, acts through typed tools and escalates with a structured handoff.
- 01
Channel adapter
LINE, web chat and email messages normalized into one conversation model
- 02
Language and intent
Claude classifies language and intent per message; conversation state stored in Redis
- 03
Retrieve or act
Hybrid pgvector + BM25 retrieval with citations, or typed tool calls to the order and return APIs
- 04
Confidence gate
Low confidence or a request for a person → Zendesk handoff with a draft reply
- 05
Reply and learn
Reply streamed to the channel; staff edits and rejections flow back into the eval set
Screens
Decisions
- LangGraphAI / ML
- The support flow has explicit states (classify, retrieve, act, confirm, escalate), and the agent must never start a return without a confirmation step. With an explicit graph we could prove that. With a free-form ReAct loop we could not.
- Claude Sonnet 4AI / ML
- Of the models we tested on 400 real tickets, it wrote the best Japanese and kept keigo the most consistent. It was also the most reliable at declining to answer when retrieval found nothing relevant.
- pgvectorData
- The knowledge base is under 50k chunks and already sat next to the order data in Postgres. Keeping retrieval in the same database made citations, versioning and the nightly freshness checks simpler.
- LINE Messaging APIBackend
- LINE is the main support channel for Japanese customers. The channel adapter lets one agent graph serve LINE, web chat and email with no channel-specific prompts.
What went wrong
Customers who switch languages
- Problem
- Overseas customers living in Japan often mixed languages in one conversation: a Japanese product name inside an English sentence, or a switch to Japanese for an address. Detecting the language once per conversation produced a reply in the wrong language about 18% of the time.
- Fix
- We moved detection to each message, with a short rolling context, and made the reply follow the language of the customer's latest message. We also built an eval set of 120 mixed-language exchanges from real tickets.
- Result
- Wrong-language replies fell from 18% to under 2%. Satisfaction on English tickets matched Japanese tickets for the first time.
Citing outdated policies
- Problem
- The retailer updated its return policy every season, but old PDF versions stayed in the document store. The agent quoted them accurately and still gave wrong answers, because the cited document had been superseded.
- Fix
- We added versioning with effective dates to the ingestion pipeline. A nightly job flags chunks whose source has a newer version, and retrieval only returns policy that is currently in effect. Superseded chunks are kept for audit but never cited.
- Result
- Policy-related escalations fell 64% the following quarter. The support lead can see which policy version answered any ticket.
Team and stack
Hiroshi led the architecture, designed the LangGraph conversation graph and built the web chat UI and agent console. Kenta built the retrieval and escalation services and the LINE adapter. Hiroki built knowledge-base ingestion and versioning.
- LangGraph
- Claude Sonnet 4
- Next.js 15
- FastAPI
- pgvector
- LINE Messaging API
- Redis
- Zendesk API