· 12 min read
Notes on running RAG in production
What we learned running RAG for thousands of queries a day: hybrid retrieval, chunking messy documents, evaluation in CI, and the metrics we watch.
Most tutorials build a RAG system in 20 lines of code: load some documents, chunk them, embed them and query an LLM. That gets you a demo. Production takes a lot more.
We have built several RAG pipelines that now handle thousands of queries a day. This is what we learned about retrieval, evaluation and running them.
How RAG systems fail in production
These are the failure modes we have seen repeatedly:
- Retrieval degrades quietly. The system returns plausible but wrong answers, and nobody notices until a customer complains.
- Chunking breaks on real documents. PDFs with tables, mixed-language content and nested headers do not split cleanly.
- Embeddings drift. As new documents are added, the semantic space shifts and older queries get worse results.
- There is no evaluation pipeline. Without systematic measurement, the three problems above stay invisible.
Hybrid retrieval
Vector similarity search on its own struggles with exact matches, specific identifiers and structured queries. We handle this with hybrid retrieval, which runs dense vector search and sparse keyword matching side by side.
Dense and sparse pipeline
This is the retrieval step we run in production:
interface RetrievalResult {
content: string
score: number
source: 'dense' | 'sparse' | 'fused'
metadata: DocumentMetadata
}
async function hybridRetrieve(
query: string,
options: RetrievalOptions
): Promise<RetrievalResult[]> {
// Run dense and sparse retrieval in parallel
const [denseResults, sparseResults] = await Promise.all([
vectorStore.similaritySearch(query, options.topK),
bm25Index.search(query, options.topK),
])
// Reciprocal Rank Fusion
return reciprocalRankFusion(denseResults, sparseResults, {
k: 60, // RRF constant
weights: { dense: 0.6, sparse: 0.4 },
})
}The two result lists are merged with Reciprocal Rank Fusion (RRF). Normalizing scores across different retrieval methods is unreliable, so RRF merges by rank position instead:
def reciprocal_rank_fusion(
results: list[list[Document]],
k: int = 60,
weights: list[float] | None = None,
) -> list[Document]:
"""Merge ranked lists using Reciprocal Rank Fusion."""
if weights is None:
weights = [1.0] * len(results)
scores: dict[str, float] = {}
for weight, ranked_list in zip(weights, results):
for rank, doc in enumerate(ranked_list):
if doc.id not in scores:
scores[doc.id] = 0
scores[doc.id] += weight * (1.0 / (k + rank + 1))
return sorted(scores.items(), key=lambda x: x[1], reverse=True)Vector search vs hybrid search
| Scenario | Vector search | Hybrid search |
|---|---|---|
| Exact ID lookup ('order #12345') | Poor (embeds semantically) | Excellent (BM25 exact match) |
| Conceptual questions | Excellent (semantic similarity) | Excellent (the dense side handles these) |
| Acronyms and jargon | Inconsistent (depends on training) | Good (sparse catches exact terms) |
| Multi-language queries | Good with a multilingual model | Better (the two signals complement each other) |
| Latency (p99) | ~50ms | ~80ms (parallel, bounded by the slower one) |
Chunking real documents
The default "split by N tokens with M overlap" approach falls apart quickly on real documents. What has worked better for us:
Semantic chunking
Split at topic boundaries instead of at a fixed size. Embed each sentence and start a new chunk where the cosine similarity between adjacent sentences drops below a threshold.
Hierarchical chunking
Keep the document structure. Each chunk should know its parent section, the document title and its position in the hierarchy. The LLM needs that context to answer accurately.
Metadata enrichment
Before storing chunks, add extracted metadata: dates, entities, categories and a generated summary. That lets you filter before retrieval, which improves relevance.
Evaluation-driven development
The most important thing we learned: build the evaluation pipeline before the RAG pipeline. Without numbers, every change is a guess.
The eval framework
We evaluate at three layers:
interface RAGEvalSuite {
// Layer 1: Retrieval quality
retrieval: {
metrics: ['recall@k', 'precision@k', 'mrr', 'ndcg']
dataset: RetrievalEvalDataset // query → relevant_doc_ids
}
// Layer 2: Generation quality
generation: {
metrics: ['faithfulness', 'relevance', 'completeness']
judge: 'gpt-4' | 'claude-3.5' // LLM-as-judge
}
// Layer 3: End-to-end
e2e: {
metrics: ['answer_correctness', 'hallucination_rate']
dataset: E2EEvalDataset // query → expected_answer
threshold: { correctness: 0.85, hallucination: 0.05 }
}
}CI/CD integration
Every PR that touches the retrieval pipeline runs the eval suite:
- name: Run RAG Evaluation
run: |
python -m pytest tests/eval/ \
--eval-dataset=datasets/golden_qa.json \
--min-retrieval-recall=0.90 \
--max-hallucination-rate=0.05 \
--report=eval-report.json
- name: Comment PR with Results
uses: actions/github-script@v7
with:
script: |
const report = require('./eval-report.json')
// Post metrics comparison to PRProduction monitoring
Once the system is live, we track these metrics:
| Metric | Target | Alert threshold |
|---|---|---|
| Retrieval recall@10 | > 0.90 | < 0.85 |
| Answer faithfulness | > 0.92 | < 0.88 |
| Hallucination rate | < 0.05 | > 0.08 |
| p95 latency | < 2s | > 3s |
| Token cost per query | < $0.03 | > $0.05 |
Where to start
Written by the MiraiTech team in Tokyo
If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.