Skip to content
MiraiTech
All writing
AI engineering

· 12 min read

Notes on running RAG in production

What we learned running RAG for thousands of queries a day: hybrid retrieval, chunking messy documents, evaluation in CI, and the metrics we watch.

Most tutorials build a RAG system in 20 lines of code: load some documents, chunk them, embed them and query an LLM. That gets you a demo. Production takes a lot more.

We have built several RAG pipelines that now handle thousands of queries a day. This is what we learned about retrieval, evaluation and running them.

How RAG systems fail in production

These are the failure modes we have seen repeatedly:

  • Retrieval degrades quietly. The system returns plausible but wrong answers, and nobody notices until a customer complains.
  • Chunking breaks on real documents. PDFs with tables, mixed-language content and nested headers do not split cleanly.
  • Embeddings drift. As new documents are added, the semantic space shifts and older queries get worse results.
  • There is no evaluation pipeline. Without systematic measurement, the three problems above stay invisible.

Hybrid retrieval

Vector similarity search on its own struggles with exact matches, specific identifiers and structured queries. We handle this with hybrid retrieval, which runs dense vector search and sparse keyword matching side by side.

Dense and sparse pipeline

This is the retrieval step we run in production:

retrieval/hybrid.ts
interface RetrievalResult {
  content: string
  score: number
  source: 'dense' | 'sparse' | 'fused'
  metadata: DocumentMetadata
}
 
async function hybridRetrieve(
  query: string,
  options: RetrievalOptions
): Promise<RetrievalResult[]> {
  // Run dense and sparse retrieval in parallel
  const [denseResults, sparseResults] = await Promise.all([
    vectorStore.similaritySearch(query, options.topK),
    bm25Index.search(query, options.topK),
  ])
 
  // Reciprocal Rank Fusion
  return reciprocalRankFusion(denseResults, sparseResults, {
    k: 60, // RRF constant
    weights: { dense: 0.6, sparse: 0.4 },
  })
}

The two result lists are merged with Reciprocal Rank Fusion (RRF). Normalizing scores across different retrieval methods is unreliable, so RRF merges by rank position instead:

retrieval/rrf.py
def reciprocal_rank_fusion(
    results: list[list[Document]],
    k: int = 60,
    weights: list[float] | None = None,
) -> list[Document]:
    """Merge ranked lists using Reciprocal Rank Fusion."""
    if weights is None:
        weights = [1.0] * len(results)
 
    scores: dict[str, float] = {}
 
    for weight, ranked_list in zip(weights, results):
        for rank, doc in enumerate(ranked_list):
            if doc.id not in scores:
                scores[doc.id] = 0
            scores[doc.id] += weight * (1.0 / (k + rank + 1))
 
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)
ScenarioVector searchHybrid search
Exact ID lookup ('order #12345')Poor (embeds semantically)Excellent (BM25 exact match)
Conceptual questionsExcellent (semantic similarity)Excellent (the dense side handles these)
Acronyms and jargonInconsistent (depends on training)Good (sparse catches exact terms)
Multi-language queriesGood with a multilingual modelBetter (the two signals complement each other)
Latency (p99)~50ms~80ms (parallel, bounded by the slower one)

Chunking real documents

The default "split by N tokens with M overlap" approach falls apart quickly on real documents. What has worked better for us:

undefined

Semantic chunking

Split at topic boundaries instead of at a fixed size. Embed each sentence and start a new chunk where the cosine similarity between adjacent sentences drops below a threshold.

undefined

Hierarchical chunking

Keep the document structure. Each chunk should know its parent section, the document title and its position in the hierarchy. The LLM needs that context to answer accurately.

undefined

Metadata enrichment

Before storing chunks, add extracted metadata: dates, entities, categories and a generated summary. That lets you filter before retrieval, which improves relevance.

Evaluation-driven development

The most important thing we learned: build the evaluation pipeline before the RAG pipeline. Without numbers, every change is a guess.

The eval framework

We evaluate at three layers:

eval/framework.ts
interface RAGEvalSuite {
  // Layer 1: Retrieval quality
  retrieval: {
    metrics: ['recall@k', 'precision@k', 'mrr', 'ndcg']
    dataset: RetrievalEvalDataset  // query → relevant_doc_ids
  }
 
  // Layer 2: Generation quality
  generation: {
    metrics: ['faithfulness', 'relevance', 'completeness']
    judge: 'gpt-4' | 'claude-3.5'  // LLM-as-judge
  }
 
  // Layer 3: End-to-end
  e2e: {
    metrics: ['answer_correctness', 'hallucination_rate']
    dataset: E2EEvalDataset  // query → expected_answer
    threshold: { correctness: 0.85, hallucination: 0.05 }
  }
}

CI/CD integration

Every PR that touches the retrieval pipeline runs the eval suite:

.github/workflows/rag-eval.yml
- name: Run RAG Evaluation
  run: |
    python -m pytest tests/eval/ \
      --eval-dataset=datasets/golden_qa.json \
      --min-retrieval-recall=0.90 \
      --max-hallucination-rate=0.05 \
      --report=eval-report.json
 
- name: Comment PR with Results
  uses: actions/github-script@v7
  with:
    script: |
      const report = require('./eval-report.json')
      // Post metrics comparison to PR

Production monitoring

Once the system is live, we track these metrics:

MetricTargetAlert threshold
Retrieval recall@10> 0.90< 0.85
Answer faithfulness> 0.92< 0.88
Hallucination rate< 0.05> 0.08
p95 latency< 2s> 3s
Token cost per query< $0.03> $0.05

Where to start

Written by the MiraiTech team in Tokyo

If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.