Experiment · RAG
RAG evaluation harness
A framework for testing RAG pipelines against question sets with known answers. It compares fixed, semantic and recursive chunking; OpenAI, Cohere and open-source embeddings; and dense, sparse and hybrid retrieval.
- Type
- RAG
- Status
- Live
- Models
- Claude Sonnet, GPT-4o, all-MiniLM-L6
Results
- 12 metrics
- 6 chunking strategies
- 4 embedding models
We tested 72 pipeline configurations (6 chunking x 4 embeddings x 3 retrieval) on 150 questions. The best configuration depends on the domain. Best overall was semantic chunking with OpenAI embeddings and hybrid retrieval (0.92 faithfulness, 0.89 context precision). Best for cost was recursive chunking with BGE-large and hybrid retrieval (0.87 faithfulness, 0.83 context precision, $0 embedding cost).
What we tested
Most RAG tutorials judge quality by reading a few answers. This experiment is an automated evaluation setup that tests each variable in a RAG pipeline (chunking strategy, embedding model, retrieval method and generation model) against ground-truth Q&A datasets, using 12 standard metrics. We wanted RAG decisions to rest on measurements.
Method
We built 3 evaluation datasets (technical documentation, legal contracts and academic papers) with 50 ground-truth Q&A pairs each. On each dataset we tested 6 chunking strategies (fixed 512, fixed 1024, recursive, semantic, sentence-window, parent-document), 4 embedding models (OpenAI text-embedding-3-large, Cohere embed-v3, all-MiniLM-L6, BGE-large) and 3 retrieval methods (dense, sparse BM25, hybrid). Every combination was scored on 12 RAGAS and DeepEval metrics, including faithfulness, answer relevancy, context precision, context recall and hallucination rate.
Findings
- 01
Semantic chunking won on technical docs and lost on legal
Semantic chunking reached 0.91 context precision on technical docs but only 0.74 on legal contracts, where clause boundaries matter more than semantic similarity. Fixed 512 with overlap did best on legal.
- 02
Hybrid retrieval is worth the extra complexity
Dense plus BM25 improved context recall by 8-15% over dense retrieval alone on every dataset. The gain was largest on queries with specific identifiers such as error codes and section numbers.
- 03
OpenAI embeddings led, and BGE-large was 93% as good
text-embedding-3-large scored highest on every metric. BGE-large reached 93% of its performance with no API cost, which is a good trade when cost matters.
- 04
Evaluation has to be domain-specific
The same chunking strategy scored 0.91 on technical docs and 0.74 on legal text. No configuration was best everywhere, so each team needs to evaluate on its own documents.
How it is built
Evaluation runs in three phases. Ingestion: documents go through every chunking strategy in parallel, are embedded with every model and are stored in separate pgvector collections. Retrieval: each query runs against every combination of retrieval method and embedding model, and the top-k results are collected. Evaluation: retrieved contexts and generated answers are scored against the ground truth with DeepEval and RAGAS, and the results are aggregated into a comparison matrix.
DeepEval and RAGAS provide the 12 metrics. pgvector stores embeddings for dense retrieval, and a custom BM25 implementation handles sparse retrieval. A Streamlit dashboard is used to compare results. Python runs the evaluation pipeline, with configurations evaluated in parallel.
- Python
- DeepEval
- RAGAS
- pgvector
- Streamlit
Problems we hit
Metrics that disagreed
RAGAS faithfulness and DeepEval hallucination scores sometimes contradicted each other. When they disagree, a person breaks the tie, and the more reliable metric gets more weight in the composite score.
Embedding cost
Testing 4 embedding models x 150 questions x 6 chunking strategies took 3,600 embedding API calls. We cache embeddings aggressively, so a re-run only embeds new or changed chunks.