Case study · AI
Search over company documents
Employees ask a question and get an answer in about two seconds, with links to the exact pages it came from. Behind it is hybrid retrieval with reranking, and a test suite that blocks any deploy that lowers accuracy.
- Year
- 2025
- Duration
- 4 months
- Type
- AI
- Status
- In production

Results
- 92% F1 score
- 2.1 s response time
- 10k queries a day at $0.02 each
The problem
Companies with 500+ page handbooks, scattered Notion wikis and old PDF archives have no quick way to get a correct answer. Employees spend 30+ minutes looking for a policy, and new hires take weeks to get up to speed. Generic ChatGPT setups hallucinate on 30%+ of questions and show no sources.
What we built
We built a RAG pipeline that combines BM25 keyword search, dense vector retrieval and ColBERT reranking. It reaches 92% F1. Documents are parsed with Unstructured.io, with LlamaParse for tables, then embedded with Sentence Transformers and stored in pgvector. A DeepEval check blocks any deploy that would take accuracy below 92%.
- Hybrid retrieval
- BM25 keyword search plus dense vector search, with MMR for diversity, then ColBERT reranking. On domain-specific queries it beats pure semantic search by 18%.
- Deploys gated on evals
- CI runs the DeepEval suites on every push. If F1 drops below 0.92, the deploy is blocked and the regression stays out of production.
- Streaming answers with citations
- PydanticAI structures each answer with inline citations to specific page numbers. Users can check an answer against the source document in one click.
- Drag-and-drop ingestion
- Users upload PDFs, Notion exports and Google Drive files. Chunking, table extraction and embedding run in the background, with progress shown in the UI.
- Cost dashboard
- Live per-query cost ($0.02 on average), token usage and retrieval latency.
How it works
Hybrid search and reranking, with every deploy gated on eval scores.
- 01
Ingestion
Upload → Unstructured.io (PDF parsing) + LlamaParse (tables) → sentence chunking
- 02
Embedding
Sentence Transformers (all-MiniLM-L6-v2) → pgvector, plus a BM25 index
- 03
Retrieval
Query → hybrid search (dense + BM25 + MMR) → top 20 candidates
- 04
Reranking
ColBERT scores the top 20 → 5 most relevant passages
- 05
Synthesis
Claude 3.5 Sonnet writes the answer, with citations structured by PydanticAI
- 06
Delivery
FastAPI streams over SSE → Next.js frontend with inline source links
Screens
Decisions
- pgvector + BM25 hybridData
- Using pgvector rather than Pinecone kept retrieval and structured data in one database. We added pg_trgm for keyword matching, and on technical documentation the hybrid search was 18% more accurate than vector search alone.
- ColBERT rerankerAI / ML
- ColBERT was 3x faster than a cross-encoder at similar accuracy. Late interaction keeps token-level matching without the latency of full cross-attention.
- FastAPI + streamingBackend
- We chose FastAPI over Django because async streaming is built in. Server-Sent Events push tokens to the browser as they are generated, which feels like ChatGPT without the extra work of running WebSockets.
- DeepEvalInfrastructure
- DeepEval over RAGAS because its test runner fits into CI. Eval suites are pytest fixtures, so blocking a deploy on an accuracy regression needed no custom infrastructure.
What went wrong
Tables mangled by PDF parsing
- Problem
- Standard PDF parsers flattened tables, so financial and policy tables turned into meaningless text chunks. Table data was involved in 35% of handbook queries.
- Fix
- We send table-heavy pages through LlamaParse, which rebuilds the table structure with a vision model. A table-aware chunker keeps row and column relationships intact in the embeddings.
- Result
- Accuracy on table questions went from 61% to 89%. The finance team said the system finally "understood" the compensation tables.
Slow retrieval at 50k chunks
- Problem
- With 50k+ document chunks, a hybrid search query took 4.2 seconds. Users of an interactive search tool expect an answer in under 2 seconds.
- Fix
- We added an HNSW index in pgvector (ef_construction=200, m=16) and a Redis cache for repeated questions. The cache ended up serving 40% of production traffic.
- Result
- P95 retrieval latency fell from 4.2s to 380ms. Cached queries return in under 50ms.
Team and stack
Hiroshi designed the retrieval pipeline and built the streaming frontend. Kenta built reranking and the DeepEval gate in CI, and got the cost down to $0.02 a query. Hiroki built document ingestion and the pgvector indexes.
- Next.js 15
- FastAPI
- pgvector
- ColBERT
- Claude 3.5 Sonnet
- PydanticAI
- LlamaParse
- DeepEval