Skip to content
MiraiTech
All work

Case study · AI

Search over company documents

Employees ask a question and get an answer in about two seconds, with links to the exact pages it came from. Behind it is hybrid retrieval with reranking, and a test suite that blocks any deploy that lowers accuracy.

Year
2025
Duration
4 months
Type
AI
Status
In production
Search over company documents, main screen

Results

  • 92% F1 score
  • 2.1 s response time
  • 10k queries a day at $0.02 each

The problem

Companies with 500+ page handbooks, scattered Notion wikis and old PDF archives have no quick way to get a correct answer. Employees spend 30+ minutes looking for a policy, and new hires take weeks to get up to speed. Generic ChatGPT setups hallucinate on 30%+ of questions and show no sources.

What we built

We built a RAG pipeline that combines BM25 keyword search, dense vector retrieval and ColBERT reranking. It reaches 92% F1. Documents are parsed with Unstructured.io, with LlamaParse for tables, then embedded with Sentence Transformers and stored in pgvector. A DeepEval check blocks any deploy that would take accuracy below 92%.

Hybrid retrieval
BM25 keyword search plus dense vector search, with MMR for diversity, then ColBERT reranking. On domain-specific queries it beats pure semantic search by 18%.
Deploys gated on evals
CI runs the DeepEval suites on every push. If F1 drops below 0.92, the deploy is blocked and the regression stays out of production.
Streaming answers with citations
PydanticAI structures each answer with inline citations to specific page numbers. Users can check an answer against the source document in one click.
Drag-and-drop ingestion
Users upload PDFs, Notion exports and Google Drive files. Chunking, table extraction and embedding run in the background, with progress shown in the UI.
Cost dashboard
Live per-query cost ($0.02 on average), token usage and retrieval latency.

How it works

Hybrid search and reranking, with every deploy gated on eval scores.

  1. 01

    Ingestion

    Upload → Unstructured.io (PDF parsing) + LlamaParse (tables) → sentence chunking

  2. 02

    Embedding

    Sentence Transformers (all-MiniLM-L6-v2) → pgvector, plus a BM25 index

  3. 03

    Retrieval

    Query → hybrid search (dense + BM25 + MMR) → top 20 candidates

  4. 04

    Reranking

    ColBERT scores the top 20 → 5 most relevant passages

  5. 05

    Synthesis

    Claude 3.5 Sonnet writes the answer, with citations structured by PydanticAI

  6. 06

    Delivery

    FastAPI streams over SSE → Next.js frontend with inline source links

Screens

Search over company documents, screen 1
1 / 2

Decisions

pgvector + BM25 hybridData
Using pgvector rather than Pinecone kept retrieval and structured data in one database. We added pg_trgm for keyword matching, and on technical documentation the hybrid search was 18% more accurate than vector search alone.
ColBERT rerankerAI / ML
ColBERT was 3x faster than a cross-encoder at similar accuracy. Late interaction keeps token-level matching without the latency of full cross-attention.
FastAPI + streamingBackend
We chose FastAPI over Django because async streaming is built in. Server-Sent Events push tokens to the browser as they are generated, which feels like ChatGPT without the extra work of running WebSockets.
DeepEvalInfrastructure
DeepEval over RAGAS because its test runner fits into CI. Eval suites are pytest fixtures, so blocking a deploy on an accuracy regression needed no custom infrastructure.

What went wrong

Tables mangled by PDF parsing

Problem
Standard PDF parsers flattened tables, so financial and policy tables turned into meaningless text chunks. Table data was involved in 35% of handbook queries.
Fix
We send table-heavy pages through LlamaParse, which rebuilds the table structure with a vision model. A table-aware chunker keeps row and column relationships intact in the embeddings.
Result
Accuracy on table questions went from 61% to 89%. The finance team said the system finally "understood" the compensation tables.

Slow retrieval at 50k chunks

Problem
With 50k+ document chunks, a hybrid search query took 4.2 seconds. Users of an interactive search tool expect an answer in under 2 seconds.
Fix
We added an HNSW index in pgvector (ef_construction=200, m=16) and a Redis cache for repeated questions. The cache ended up serving 40% of production traffic.
Result
P95 retrieval latency fell from 4.2s to 380ms. Cached queries return in under 50ms.

Team and stack

Hiroshi designed the retrieval pipeline and built the streaming frontend. Kenta built reranking and the DeepEval gate in CI, and got the cost down to $0.02 a query. Hiroki built document ingestion and the pgvector indexes.

  • Next.js 15
  • FastAPI
  • pgvector
  • ColBERT
  • Claude 3.5 Sonnet
  • PydanticAI
  • LlamaParse
  • DeepEval