AI / SEARCH
Research Paper RAG System
A CPU-optimised research-paper RAG API with FastEmbed, BM25 + RRF fusion, cross-encoder reranking, shared-vector multi-user access, and conversational query rewriting.
Overview
A production-ready research-paper RAG system that goes beyond vector-only tutorials: Qdrant retrieval is followed by BM25 keyword ranking, Reciprocal Rank Fusion, and a MiniLM cross-encoder — all on CPU (~400ms added latency). Multi-user uploads share one embedding set via a user_ids payload model, concurrent ingest uses a DB conditional lock, and follow-up questions are rewritten into standalone queries before retrieval.
The Problem
Closest-in-vector-space is not best-answer. Research papers need exact term matching for acronyms and notation, multi-user sharing without duplicate embeddings, safe deletion that does not break other users, and conversational follow-ups that still retrieve the right chunks.
Architecture
Key Decisions
3-stage CPU reranker (BM25 → RRF → Cross-Encoder)
Wide recall (top-30) then precision funnel to top-5 keeps the LLM focused and avoids lost-in-the-middle — without requiring a GPU.
Shared vectors with user_ids membership
One PDF → one embedding set; users are appended to payload user_ids. Delete is access revocation; vectors are physically removed only when the list is empty.
Query rewriting before retrieval
A small/cheap LLM rewrites follow-ups ("why is it useful?") into standalone questions for embedding, while the original question is still passed to the main LLM.
DB conditional lock for concurrent upload
pending → processing conditional UPDATE acts as a distributed lock so two users uploading the same PDF cannot both embed it.
Failure Modes
Irrelevant vector-only retrieval
BM25 + RRF + cross-encoder catch exact terms and "right words, wrong answer" cases vectors miss.
Duplicate embeddings on concurrent upload
Only one worker wins the pending→processing lock; losers poll until embedded, then grant access.
Delete breaking another user's shared doc
Deletion strips one user_id; the point stays until zero owners remain.
Vague conversational follow-ups
Query rewriting resolves pronouns against chat history before the embed + search step.
Implementation
- PyMuPDF ingestion + LlamaIndex semantic chunking
- FastEmbed (ONNX) embeddings into Qdrant with payload indexes
- BM25 + Reciprocal Rank Fusion + MiniLM cross-encoder funnel
- SHA-256 document-hash dedup and shared user_ids access model
- Conversational query rewriting with a small LLM
- JWT auth, streaming LLM responses with inline citations + SOURCES block
- Race-safe ingest via conditional DocumentHash status updates
Performance
~400ms
Rerank latency (CPU)
30 → 10 → 5
Retrieval funnel
MiniLM-L-2
Cross-encoder
Lessons
- —RRF beats score normalisation when BM25 and cosine live on incompatible scales.
- —Deletion of shared vectors must be ownership revocation, not a hard delete.
- —Rewrite for retrieval, answer with the original question — do not lose user intent.