Private RAG Document Intelligence System
How I built a self-hosted AI knowledge base that lets a researcher query hundreds of complex biological PDFs in plain English.
▢ Key Challenge
Standard RAG pipelines choke on complex scientific PDFs with dense tables, multi-column layouts, and embedded figures. Naive text extraction produces garbage embeddings that destroy retrieval accuracy.
The Problem: Garbage In, Garbage Out
A biology researcher had amassed hundreds of scientific PDFs : research papers, lab manuals and protocol documents, spread across their local machine. Finding specific information meant either memorising paper locations or spending hours manually searching. They wanted to simply ask questions in plain English and get accurate, sourced answers.
They had tried connecting AnythingLLM directly to a folder of PDFs. The results were frustratingly inaccurate. The root cause was the ingestion layer: standard PDF text extractors flatten complex scientific documents, mangling table data, misreading figure captions, and producing nonsensical chunked text that confuses the embedding model. Bad chunks → bad embeddings → bad retrieval → bad answers.
The Architecture
I designed a custom ingestion pipeline that sits between the raw PDFs and the vector store, solving the quality problem before any data reaches the LLM.
The Smart Ingestion Pipeline
The key differentiator is the hi-res PDF partitioning step. Instead of reading a PDF like a flat text file, the system uses OCR and layout analysis to understand the document's visual structure. Tables are preserved as structured data. Headers establish semantic hierarchy. Multi-column layouts are read in the correct reading order.
# Two Docker containers, one command
docker-compose up --build -d
# Services started:
# chroma → ChromaDB on :8000 (vector store)
# rag-backend → FastAPI on :8001 (ingestion + query API)The /query endpoint was a particularly useful debugging tool, it lets you inspect raw vector matches before they reach the LLM, making it easy to verify that chunking quality is actually good before handing off to AnythingLLM.
Deployment on Client's Local Machine
Privacy was non-negotiable the client's biological research data could not touch any external server. The entire system runs locally via Docker Compose. I wrote a detailed deployment guide covering the full setup, including how to configure AnythingLLM to use the local ChromaDB collection as its knowledge source.
Tech Stack at a Glance
| Layer | Technology | Purpose |
|---|---|---|
| Language | Python 3.10 | Core backend |
| API | FastAPI + Uvicorn | REST endpoints for upload & query |
| PDF Processing | Unstructured (hi-res) | OCR + layout-aware partitioning |
| Chunking | LangChain | Semantic text splitting |
| Embeddings | all-MiniLM-L6-v2 | 384-dim sentence embeddings |
| Vector DB | ChromaDB | Persistent local vector store |
| Containers | Docker + Compose | Isolated, reproducible environment |
| Chat UI | AnythingLLM | Client's existing chat interface |
Key Technical Decisions
Results & Impact
Delivered a fully private, containerized RAG pipeline. The client can now ask natural-language questions across hundreds of biological research PDFs and receive accurate, cited answers, all running locally on their own machine with zero cloud dependency.