Self-hosted RAG that actually answers
For chatting with your own documents on your own hardware, start with AnythingLLM if you want one container running today, RAGFlow if your PDFs are hostile, or Onyx if a whole team needs it. The platform choice is the easy part. What separates a stack that answers from one that disappoints is parsing, chunking, and hybrid search, and this page spends its time there.
Updated July 11, 2026
The short answer
| You want | Run this |
|---|---|
| Chat with your documents today, one container |  AnythingLLM |
| Ugly PDFs: tables, scans, multi-column layouts |  RAGFlow |
| A company knowledge base with permissions |  Onyx |
| RAG built into your own application |  LlamaIndex+ Qdrant |
| Questions that span many documents at once |  LightRAG |
The turnkey platforms
All four bundle ingestion, a vector store, and a chat interface, and all four speak to local models through Ollama or any OpenAI-compatible runner, so they pair with whatever your hardware already serves. What separates them is how much operational weight they carry and what they do to a difficult document.
A gentle starting point for a personal document-chat setup. Test it against the documents and question patterns that matter to you.
A choice for documents with tables, scans, and multi-column layouts. Plan the deployment around its documented services.
A choice for a company knowledge base with connectors and permission-aware retrieval. Confirm each access-control requirement in your deployment.
A graph-based retrieval option for questions that span related documents. Evaluate its indexing and operational trade-offs against your corpus.
Parsing is the real project
Every disappointing RAG deployment traces back to the same place: the retrieval index is full of garbage because the PDFs went in through a naive text extractor. Tables collapse into word soup, multi-column layouts interleave, scanned pages contribute nothing at all. The model then hallucinates confidently on top of the wreckage, and the vector database takes the blame.
Use a practical playbook. Test the parser on your ugliest documents first, not your cleanest. Use a layout-aware tool for anything with structure, and keep an OCR fallback for scans. Chunking is the second half of the same problem: naive fixed-size splitting cuts sentences and tables in half, and the difference between that and structure-aware chunking with headings preserved can materially improve what retrieval receives.
A document parser worth testing on your hardest files before committing to a platform. Retrieval quality starts with usable extracted structure.
A framework for custom ingestion and retrieval pipelines when a turnkey platform no longer fits your requirements.
The two retrieval upgrades that matter
Pure vector search is where most stacks start and where the disappointment comes from: embeddings are great at meaning and bad at exactness, so acronyms, product codes, and names can slip through. The first upgrade to evaluate is hybrid search: keyword retrieval fused with vector results.
The second upgrade is a reranker: a small cross-encoder that rescores the retrieved chunks before they reach the model. Evaluate it alongside hybrid retrieval before concluding that your documents, embedder, or self-hosting itself is the problem.
Do you even need a vector database?
Choose based on deployment and retrieval needs rather than a fixed chunk count: use an embedded index for local experiments, pgvector when Postgres is already a good operational fit, and a dedicated vector engine when its filtering, hybrid retrieval, or deployment model fits your workload.
| Database | Sweet spot | Memory planning | Hybrid search |
|---|---|---|---|
pgvector The default if you already run Postgres | Existing Postgres deployments | Measure with your index | Via tsvector or ParadeDB |
Qdrant Single Docker container, strong filtering | Filtering and hybrid retrieval | Measure with your index | Native dense + sparse |
Chroma pip install, fine until concurrency matters | Prototypes and small deployments | Measure with your index | Built-in BM25 |
Milvus Distributed; brings etcd and real ops work | Distributed deployments | Measure with your index | Native BM25 |
Index memory depends on embeddings, index settings, filtering, and the workload. Benchmark with representative data before sizing infrastructure. The two dedicated engines get a full head-to-head in Qdrant vs Milvus.
When hosted is honestly enough
Hosted tools can be enough for a bounded research project with no sensitive material. Compare their current limits, retention policies, and workflow fit with the operational work of running your own stack.
The cases that genuinely require your own stack are the same ones as for replacing ChatGPT itself: documents that cannot leave your network, corpora past the hosted caps, and pipelines you need to shape (your own chunker, your own reranker, retrieval wired into an application). Hardware-wise the retrieval layer is the cheap part; if the generation model is also local, size the machine for it using the 8 GB VRAM guide, or see what runs with no GPU at all, where an ingestion pipeline is a natural overnight batch job.
Questions people actually ask
How much hardware does self-hosted RAG need?
Plan separately for the platform, document processing, vector index, embedding model, and generation model. Retrieval can run on CPU for many workloads, while a local generation model often drives the hardware decision. Measure the full pipeline with representative documents before committing to a host.
Do I need a GPU?
Not necessarily. Embedding and search can run on CPU, while document processing and a local generation model may benefit from acceleration. If answers come from a local model, size the machine for that model first.
Why are my answers worse than NotebookLM?
Often retrieval, not the model. Pure vector search can miss acronyms, part numbers, and exact phrases. Test hybrid retrieval and reranking with a representative evaluation set before changing models or abandoning the stack.
Which embedding model should I pick?
Choose an embedding model that matches your languages, retrieval approach, and operating constraints, then evaluate it on your own queries. Changing embedders requires re-embedding and re-indexing the corpus, so make that choice deliberately.
Sources
If audio recordings are part of your document pipeline, transcribe them before ingestion with the self-hosted speech-to-text guide.
Every project above is one row in the Awesome Open Source AI registry, which resyncs twice a day from the curated GitHub list. The Retrieval-Augmented Generation & Knowledge category holds the rest of the pipeline: rerankers, evaluation harnesses, and the vector databases from the table.
