A Complete End-to-End Guide to Deploying Retrieval-Augmented Generation Entirely On-Premise -- From Ingestion to Citation, With Docker Compose, Hardware Specs, Monitoring, Backup, and Scale Considerations
Deploying Retrieval-Augmented Generation End to End, On Your Own Hardware, Under Your Own Control
A Retrieval-Augmented Generation system is not a single model. It is a pipeline of cooperating services -- document parsers, chunkers, embedding servers, vector databases, metadata stores, re-rankers, language models, citation trackers, and user-facing interfaces -- each running on hardware you control. This guide walks through every stage of that pipeline, explains the sovereign component options at each stage, and shows how to deploy the complete stack on a single machine using Docker Compose. We cover hardware requirements from a CPU-only laptop to a multi-GPU production server, monitoring with Prometheus and Grafana, automated backup of vectors, metadata, and files, and the scaling path from one hundred documents to one million. Every component is open source, runs locally, and keeps data inside your network perimeter. No API keys. No data egress. No vendor lock-in.
A RAG system is a pipeline. A document enters at one end, gets parsed, cleaned, split into chunks, embedded into vectors, and stored. A question enters at the other end, gets embedded, matched against stored vectors, re-ranked, fed with its retrieved context into a language model, and returned to the user with citations. Ten distinct stages connect these two endpoints. Each stage is a separate service or component with its own configuration, its own failure modes, and its own optimization knobs. Understanding the pipeline as a whole -- and where each piece fits -- is the prerequisite for building, debugging, and scaling it.
The diagram below shows the end-to-end data flow. The left half is the indexing path -- documents in, vectors stored. The right half is the query path -- question in, cited answer out. Both paths share the embedding model, the vector database, and the metadata store. The indexing path is asynchronous and batch-oriented. The query path is synchronous and latency-sensitive. Every design decision in this guide ultimately serves one of these two paths.
The indexing path can take seconds to minutes per document depending on size and whether OCR or transcription is involved. The query path must complete in one to five seconds to feel responsive. This latency budget shapes every component choice: embedding models must be fast, retrieval must be sub-100ms, re-ranking must be parallel, and the LLM must stream tokens rather than waiting to emit the full response. The rest of this guide examines each stage in turn, then shows how to deploy them together.
1The ingestion layer is where documents enter the system. It is the front door -- and like any front door, it needs to accept visitors from multiple entrances. A sovereign RAG stack typically accepts documents through five channels: a file-system watcher that picks up files dropped into a monitored directory, a web upload UI for interactive document submission, a REST API endpoint for programmatic ingestion from other applications, an email ingestion path that monitors a mailbox for incoming documents, and a batch import job that loads archives of historical documents. All five channels converge on the same processing queue.
Once a document is in hand, it must be parsed into text. This is where the real work begins, and where the choice of parsing tools determines what the rest of the pipeline can do. Three tools cover virtually every document format you will encounter:
Apache Tika is the workhorse of document parsing. It detects and extracts text and metadata from over a thousand file formats -- PDF, Word, Excel, PowerPoint, OpenDocument, RTF, HTML, XML, EPUB, and dozens more. Tika runs as a Java service and is typically deployed in its own Docker container. For a sovereign stack, the Tika server image (apache/tika) listens on port 9998 and accepts document uploads via HTTP, returning extracted text and metadata as JSON. It handles format detection automatically, so the ingestion orchestrator does not need to know the file type in advance. Tika is the default parser for everything that is not an image, audio, or video file.
Tesseract is the open-source OCR engine. When Tika encounters a scanned PDF -- one where the pages are images, not text -- it returns empty content. The ingestion layer detects this (extracted text length is zero or near-zero) and routes the document to Tesseract. Tesseract converts each page to an image (if not already an image) and runs optical character recognition, producing text. For a sovereign stack, Tesseract runs in a Docker container with the language packs installed. It supports over 100 languages, though accuracy drops for non-Latin scripts without language-specific training data. For handwritten text or complex layouts, Tesseract accuracy may be insufficient -- but for printed documents, it is the standard. GPU acceleration is available but optional; CPU Tesseract is adequate for batch ingestion.
OpenAI's Whisper model, running locally, transcribes audio files to text. For organizations that need to ingest recorded meetings, dictations, voicemails, or customer calls, Whisper is the bridge from audio to the text pipeline. The large-v3 model provides near-human accuracy for English and strong accuracy for many other languages. It runs on GPU -- the large model requires approximately 10 GB VRAM -- and is typically deployed as a containerized service exposing a transcription API. Whisper also performs language identification and timestamping, both of which feed into the processing layer. For a sovereign deployment, Whisper never sends audio to any external service; transcription happens entirely on your hardware.
2Once text is extracted, it must be cleaned, enriched, and made safe before chunking. The processing layer sits between raw extracted text and the chunker, and it performs four critical functions: format detection (confirming what Tika or Tesseract thinks the document is), metadata extraction (pulling author, title, dates, and structural metadata), language detection (identifying the dominant language so the right embedding model and tokenizer are used), and PII detection and redaction (finding and masking personally identifiable information before it enters the vector database).
Metadata extraction is straightforward but important. Every document carries metadata -- author, creation date, modification date, title, subject, keywords, and application-specific fields. This metadata is stored in Postgres alongside the document record and becomes a retrieval filter. Users can ask for documents by date range, by author, by document type. Without metadata extraction, retrieval is limited to pure semantic similarity, which ignores document-level context.
Language detection determines which embedding model to use. A multilingual embedding model like BGE-M3 handles over 100 languages, but if your corpus is monolingual English, a monolingual model like BGE-base-en-v1.5 will outperform it. For mixed-language corpora, the processing layer tags each chunk with its detected language, and the retrieval layer can filter by language at query time.
en_core_web_lg or a fine-tuned Presidio model -- to identify names, dates, addresses, SSNs, phone numbers, email addresses, and medical record numbers. Redaction strategy depends on the use case: replace with tokens ([NAME], [DATE], [MRN]) for compliance, or hash for pseudonymization. Store the mapping between tokens and original values in an encrypted Postgres table accessible only to authorized roles. The redacted text is what flows through the rest of the pipeline. The LLM sees redacted text, the vector database stores redacted embeddings, and only the citation layer -- with its access controls -- can dereference tokens back to original values for authorized users.
3The chunking layer splits processed text into retrievable units. Chunking strategy has a larger impact on retrieval quality than the choice of embedding model -- a well-chunked document with a mediocre embedding model will outperform a poorly chunked document with the best embedding model. The chunking layer is covered in detail in the companion Chunking Strategies for RAG guide. Here we focus on how chunking fits into the infrastructure stack.
The key architectural decision is that chunking strategy is configured per document type, not globally. The chunking layer maintains a routing table that maps document types to chunking configurations. A PDF research paper uses recursive chunking at 512 tokens with 10% overlap. A legal contract uses document-aware chunking that respects clause boundaries. A medical record uses paragraph-aware chunking with small chunk sizes (256 tokens) because clinical notes are dense. An email uses sentence-aware chunking because emails are short and each sentence may be a distinct retrieval target. The orchestrator reads the document type from the metadata extracted in stage 2 and selects the appropriate chunker.
| Document Type | Chunking Strategy | Chunk Size | Overlap |
|---|---|---|---|
| Research papers (PDF) | Recursive | 512 tokens | 50 tokens |
| Legal contracts | Document-aware (clause boundaries) | Variable | 1 sentence |
| Medical records | Paragraph-aware | 256 tokens | 32 tokens |
| Emails | Sentence-aware | Variable | 0 |
| Technical documentation | Document-aware (section headers) | 512 tokens | 50 tokens |
| Transcribed audio | Sentence-aware (timestamp-based) | Variable | 1 sentence |
| Spreadsheets | Row-based (one row per chunk) | Variable | 0 |
| Unknown / mixed | Recursive (fallback) | 512 tokens | 50 tokens |
Each chunk produced by this layer carries its lineage: the document ID it came from, the character offset within the source text, the chunk index within the document, and the chunking strategy that produced it. This lineage is what the citation layer uses later to point a user back to the exact passage in the original document.
4The embedding layer converts text chunks into dense vector representations. These vectors are what the retrieval layer searches over. The embedding model is the heart of the RAG system -- it determines what "similar" means for your corpus. The embedding layer is used twice in the pipeline: once during indexing (to embed chunks for storage) and once during querying (to embed the user's question for retrieval). It must be the same model both times, and it must be fast enough for interactive query latency.
For a sovereign stack, four embedding model families are the primary choices. All run locally, all are open source, and all have been benchmarked extensively on retrieval tasks:
| Model | Dimensions | Max Input | Speed (GPU) | Best For |
|---|---|---|---|---|
| BGE-base-en-v1.5 | 768 | 512 tokens | ~2000 chunks/s | English, general purpose |
| BGE-large-en-v1.5 | 1024 | 512 tokens | ~800 chunks/s | High accuracy English |
| BGE-M3 | 1024 | 8192 tokens | ~500 chunks/s | Multilingual, long documents |
| E5-large-v2 | 1024 | 512 tokens | ~900 chunks/s | Strong zero-shot retrieval |
| nomic-embed-text-v1.5 | 768 | 8192 tokens | ~1500 chunks/s | Long context, open data |
| all-MiniLM-L6-v2 | 384 | 256 tokens | ~5000 chunks/s | Fast, CPU-friendly, prototyping |
| all-mpnet-base-v2 | 768 | 384 tokens | ~1500 chunks/s | Strong sentence similarity |
GPU acceleration is essential. On CPU, embedding a single chunk takes 50-200 ms. On GPU, the same operation takes 0.5-2 ms. For batch indexing of 10,000 chunks, that is the difference between 30 minutes and 30 seconds. The embedding service runs as a Docker container with GPU passthrough (the --gpus all Docker flag) and exposes an HTTP endpoint. Both the indexing orchestrator and the query handler call this endpoint. For indexing, chunks are batched (50-100 at a time) to maximize GPU utilization. For querying, a single embedding is requested and must return in under 50 ms.
Model persistence: The embedding model is loaded once at service startup and kept in GPU memory. Subsequent embedding requests skip model loading entirely, which is why the first request after a restart is slow (3-5 seconds for model load) but all subsequent requests are fast. The embedding service should be configured to pre-load the model on startup rather than lazily on first request, so the first user query does not pay the load penalty.
5The storage layer is a three-layer architecture. Each layer stores a different kind of data and is optimized for a different access pattern. Together, they form the persistent memory of the RAG system.
Qdrant is the primary vector database. It stores the embedding vectors and their associated payloads (metadata about each chunk: document ID, chunk text, source, page number, timestamps). Qdrant supports HNSW (Hierarchical Navigable Small World) indexing for fast approximate nearest neighbor search, with configurable ef_search and ef_construction parameters that trade recall for speed. For a sovereign stack, Qdrant runs as a Docker container with a local volume mount for persistence. It exposes a REST API on port 6333 and a gRPC API on port 6334. Qdrant handles filtering (pre-filter by metadata, then vector search) natively, which is critical for queries like "find chunks about X in documents from 2024." It supports on-disk storage for collections larger than RAM, quantization (scalar and binary) to reduce memory footprint, and snapshot-based backups. For collections under 1 million vectors, a single Qdrant node is sufficient. Beyond that, Qdrant supports distributed deployment with sharding and replication.
PostgreSQL stores everything that is not a vector: document records (title, author, source, ingestion date, document type, file path), chunk metadata (chunk index, character offset, chunking strategy), user data, access control lists, query logs, and system configuration. Crucially, Postgres also provides the sparse retrieval path via its built-in full-text search (tsvector and tsquery with GIN indexes). This is what powers hybrid search -- the retrieval layer queries both Qdrant (dense, semantic) and Postgres (sparse, keyword) and merges the results. Postgres runs as a standard Docker container with a local volume. For larger deployments, consider adding pgvector as a Postgres extension for a unified dense+sparse store, but for separation of concerns and independent scaling, the Qdrant + Postgres split is the cleaner architecture.
The original documents -- PDFs, Word files, audio recordings, images -- are stored on the local filesystem or a network-attached storage (NAS) device. This is the source of truth. The text extracted by Tika, Tesseract, and Whisper is a derivative; the original file is what the citation layer links back to and what the user views when they click a citation. File storage is organized by ingestion date and document ID, with a directory structure that mirrors the Postgres document records. For production, the file store should be on a dedicated SSD volume (for fast access during citation retrieval) and backed up separately (see the Backup Strategy section). For multi-node deployments, a NAS or distributed filesystem (GlusterFS, Ceph) makes the file store accessible to all nodes.
| Storage Layer | Technology | Stores | Access Pattern | Port |
|---|---|---|---|---|
| Vector DB | Qdrant | Embeddings, chunk payloads | ANN search, filtered search | 6333 / 6334 |
| Metadata | PostgreSQL | Doc records, ACLs, FTS index | SQL, full-text search | 5432 |
| Files | Local SSD / NAS | Original documents | Read on citation click | Filesystem |
| Cache (optional) | Redis | Query cache, session state | Key-value, TTL | 6379 |
| Queue | RabbitMQ / Redis | Ingestion jobs, parse tasks | Pub/sub, work queues | 5672 / 6379 |
6The retrieval layer is where the user's question meets the stored knowledge. It takes the query embedding (produced by the same model that embedded the chunks) and searches the vector database for the most similar chunks. For production RAG, pure dense retrieval is not enough. The retrieval layer implements hybrid search -- a combination of dense (semantic) and sparse (keyword) retrieval -- and merges the results into a single ranked list.
Dense retrieval uses Qdrant. The query embedding is sent to Qdrant with a top-K parameter (typically 20-50), and Qdrant returns the K nearest vectors along with their payloads and similarity scores. Dense retrieval is good at semantic matching -- it finds chunks that mean the same thing as the query even if they share no words. It is weak at exact-term matching -- a query for "ICD-10 code E11.9" will not match well against a chunk that contains that exact string, because the embedding captures semantics, not string identity.
Sparse retrieval uses Postgres full-text search. The query string is converted to a tsquery, and Postgres matches it against the tsvector index of chunk text using GIN indexes. Sparse retrieval is good at exact-term matching -- it finds chunks that contain the specific words or phrases the user asked for. It is weak at semantic matching -- "heart attack" will not match a chunk about "myocardial infarction" unless the search configuration includes synonyms.
Hybrid search combines both. The retrieval layer runs both queries in parallel, retrieves top-K from each, and merges the results using Reciprocal Rank Fusion (RRF) -- a simple, parameter-free method that combines rankings based on position rather than raw scores. RRF works well because dense and sparse scores are not directly comparable (cosine similarity vs. TF-IDF), but ranks are.
The top-K from hybrid retrieval (typically 50-100 candidates) is then passed to the re-ranking layer. Retrieval latency target: under 100 ms for the combined dense + sparse search on a corpus of 100,000 chunks.
7The re-ranking layer takes the top-K candidates from retrieval and re-scores them with a more powerful model. Retrieval is fast but approximate -- it uses a bi-encoder (the embedding model) that embeds the query and each chunk independently and compares them by cosine similarity. Re-ranking uses a cross-encoder that takes the query and each chunk together as input and produces a relevance score. Cross-encoders are more accurate because they see both the query and the chunk at the same time and can model their interaction, but they are slower because they must process each query-chunk pair individually.
The standard approach: retrieve 50-100 candidates with hybrid search (fast, bi-encoder), then re-rank the top 50 with a cross-encoder and keep the top 5-10 for the generation layer. This gives near-cross-encoder accuracy at near-bi-encoder latency.
| Re-Ranker Model | Type | Speed (GPU, 50 docs) | Accuracy | Best For |
|---|---|---|---|---|
| bge-reranker-base | Cross-encoder | ~200 ms | High | General purpose, balanced |
| bge-reranker-large | Cross-encoder | ~500 ms | Very high | High-accuracy production |
| ms-marco-MiniLM-L-12-v2 | Cross-encoder | ~300 ms | High | Trained on MS MARCO, web queries |
| ms-marco-electra-base | Cross-encoder | ~150 ms | High | Fast, good accuracy |
| bge-reranker-v2-m3 | Cross-encoder | ~250 ms | Very high | Multilingual, long context |
8The generation layer takes the user's question and the re-ranked retrieved chunks and produces an answer. This is the LLM. For a sovereign stack, the LLM runs locally via Ollama -- a lightweight, Docker-friendly runtime that serves open-weight models on your own GPU. No external API calls. No data leaving your network.
The prompt template is critical. It must instruct the model to answer only from the retrieved context, to cite sources, and to say "I don't know" when the context does not contain the answer. A good RAG prompt is explicit about these constraints. The retrieved chunks are formatted with their source metadata (document title, page number) so the model can reference them in its answer.
| Model | Parameters | VRAM (Q4) | Context | Best For |
|---|---|---|---|---|
| llama3.1:8b | 8B | ~6 GB | 128K | General purpose, fast |
| qwen2.5:14b | 14B | ~10 GB | 32K | Strong reasoning, multilingual |
| mistral:7b | 7B | ~5 GB | 32K | Efficient, good instruction following |
| llama3.1:70b | 70B | ~40 GB | 128K | Highest quality (needs 2x 24GB GPU) |
| qwen2.5:32b | 32B | ~20 GB | 32K | Balance of quality and speed |
| phi3:14b | 14B | ~10 GB | 128K | Microsoft, strong on reasoning |
Temperature for RAG: 0.1 to 0.3. RAG is a retrieval task, not a creative writing task. You want the model to faithfully report what the retrieved context says, not to hallucinate or embellish. Low temperature keeps the output grounded. Temperature 0 is acceptable but can produce repetitive phrasing across queries; 0.1-0.3 gives slight variation while staying grounded. Never use temperature above 0.5 for factual RAG -- the model will start to "creatively" deviate from the source material, which is exactly what RAG is designed to prevent.
Streaming: For the response layer, Ollama supports token streaming ('stream': true). The generation layer should stream tokens to the user as they are produced rather than waiting for the full response. This makes the system feel responsive even when generation takes 3-5 seconds for a long answer. The citation layer then parses the streamed response to extract and validate citations after the fact.
9The citation layer is what separates a RAG system from a chatbot. When the LLM generates an answer, the citation layer maps each claim in that answer back to the specific source passage it came from, tracks confidence scores, and presents the citations to the user in a usable format. Without citations, the user has no way to verify whether the answer is grounded in real documents or hallucinated. With citations, the user can click through to the original document, read the passage in context, and trust (or reject) the answer.
Source tracking works because the retrieval and re-ranking layers preserve chunk lineage. Each chunk passed to the generation layer carries its document ID, document title, page number, character offset, and chunk text. When the LLM says "[passage 2]", the citation layer resolves that reference to the actual chunk metadata, constructs a URL or file reference to the original document, and links the claim to the source.
Confidence scores come from two sources. The retrieval score (cosine similarity or RRF score) tells you how well the query matched the chunk. The re-ranking score tells you how relevant the chunk is to the query after cross-encoder evaluation. The citation layer combines these into a single confidence indicator: high confidence (re-rank score above threshold, retrieval score above median), medium confidence, or low confidence. This is presented to the user as a visual indicator on each citation -- a green, yellow, or red badge.
How to present citations to users:
10The response layer is how the user interacts with the RAG system. Three interfaces are standard for a sovereign deployment: a web UI, a REST API, and a voice interface for conversational RAG. Each serves a different user population and a different use case.
The web UI is the primary interface for most users. It provides a chat-style interface where the user types a question, sees the answer stream in real-time, and can click citations to view source passages. The UI is a single-page application (React, Vue, or Svelte) that talks to a backend API gateway. For a sovereign stack, the UI is served from the same machine as the rest of the stack, behind your reverse proxy (Nginx or Traefik), and requires no external CDN or JavaScript from third-party domains. The web UI also includes a document upload page (for the ingestion layer's upload channel), a document browser (showing ingested documents with their metadata), and an admin dashboard (showing system health, vector counts, query logs).
The REST API is the programmatic interface. Other applications in your network -- internal tools, automation scripts, other AI agents -- call the RAG API to submit queries and receive structured JSON responses with answers, citations, and confidence scores. The API is authenticated (API keys or OAuth tokens, validated against Postgres-stored credentials) and rate-limited. The API exposes endpoints for querying (POST /api/query), ingesting (POST /api/ingest), browsing documents (GET /api/documents), and health checks (GET /api/health). This is the integration point for the rest of your infrastructure.
For hands-free or mobile use cases -- a clinician asking about a patient's records while examining them, a lawyer querying case files while reviewing evidence -- a voice interface enables spoken queries and spoken responses. The user speaks a question; Whisper (running locally) transcribes it to text; the text goes through the RAG pipeline; the LLM generates a text answer; a local text-to-speech model (Piper or Coqui TTS) converts the answer to audio; the audio is played back to the user. All processing is local. This is conversational RAG -- and it is entirely sovereign, running on your hardware, with no audio leaving the network.
The diagram below shows all components of the sovereign RAG stack and how data flows between them. Every box is a Docker container running on your hardware. Every arrow is a network call within the Docker network -- no traffic leaves the host. The diagram is organized left-to-right by pipeline stage, with the storage layer at the bottom (accessed by both indexing and query paths) and the monitoring layer at the top (observing everything).
Every component in this diagram is a Docker container defined in a single Docker Compose file. The next section shows that file in full.
The entire RAG stack deploys with a single docker-compose.yml file. Every service -- ingestion parsers, processing, chunking, embedding, storage, retrieval, re-ranking, generation, citation, response, and monitoring -- is a container on a shared Docker network. The file below is a production-ready reference architecture. It assumes one machine with an NVIDIA GPU and the NVIDIA Container Toolkit installed. Adjust volume paths to match your storage layout.
deploy.resources.reservations.devices block in each GPU service is the Docker Compose v3.8 syntax for GPU passthrough. It requires the NVIDIA Container Toolkit (nvidia-container-toolkit) installed on the host. Verify with nvidia-smi inside any GPU container. All GPU services (embedding, re-ranker, Whisper, Ollama) share the same physical GPU -- Docker schedules them sequentially. If you have multiple GPUs, assign specific services to specific GPUs using the device_ids field instead of count: 1.
Network architecture: All services communicate over the default Docker Compose bridge network. Services reference each other by service name (e.g., http://qdrant:6333) -- Docker's internal DNS resolves these. No service except Nginx is exposed to the host network. Nginx is the single entry point: port 80/443 for the web UI and API, with optional port 3000 for Grafana if you want the monitoring dashboard externally accessible. For production, put everything behind a VPN or WireGuard tunnel rather than exposing ports publicly.
Volume mounts: All persistent data lives under ./data/ on the host. This is the directory you back up. The structure is:
Hardware requirements scale with corpus size, query volume, and latency targets. Three tiers cover the vast majority of sovereign RAG deployments: a CPU-only tier for development and small corpora, a GPU-recommended tier for typical production, and a production tier for larger corpora and concurrent users.
| Tier | Use Case | GPU | RAM | Storage | Corpus Size | Query Latency |
|---|---|---|---|---|---|---|
| CPU-Only | Dev, prototyping, tiny corpora | None | 8 GB | 50 GB SSD | < 1,000 docs | 10-30 sec |
| CPU-Only (optimized) | Small team, limited budget | None | 16 GB | 500 GB SSD | < 5,000 docs | 5-15 sec |
| GPU (minimum) | Small production | 8 GB VRAM | 16 GB | 500 GB SSD | < 20,000 docs | 2-5 sec |
| GPU (recommended) | Typical production | 12-16 GB VRAM | 32 GB | 1 TB NVMe SSD | < 100,000 docs | 1-3 sec |
| Production | Large team, high query volume | 24+ GB VRAM | 64 GB | 2 TB NVMe SSD | < 500,000 docs | 1-2 sec |
| Multi-GPU Production | Enterprise, 1M+ vectors | 2x 24 GB VRAM | 128 GB | 4 TB NVMe SSD + NAS | 1M+ docs | < 2 sec |
CPU-only deployment is viable for development, prototyping, and very small corpora (under 1,000 documents). Use the all-MiniLM-L6-v2 embedding model (384 dimensions, fast on CPU), a small LLM (llama3.2:3b or phi3:3.8b quantized to Q4), and skip re-ranking. Embedding takes 50-200 ms per chunk on CPU; a 1,000-document corpus with 10 chunks each is 10,000 embeddings, taking 10-30 minutes for full indexing. Query latency is dominated by LLM generation on CPU (5-15 seconds per query). This tier is for validating the pipeline and testing ingestion flows, not for production use.
A single NVIDIA GPU with 12-16 GB VRAM is the sweet spot for sovereign RAG. It fits the embedding model (BGE-base, ~1 GB), the re-ranker (bge-reranker-base, ~1 GB), and a 7B-14B LLM (Q4 quantized, 5-10 GB) simultaneously with room for context. Indexing throughput is 1,000-5,000 chunks per second. Query latency drops to 1-3 seconds (embedding 50 ms + retrieval 50 ms + re-ranking 200 ms + generation 1-2 s). This tier handles up to 100,000 documents and moderate concurrent query volume (5-10 simultaneous users). 32 GB of system RAM gives Postgres and Qdrant enough headroom for the metadata and vector index.
A 24 GB GPU (RTX 4090, RTX 6000 Ada, or L40S) allows running a larger LLM (Qwen 32B or Llama 70B split across two GPUs) alongside the embedding and re-ranking models. NVMe SSD is essential -- the vector index and Postgres FTS index are I/O-bound, and SATA SSDs become a bottleneck at 500,000+ vectors. 64 GB of system RAM allows Qdrant to keep the full HNSW index in memory for sub-50ms retrieval. This tier supports 10-20 concurrent users with sub-2-second query latency. For 1M+ vectors, see the Scale Considerations section for distributed Qdrant deployment.
A RAG stack is ten services running simultaneously, and every one of them can fail, degrade, or become a bottleneck. Monitoring is not optional. The standard sovereign monitoring stack is three services: Prometheus for metrics collection, Grafana for dashboards and alerting, and cAdvisor for container-level resource metrics. All three run as Docker containers and are included in the Docker Compose file above.
What to monitor:
| Metric | Source | Why It Matters | Alert Threshold |
|---|---|---|---|
| Embedding latency (p50, p99) | Orchestrator | If embedding slows down, both indexing and querying degrade | p99 > 200 ms |
| Retrieval latency (p50, p99) | Orchestrator | Qdrant + Postgres FTS combined search time | p99 > 500 ms |
| Re-ranking latency (p50, p99) | Orchestrator | Cross-encoder inference time for top-50 candidates | p99 > 1000 ms |
| Generation latency (p50, p99) | Orchestrator | Ollama LLM response time (time to first token + total) | p99 > 10 sec |
| End-to-end query latency | Orchestrator | Total time from question to answer | p99 > 5 sec |
| GPU utilization | cAdvisor / nvidia-smi exporter | If GPU is at 100%, services are competing for compute | > 95% for 5 min |
| GPU memory usage | cAdvisor | If VRAM fills up, models will OOM and crash | > 90% |
| Vector count | Qdrant API | Tracks corpus growth; alerts on sudden drops (data loss) | Sudden drop > 10% |
| Query volume | Orchestrator | Requests per minute; detects usage spikes | Spike > 5x baseline |
| Retrieval recall (zero results) | Orchestrator | Queries returning no chunks indicate corpus gaps | > 5% of queries |
| Ingestion queue depth | Redis | Backlog of unprocessed documents | > 100 for 10 min |
| Postgres connections | Postgres exporter | Connection pool exhaustion blocks all queries | > 80% of max_connections |
| Disk usage (data volume) | cAdvisor | If the data disk fills, all services fail | > 85% |
Prometheus scrapes metrics from each service's /metrics endpoint every 15 seconds. The orchestrator exposes custom RAG metrics (embedding latency, retrieval latency, generation latency, query volume, citation rate). Qdrant exposes its own metrics endpoint. Postgres requires the postgres_exporter sidecar. GPU metrics require the nvidia-gpu-exporter. cAdvisor provides per-container CPU, memory, and network metrics out of the box.
Grafana visualizes these metrics in dashboards. The RAG dashboard should show: a latency breakdown panel (embedding, retrieval, re-ranking, generation, end-to-end as stacked bars), a GPU panel (utilization and memory as gauges), a query volume panel (requests per minute as a time series), a vector count panel (total vectors as a counter), and an ingestion panel (queue depth, documents processed per hour). Set up alerting in Grafana to send notifications (email, Slack webhook, or local systemd journal) when any metric crosses its alert threshold.
cAdvisor provides container-level resource metrics: per-container CPU usage, memory usage, and network I/O. This is how you identify which service is consuming resources when the overall system is slow. If GPU memory is at 90%, cAdvisor tells you whether it is the embedding service, the re-ranker, or Ollama that is consuming it.
The RAG stack stores three categories of data, and each requires its own backup strategy. All backups are local (to a separate disk or NAS) and automated (cron or systemd timers). No backup data leaves your network.
Qdrant supports snapshot-based backups. A snapshot captures the entire collection (vectors, payloads, and index state) to a single file. The backup job calls the Qdrant API to create a snapshot, then copies the snapshot file to the backup destination. Snapshots are consistent -- they are taken without stopping the service. For a 100,000-vector collection, a snapshot takes 5-30 seconds and produces a file of 200-500 MB (depending on payload size). Schedule nightly snapshots, retain the last 7 daily, 4 weekly, and 12 monthly. Restore by copying the snapshot file back and issuing a restore API call.
PostgreSQL uses pg_dump for logical backups. The backup job runs pg_dump against the RAG database, producing a SQL script that can restore the entire database (schema, data, indexes, full-text search configuration). For a database with 100,000 document records and 500,000 chunk metadata rows, pg_dump takes 1-5 minutes and produces a 50-200 MB compressed file. Schedule nightly dumps with the same retention policy as Qdrant (7 daily, 4 weekly, 12 monthly). For larger databases, consider pg_basebackup for physical backups or WAL archiving for point-in-time recovery. Restore by running the SQL script against a fresh Postgres instance.
The original documents on the file storage layer are the source of truth. If you lose them, you lose the ability to show users the original context for any citation. File backup uses rsync to an incremental backup destination -- a separate disk, a NAS, or an offsite server (still on your network). rsync -a --delete mirrors the file store efficiently (only changed files are transferred). Schedule nightly rsync runs. Since documents are immutable once ingested (new versions are new documents, not overwrites), the file store only grows -- rsync is fast because it only copies new files. Verify backup integrity with periodic test restores: restore the Qdrant snapshot, Postgres dump, and file backup to a test server, and verify that queries return correct results.
The single-machine Docker Compose deployment handles up to 100,000 documents and moderate concurrent query volume. Beyond that, the architecture needs to scale. The scaling path has three stages: single machine (where most deployments stay), multi-GPU (for compute-bound embedding or generation), and distributed Qdrant (for storage-bound vector search).
| Scale | Corpus Size | Architecture | Key Change | Query Volume |
|---|---|---|---|---|
| Single machine | < 100K docs | Docker Compose, one GPU | Starting point | 5-20 concurrent users |
| Single machine, tuned | 100K - 500K docs | Qdrant quantization, larger GPU | Enable scalar quantization in Qdrant | 20-50 concurrent users |
| Multi-GPU | 500K - 1M docs | Separate GPUs for embedding vs LLM | Assign embedding to GPU 0, LLM to GPU 1 | 50-100 concurrent users |
| Distributed Qdrant | 1M+ docs | Qdrant cluster (3+ nodes), shared Postgres | Shard Qdrant collection across nodes | 100+ concurrent users |
| Full distributed | 10M+ docs | Multi-node, Kubernetes, distributed everything | K8s deployment, Qdrant cluster, Postgres replication | 500+ concurrent users |
The Docker Compose deployment described in this guide runs on a single machine and handles up to 100,000 documents with 5-20 concurrent users. This covers the vast majority of sovereign RAG use cases. At this scale, all services share one GPU and one disk. The main tuning knobs are Qdrant's ef_search (lower it for faster retrieval at the cost of recall), the embedding batch size (increase to 200 if GPU memory allows), and Ollama's num_predict limit (cap at 512-1024 tokens to prevent runaway generation). Monitor GPU utilization -- if it consistently exceeds 90%, you are compute-bound and need a second GPU.
When a single GPU becomes the bottleneck, add a second. The most effective split is to dedicate one GPU to the embedding and re-ranking services and the other to Ollama (the LLM). This eliminates the contention where a query's embedding step waits for the LLM to finish generating the previous answer. With two 24 GB GPUs, you can run a 70B LLM (split across both) alongside the embedding and re-ranking models. In Docker Compose, assign services to specific GPUs using device_ids: ['0'] and device_ids: ['1'] in the deploy block. Multi-GPU also helps for batch re-indexing: run embedding on GPU 0 while the system serves queries on GPU 1.
Beyond 1 million vectors, a single Qdrant node becomes the bottleneck -- both for storage (the HNSW index no longer fits in RAM) and for throughput (a single node can only handle so many concurrent searches). Qdrant supports distributed deployment with sharding (each node holds a subset of the collection) and replication (each shard has copies on multiple nodes for fault tolerance). A three-node Qdrant cluster with the collection sharded across all nodes handles 10M+ vectors with sub-100ms retrieval. Deploy Qdrant cluster nodes on separate machines, configure them with a shared consensus address, and set the collection shard count equal to the number of nodes. Postgres can be replicated similarly (read replicas for query throughput, streaming replication for fault tolerance). At this scale, you are likely running on Kubernetes rather than Docker Compose, and the deployment complexity increases significantly -- but the architecture principles remain the same.
Suggested Citation:
Building a Sovereign RAG Infrastructure Stack: Complete End-to-End Guide to On-Premise Retrieval-Augmented Generation. Avondale.AI Technical Documentation Library. Accessed June 2026. https://avondale.ai/technical/rag/rag-infrastructure-stack.html