RAG Explained: From Basic to Advanced Architectures

1. What Is RAG?

Retrieval-Augmented Generation (RAG) is a technique that augments a large language model with an external knowledge base. Instead of relying solely on knowledge baked into the model's parameters during training, a RAG system retrieves relevant documents at inference time and feeds them into the model's context window as it generates a response. The model then conditions its answer on both its parametric memory and the freshly retrieved evidence.

The original formulation, introduced by Lewis et al. at Facebook AI Research in 2020, was a single, linear pipeline: chunk your corpus into passages, embed each passage into a vector index, retrieve the top-k passages most similar to the user's query, and pass them as context to a seq2seq generator. This design -- now called Naive RAG -- remains the conceptual backbone of every RAG system built today. But the simplicity is deceptive: in production, naive RAG fails on anything beyond simple factual lookups. Documents get missed when the query phrasing differs from the document phrasing. Retrieved passages include noise that confuses the generator. Multi-step questions go unanswered because a single retrieval pass cannot surface all the needed facts.

The intervening years have produced a family of architectures that address these failure modes. Advanced RAG adds query transformation, hybrid search, and re-ranking. Modular RAG makes every component swappable. Agentic RAG lets the model decide when and what to retrieve, iterating over multiple rounds. Self-RAG trains the model to self-assess retrieved passages and discard irrelevant ones. Corrective RAG evaluates retrieval quality and falls back to alternative sources when the primary index underperforms. Adaptive RAG routes between architectures based on query complexity. Graph RAG replaces or augments vector similarity with knowledge-graph traversal.

This article covers each in depth. Throughout, we emphasize sovereign, on-premise deployment: every architecture here can run entirely on your own hardware, with no data ever leaving your network. This matters for regulated industries (healthcare, defense, finance), classified environments, and any organization that cannot accept the compliance, latency, or sovereignty risk of sending proprietary data to a third-party API.

2. Why RAG Over Fine-Tuning?

Before diving into architectures, it is worth understanding why RAG exists as a distinct approach rather than simply fine-tuning a model on your documents. Both methods inject custom knowledge into an LLM, but they solve fundamentally different problems.

The two are not mutually exclusive. A common production pattern is to fine-tune a model on domain-specific language and reasoning patterns, then deploy it behind a RAG pipeline for factual grounding. The fine-tune teaches the model how to talk; the RAG layer teaches it what to say. In a sovereign deployment, both the fine-tuned model and the RAG index run entirely on-premise.

3. Naive RAG

Naive RAG is the original three-stage pipeline: index, retrieve, generate. It is the simplest architecture that qualifies as RAG, and it remains the right starting point for many use cases. Every more advanced architecture is a modification or extension of this baseline.

The Three Stages

Stage 1: Index

Your document corpus -- PDFs, internal wikis, code, support tickets, medical records, whatever -- is split into chunks of a fixed token size (typically 256-1024 tokens) with optional overlap. Each chunk is passed through an embedding model that converts the text into a dense vector (commonly 768-1536 dimensions). These vectors are stored in a vector index -- a data structure optimized for fast approximate nearest-neighbor (ANN) search. Popular on-premise options include FAISS, Milvus, Qdrant, Weaviate, and Chroma.

Indexing is a batch operation. You run it once when the corpus is ingested, then incrementally re-index when documents change. The index lives on disk or in memory on your own servers. No document text needs to leave your infrastructure for indexing; the embedding model (e.g., a local BGE, E5, or gte-large model) runs on your GPUs.

Stage 2: Retrieve

When a user submits a query, the same embedding model encodes the query into a vector. The vector index performs a similarity search (usually cosine similarity or dot product) to find the top-k chunks whose vectors are closest to the query vector. A typical value for k is between 3 and 20. The retrieved chunks, along with their source metadata, are returned as candidates for context.

This is a single-step, single-shot retrieval. One query, one search, one set of results. There is no query reformulation, no iterative retrieval, and no assessment of whether the results are actually relevant.

Stage 3: Generate

The retrieved chunks are concatenated (or otherwise assembled) and inserted into the LLM's prompt as context. The prompt typically looks like:

The LLM then generates a response conditioned on this context. Because the model has explicit access to the source text, it can ground its answer in the retrieved passages and, with appropriate prompting, cite which passage a claim came from. This is the fundamental value proposition of RAG: the model's answer is constrained by evidence it can point to.

Strengths

• Simplicity: Three stages, well-understood, minimal moving parts. Can be built in a weekend.
• Low latency: One embedding call, one vector search, one LLM call. End-to-end latency is dominated by the LLM.
• Easy to debug: When it fails, you can inspect the retrieved chunks and the prompt. The failure mode is visible.
• Sufficient for simple Q&A: If your use case is "answer questions from a 500-page employee handbook," naive RAG works well.

Limitations

• No query transformation: If the user asks "What's the policy on remote work?" but the document says "telecommuting guidelines," the embedding similarity may be too low to retrieve the right chunk. Naive RAG does not rewrite the query.
• No re-ranking: The top-k chunks are returned in raw similarity order. A highly similar but irrelevant chunk can crowd out a less-similar-but-correct chunk. Cross-encoders, which are far more accurate at relevance scoring than bi-encoders, are not used.
• No multi-hop: A question like "Who is the CTO of the company that acquired the startup founded by the author of paper X?" requires multiple retrieval steps. Naive RAG does one search and hopes the right documents surface.
• No quality assessment: The system has no mechanism to detect that retrieval failed. If the top-k chunks are all irrelevant, the LLM is still forced to answer, often hallucinating.
• Context window waste: All top-k chunks are stuffed into the prompt regardless of relevance, consuming context window capacity with noise that degrades generation quality.

4. Advanced RAG

Advanced RAG keeps the three-stage pipeline but adds optimization at each stage: pre-retrieval, retrieval, and post-retrieval. The goal is to improve the quality of what the LLM sees without changing the fundamental architecture. It is the most common production RAG pattern because it offers the best accuracy-to-complexity ratio: significant gains over naive RAG without the engineering burden of agentic loops or custom-trained models.

4.1 Pre-Retrieval Optimization

Pre-retrieval optimization operates on the query before it reaches the vector index. The insight is that users are bad at writing queries. They use vague terms, ask questions in a different register than the document, ask multi-part questions, or use domain-specific jargon that the embedding model handles poorly. Pre-retrieval techniques rewrite the query to make it more retrievable.

Query Transformation

The simplest approach is to pass the user's query to an LLM and ask it to rewrite the query in a more retrieval-friendly form. For example, "How do I reset my password?" might become "password reset procedure steps administrator." This is a single LLM call that costs ~50 tokens but can dramatically improve retrieval when the user's phrasing differs from the document's phrasing.

A more structured variant is query decomposition: break a multi-part question into sub-questions. "Compare the maternity leave policies of the US and Sweden" becomes ["maternity leave policy United States", "maternity leave policy Sweden"]. Each sub-query is retrieved independently and the results merged.

Query Expansion

Query expansion generates multiple variant queries from the original and retrieves for each, then merges the results. This hedges against the embedding model's sensitivity to phrasing. If the original query misses the right document, one of the variants may hit it. Typical expansion counts are 3-5 variants.

Expansion can be keyword-based (adding synonyms from a domain thesaurus), LLM-based (asking the LLM to generate paraphrases), or pseudo-relevance-feedback-based (retrieving with the original query, taking the top results, extracting keywords from them, and re-querying with those keywords added).

HyDE (Hypothetical Document Embeddings)

HyDE, introduced by Gao et al. in 2022, is a particularly effective pre-retrieval technique. The insight: the embedding of a question is a poor query for finding answers, because questions and answers live in different semantic regions of the embedding space. HyDE bridges this gap by having the LLM generate a hypothetical answer to the question first -- a plausible-sounding paragraph that may contain factual errors but will be stylistically and topically similar to the real answer. This hypothetical document is then embedded, and the resulting vector is used to search the index.

Because the hypothetical document is in "answer space" rather than "question space," its embedding matches real answer documents more closely than the question's embedding would. The LLM's hallucination in the hypothetical document is not a problem -- it is a feature, because the hallucination has the right shape even if the facts are wrong.

Step-Back Prompting

Step-back prompting asks the LLM to extract a more general, abstract version of the question before retrieval. "What was the revenue of company X in Q3 2024?" becomes "What is company X's financial performance?" The step-back question retrieves broader context documents that may contain the specific fact plus the surrounding context needed to interpret it. This is especially useful when the specific fact is buried in a larger document that would not be retrieved by the specific query.

4.2 Retrieval Optimization

Retrieval optimization improves the search itself. Even with a perfect query, single-vector similarity search has known weaknesses: it struggles with keyword matching (exact terms), handles rare terms poorly, and is dominated by semantic similarity that may not align with relevance. The two main techniques are hybrid search and re-ranking.

Hybrid Search

Hybrid search combines dense vector retrieval (semantic similarity) with sparse lexical retrieval (BM25 or similar TF-IDF-based scoring). Vector search excels at finding semantically related content but misses exact term matches. BM25 excels at exact term matching but misses synonyms and paraphrase. By running both and merging the results (via reciprocal rank fusion or a learned weight), you get the strengths of both.

In practice, hybrid search is one of the highest-impact changes you can make to a RAG pipeline. It is cheap to implement (most vector databases support hybrid search natively), adds minimal latency, and meaningfully improves retrieval recall, especially on technical documents with lots of proper nouns, code identifiers, and numeric values.

Re-Ranking with Cross-Encoders

The embedding model used for retrieval is a bi-encoder: it encodes the query and each document independently, then compares them via cosine similarity. This is fast (document embeddings are pre-computed) but weak, because the query and document never interact. A cross-encoder takes the query and document together as a single input, runs the full transformer over the concatenated pair, and outputs a relevance score. This is far more accurate because the model can attend to the interaction between query and document tokens.

The standard pattern is two-stage retrieval: use the fast bi-encoder to retrieve top-100 candidates (high recall), then use the slow-but-accurate cross-encoder to re-rank those 100 down to the top-10 (high precision). Popular cross-encoders include BGE-reranker, Cohere Rerank (cloud; not suitable for sovereign use), and ColBERT. On-premise options are mature: BGE-reranker-large runs on a single GPU and re-ranks 100 candidates in under 100ms.

4.3 Post-Retrieval Optimization

Post-retrieval optimization processes the retrieved chunks after retrieval but before generation. The goal is to ensure the LLM sees only the most useful information, presented in the most effective form.

Context Compression

Retrieved chunks often contain irrelevant padding. A 512-token chunk retrieved for a single sentence of relevant content wastes 98% of the context window on noise. Context compression techniques extract only the relevant portions:

• Sentence-level filtering: Score each sentence in the chunk for relevance to the query; keep only the top sentences.
• LLM-based compression: Ask the LLM to summarize the chunk down to the parts relevant to the query. Higher quality, higher latency.
• Contextual embeddings: At index time, use an LLM to add a short contextual prefix to each chunk ("This chunk is from the Q3 2024 financial report, in the section on operating expenses...") so the embedding captures document context, not just the local chunk text. This is the approach of Anthropic's contextual retrieval.

Citation Tracking

Advanced RAG systems track which chunks were used to generate which claims, enabling inline citations in the response. This requires the LLM to be prompted to cite sources (e.g., "[1]" after each claim) and a mapping from citation numbers back to chunk identifiers. In on-premise deployments, citation tracking is critical for regulated industries: a medical or legal answer must be traceable to the specific source document and page.

5. Modular RAG

Modular RAG is less a single architecture than a design philosophy: treat the RAG pipeline as a composition of independent, swappable modules. Naive RAG has three modules (indexer, retriever, generator). Advanced RAG adds more (query transformer, re-ranker, context compressor). Modular RAG makes the boundaries explicit and interchangeable.

The key idea is that each module has a well-defined interface (input and output types) and can be replaced without affecting the others. Want to swap your embedding model from BGE-large to OpenAI text-embedding-3-large? Change the indexer and retriever; the generator is unaffected. Want to upgrade your LLM from Llama-3-70B to a larger model? Change the generator; the retrieval pipeline is unaffected. Want to add a memory module so the system remembers previous turns in a conversation? Slot it in between retrieval and generation.

Standard Modules

Why Modularity Matters for Production

In a production system, components evolve at different rates. Embedding models improve every few months. LLMs improve every few weeks. Your document corpus changes daily. A monolithic RAG implementation -- where the indexing, retrieval, and generation code are tangled together -- makes upgrades painful. Each improvement requires re-validating the entire system.

A modular architecture isolates changes. You can A/B test a new embedding model by routing 10% of traffic to a retriever using the new model, while the rest of the pipeline is unchanged. You can roll back just the reranker without touching the index. You can swap the generator for a larger model on more powerful hardware while keeping the retrieval stack on cheaper servers.

Orchestration Frameworks

The two dominant orchestration frameworks for modular RAG are LlamaIndex and LangChain (with its LangGraph extension). Both provide abstractions for each module and allow custom implementations. For sovereign, on-premise deployment, LlamaIndex tends to have better support for local models and self-hosted vector stores, while LangGraph excels at stateful, multi-step workflows (which overlap with Agentic RAG).

6. Agentic RAG

Agentic RAG represents a qualitative shift: the LLM is no longer a passive generator that consumes whatever context it is given. It becomes an agent that actively decides when to retrieve, what to retrieve, and how many times to retrieve. Retrieval is exposed to the model as a tool -- a function the model can call with arguments it chooses, as many times as it deems necessary, in service of answering the user's question.

This is the RAG analogue of tool-use in agentic LLM frameworks. Instead of a fixed pipeline (query → retrieve → generate), the system runs an agentic loop: the LLM reasons about the question, decides whether retrieval is needed, formulates a search query, receives the results, reasons about whether the results are sufficient, and either generates an answer or retrieves again with a refined query.

The Agentic Loop

How It Works

The LLM is given a system prompt that describes its available tools. One of these tools is retrieve(query: str) -> List[Chunk]. The model is prompted to reason step-by-step (Chain of Thought) and to call the retrieve tool whenever it needs information it does not already have. After each tool call, the results are added to the model's context, and the model is prompted to continue reasoning.

This loop continues until the model either (a) decides it has enough information to answer, (b) decides retrieval is not needed (the question can be answered from parametric knowledge), or (c) hits a maximum iteration limit (a safety guardrail to prevent infinite loops).

Multi-Hop Reasoning

Agentic RAG excels at multi-hop questions -- questions that require chaining multiple facts, each dependent on the previous. The example in the diagram above ("Who is the CTO of the company that acquired the startup founded by the author of paper X?") requires three sequential retrievals, each using information from the previous step. No single retrieval pass could answer this, because the second and third queries are not known until the first query's results are seen.

This is the fundamental advantage of agentic RAG over advanced RAG: the retrieval strategy is dynamic, conditioned on intermediate results. Advanced RAG can pre-generate multiple queries, but it must do so upfront, without seeing any results. Agentic RAG generates each query in response to what it has learned so far.

When You Don't Know Which Documents Are Relevant

Agentic RAG also shines when you cannot predict which documents the question will need. In a large, heterogeneous corpus (e.g., a company's entire internal knowledge base spanning HR, engineering, legal, finance, and operations), a question like "What's the process for hiring a contractor in Germany?" might require documents from HR (hiring policy), Legal (contractor classification in Germany), and Finance (payment process for international contractors). An agentic system can explore each of these areas in sequence; a fixed-pipeline system must either retrieve all of them upfront (wasting context) or miss some.

Tradeoffs

• Latency: Each retrieval iteration adds a round trip (embedding + search + LLM reasoning). A 3-hop question may take 5-15 seconds versus <1 second for naive RAG.
• Cost: The LLM is invoked many times. With a local model, this is GPU time; with an API, it is token cost. In on-premise deployments, the cost is latency and GPU utilization, not dollars per token.
• Complexity: The agentic loop is harder to debug than a fixed pipeline. The model may retrieve unnecessarily, loop on unhelpful queries, or fail to converge. Robust prompting and iteration limits are essential.
• Determinism: The same question may take different retrieval paths on different runs, as the LLM's reasoning is stochastic. This is a problem for auditing and compliance in regulated environments.

7. Self-RAG

Self-RAG, introduced by Asai et al. in 2023, trains the model to self-reflect on the retrieval and generation process. Rather than treating retrieval as a fixed step that always happens, Self-RAG teaches the model to make two key decisions autonomously: (1) whether retrieval is needed at all for this query, and (2) whether each retrieved passage is relevant and should be used.

This is achieved through special reflection tokens that the model generates alongside its text output. These tokens act as the model's internal monologue about its own reasoning process, and they control the retrieval and grounding behavior.

The Reflection Tokens

Self-RAG introduces three types of reflection tokens:

• Retrieve token: Decides whether retrieval is needed for the current generation step. If the model decides retrieval is not needed (e.g., the question is about general knowledge the model already has), it skips retrieval entirely -- saving latency and avoiding irrelevant context.
• IsRel (relevance) token: After retrieval, the model evaluates each retrieved passage for relevance to the query. Irrelevant passages are discarded before they enter the context.
• IsSup (support) token: After generating the answer, the model checks whether each claim it made is supported by the retrieved (and relevant) passages. This is a post-generation self-check against hallucination.

Training Process

Self-RAG requires a custom training process. The model is fine-tuned on a dataset where the training examples include the reflection tokens as targets. The training data is generated by a critic model that labels whether retrieval is needed, whether passages are relevant, and whether claims are supported. The base model then learns to predict these tokens itself, internalizing the critic's judgment.

This is a significant departure from plug-and-play RAG: you cannot take an off-the-shelf LLM and make it Self-RAG by changing the prompt. The self-reflection behavior is baked into the model weights through fine-tuning. This means Self-RAG requires training infrastructure and labeled data, which is a higher barrier than the other architectures.

Effect on Hallucination

Self-RAG's primary value is hallucination reduction. By discarding irrelevant passages (IsRel) and self-checking that generated claims are supported (IsSup), the model is forced to ground its answers in retrieved evidence. In the original paper, Self-RAG showed significant improvements over both standard RAG and no-RAG baselines on factuality benchmarks, with a particular improvement on questions where the retrieval system returns irrelevant documents.

8. Corrective RAG (CRAG)

Corrective RAG (CRAG), introduced by Yan et al. in 2024, addresses a specific failure mode: the retrieval system returns documents, but those documents are wrong or low-quality. Instead of trusting the retriever's output, CRAG adds an evaluation step that scores the quality of retrieved documents and, if quality is too low, falls back to alternative retrieval sources -- typically web search.

The Three-Tier Quality Assessment

CRAG uses a lightweight retrieval evaluator (a small fine-tuned model) to classify each retrieved document into one of three tiers:

• Correct: The document is highly relevant to the query. Use it as-is.
• Incorrect: The document is irrelevant. Discard it and trigger fallback retrieval (web search or alternative index).
• Ambiguous: The document is partially relevant. Apply knowledge refinement -- extract the relevant portions and discard the rest.

Knowledge Refinement

For ambiguous documents, CRAG does not simply include or exclude the whole document. It applies a decomposition-and-compression step: the document is split into smaller knowledge strips, each strip is scored for relevance, and only the high-scoring strips are kept. This is a more granular form of the context compression used in Advanced RAG, but applied at retrieval-evaluation time rather than post-retrieval.

Fallback to Web Search

When the retrieval evaluator determines that the retrieved documents are incorrect (or all are ambiguous with low confidence), CRAG triggers a web search as a fallback. The web search results are then subjected to the same evaluation and refinement process. This is the "corrective" in Corrective RAG: the system does not passively accept bad retrieval -- it actively detects failure and seeks better information.

9. Adaptive RAG

Adaptive RAG, introduced by Jeong et al. in 2024, observes that different questions need different amounts of retrieval. A simple factual question ("What is the capital of France?") needs no retrieval at all -- the model knows it. A medium-complexity question ("What is our company's remote work policy?") needs a single retrieval pass. A complex multi-hop question ("Compare the Q3 revenue of the three companies we acquired in 2024") needs iterative retrieval.

Running the full agentic RAG pipeline for every question wastes latency on simple questions. Running naive RAG for every question fails on complex ones. Adaptive RAG solves this by classifying the query first, then routing to the appropriate RAG strategy.

The Query Complexity Classifier

A small, fast classifier (a fine-tuned model or even an LLM-based prompt) examines the incoming query and assigns it to one of three complexity tiers:

• Level A (No Retrieval): The question can be answered from the model's parametric knowledge. Examples: general knowledge, definitions, simple reasoning. Route directly to the LLM.
• Level B (Single-step Retrieval): The question requires one retrieval pass to find the relevant documents. Route to Advanced RAG (query transformation + hybrid search + re-ranking).
• Level C (Multi-step Retrieval): The question requires iterative retrieval with intermediate reasoning. Route to Agentic RAG.

Training the Classifier

The classifier can be trained in several ways:

• LLM-based classification: Prompt an LLM to assess complexity. Zero training data needed, but adds one LLM call of latency.
• Small fine-tuned classifier: Train a small model (e.g., a 1B-parameter model or even a logistic regression over embeddings) on labeled examples of question-complexity pairs. Faster inference, but requires labeled data.
• Rule-based heuristics: Use question length, presence of comparison words ("versus", "compare"), or entity count as proxies for complexity. Cheapest but least accurate.

In practice, the LLM-based approach is the most common starting point because it requires no training data and adapts to new question types. As traffic grows, a fine-tuned classifier can replace it for lower latency.

Latency Savings

The key benefit of Adaptive RAG is latency optimization. If 60% of your traffic is simple questions that need no retrieval, those questions skip the retrieval pipeline entirely and respond in LLM-generation time only. The 30% that need single-step retrieval use the efficient Advanced RAG path. Only the 10% that truly need multi-hop reasoning pay the cost of the agentic loop. The average latency across all queries drops significantly compared to running Agentic RAG for everything.

10. Graph RAG

Graph RAG replaces or augments vector-similarity retrieval with knowledge graph traversal. Instead of (or in addition to) embedding chunks into a vector index, the system builds a knowledge graph from the corpus: entities (people, organizations, concepts, products) become nodes, and relationships ("works at", "acquired by", "is a subtype of") become edges. Retrieval then traverses the graph rather than (or alongside) searching the vector index.

Graph RAG excels at relationship-heavy queries -- questions that are fundamentally about connections between entities. "Who are all the people who have worked at both company X and company Y?" is trivial in a graph (follow the "works at" edges from X, find the intersection with those who "work at" Y) but nearly impossible with vector similarity, which has no notion of entity relationships.

How It Works

The corpus is processed by an LLM (or NER + relation extraction pipeline) that extracts entities and relationships, building a structured graph. At query time, the system identifies the entities mentioned in the query, locates them in the graph, and traverses the graph to find connected entities and the paths between them. The subgraph discovered by this traversal is then converted to text and passed to the LLM as context.

Microsoft's GraphRAG (2024) popularized a specific approach: build the graph, then compute community summaries -- hierarchical summaries of clusters of related entities. At query time, the system determines which community summaries are relevant and uses them as context. This enables queries that span many documents, because the community summaries aggregate information that no single document contains.

When Graph RAG Outperforms Vector RAG

• Multi-entity questions: "List all projects that involve people from the hardware team who joined after 2023." Requires traversing team membership, hire dates, and project assignments -- all relational data.
• Cross-document reasoning: "What is the connection between person A and person B?" The answer may span multiple documents, each containing one relationship. Vector RAG retrieves one document; Graph RAG follows the chain.
• Hierarchical questions: "What are the major themes across all our Q4 reports?" Community summaries can answer this at a level no single-chunk retrieval can match.

11. Architecture Comparison Table

The following table compares all eight architectures across the dimensions that matter for architectural selection. Use it as a quick reference when deciding which approach to adopt.

Cost & Infrastructure Summary

12. Architecture Diagrams (Side-by-Side)

The following diagrams present all architectures in a uniform format for direct comparison. Each shows the data flow from user query to final answer, highlighting where decisions are made and where the architectures diverge.

Naive RAG vs. Advanced RAG

Self-RAG vs. Corrective RAG

Adaptive RAG Routing

Graph RAG (Hybrid with Vector)

13. On-Premise Deployment Guide

Every architecture in this article can run entirely on-premise, with no data leaving your network. This is a hard requirement for many deployments: classified government environments, HIPAA-regulated healthcare, GDPR-sensitive European deployments, financial institutions under audit, and any organization that treats its data as a sovereign asset. The following guidance covers the practical considerations for each layer of the stack.

Embedding Models (On-Premise)

The embedding model converts text to vectors for both indexing and querying. It must run locally. The leading open-weight options are:

Vector Databases (On-Premise)

Re-Ranking Models (On-Premise)

• BGE-reranker-large: Open-weight cross-encoder. Runs on a single GPU. Re-ranks 100 candidates in ~50ms. The default choice for on-premise re-ranking.
• BGE-reranker-v2-m3: Multilingual version; same deployment profile.
• ColBERT-v2: Late-interaction model; more accurate than cross-encoders on some benchmarks; higher storage cost (per-token vectors). Suitable when accuracy is critical and storage is not a constraint.
• Jina-reranker-v2: Open-weight; supports multilingual and long documents.

LLM Generators (On-Premise)

The generator LLM is the largest infrastructure cost. Model choice depends on the architecture:

• Naive / Advanced RAG: A 7B-14B model (Llama-3.1-8B, Qwen-2.5-14B) is often sufficient, as the retrieval pipeline does the heavy lifting and the model just needs to synthesize the provided context.
• Agentic RAG: Requires a 70B+ model with strong tool-use and reasoning. Llama-3.1-70B, Qwen-2.5-72B, or DeepSeek-V3. These require 2-4 GPUs (depending on quantization) for inference.
• Self-RAG: Requires a fine-tuned model; start with a 7B-13B base and fine-tune with reflection tokens.

All of these models are available as open weights and can be served locally with vLLM, TGI, or Ollama. No API calls leave your network.

Air-Gapped Considerations

14. Selection Decision Tree

Use the following decision tree to select the right RAG architecture for your use case. Start at the top and follow the branch that matches your constraints.

Summary & Key Takeaways

RAG has evolved from a single three-stage pipeline into a family of architectures, each addressing a distinct failure mode of its predecessor. Understanding these architectures -- and knowing when to apply each -- is the core competence of anyone building production retrieval-augmented systems.

1. Naive RAG (index → retrieve → generate) is the baseline. Simple, fast, sufficient for small corpora and simple questions. Start here.
2. Advanced RAG adds pre-retrieval (query transformation, HyDE), retrieval (hybrid search, cross-encoder re-ranking), and post-retrieval (compression, citation) optimization. The sweet spot for most production systems.
3. Modular RAG makes each component swappable. Not a different pipeline, but a design discipline that enables independent upgrades and A/B testing.
4. Agentic RAG lets the LLM decide when and what to retrieve. Necessary for multi-hop reasoning and heterogeneous corpora. Higher latency and cost.
5. Self-RAG trains the model to self-assess retrieval necessity and passage relevance. The strongest approach for hallucination reduction in regulated domains. Requires fine-tuning.
6. Corrective RAG evaluates retrieval quality and falls back to alternative sources on failure. Best for noisy corpora with variable retrieval quality.
7. Adaptive RAG classifies query complexity and routes to the cheapest sufficient pipeline. Optimizes average latency across mixed query distributions.
8. Graph RAG uses knowledge graphs for relationship-heavy queries. Unmatched for multi-entity and cross-document reasoning. Often combined with vector RAG in a hybrid approach.

Every architecture in this article runs fully on-premise. For sovereign, air-gapped, or regulated deployments, the open-weight ecosystem -- BGE embeddings, FAISS/Qdrant/Milvus indexes, BGE-reranker, Llama/Qwen/DeepSeek LLMs -- provides a complete, no-data-egress RAG stack. The choice of architecture is independent of the deployment model; what changes is the fallback source (for CRAG) and the available models (for agentic and Self-RAG). Build the simplest architecture that passes your evaluation set, and upgrade only when the data tells you to.