RAG Retrieval Methods: Complete Guide

Technical Deep-Dive · Sovereign, On-Premise RAG · June 2026

Retrieval is the backbone of any Retrieval-Augmented Generation (RAG) system. The quality of the documents you surface determines the ceiling of your answer quality—no amount of prompt engineering compensates for retrieving the wrong passages. This guide covers every major retrieval approach, their trade-offs, how they combine, and practical recommendations for deploying each method in a fully sovereign, on-premise environment where no data leaves your network.

1. Dense Vector Retrieval

Dense vector retrieval is the standard approach in modern RAG. You embed the user’s query and all documents into a shared high-dimensional vector space using a transformer-based embedding model, then find the nearest neighbors by cosine similarity (or dot product, depending on whether the vectors are normalized). Documents that are semantically similar to the query land close together in the space, even when they share no surface-level words.

How It Works

Each document chunk is passed through an embedding model—typically a small transformer (100–400M parameters)—that outputs a fixed-length vector (384, 768, 1024, or 1536 dimensions are common). These vectors are stored in a vector index. At query time, the same model embeds the query, and an approximate nearest neighbor (ANN) algorithm finds the K stored vectors with the highest cosine similarity to the query vector.

Cosine similarity between the query vector q and document vector d is:

If vectors are L2-normalized (all embeddings have unit length), cosine similarity reduces to the dot product, which is faster to compute and is the default in most vector databases.

Sovereign-Compatible Tools

Recommended Embedding Models (Local)

2. Sparse Retrieval (BM25 / TF-IDF)

Sparse retrieval is the classic keyword-based approach that predates embeddings. It relies on exact term matching: if the query contains the word “indemnification,” it finds documents containing that exact word. The dominant algorithm is BM25 (Best Matching 25), an evolution of TF-IDF that incorporates term frequency saturation and document length normalization.

The BM25 Scoring Formula

Where f(t,d) is term frequency in the document, IDF(t) is inverse document frequency, |d| is document length, avgdl is average document length, and k₁ (typically 1.2–2.0) and b (typically 0.75) are tuning parameters.

Sovereign-Compatible Tools

3. Hybrid Search

Hybrid search combines dense and sparse retrieval to capture the strengths of both. It runs both retrieval pipelines in parallel, merges their results, and returns a unified ranking. This is the recommended default for production RAG—it costs slightly more compute but dramatically reduces the failure modes of either approach used alone.

How to Combine Dense + Sparse

Method 1: Reciprocal Rank Fusion (RRF) — Recommended

RRF is the simplest and most robust fusion method. Instead of trying to normalize incomparable score scales (cosine similarity is 0–1; BM25 is unbounded), RRF uses only the rank of each document in each result list. A document ranked #1 in dense results and #3 in sparse results gets a higher fused score than one ranked #5 and #2.

Where R is the set of retrieval systems, rank_r(d) is the rank of document d in system r’s results, and k is a constant (typically 60) that dampens the influence of very high ranks. RRF is parameter-light, robust, and requires no score calibration—making it the default in production systems.

Method 2: Weighted Score Blending

Normalize each retriever’s scores to a 0–1 range, then take a weighted sum:

The weight α (typically 0.5–0.7) controls the balance. This method can outperform RRF when you have tuned the weights on your data, but it requires score normalization and is sensitive to the normalization method. Min-max and softmax normalization are common choices.

Method 3: Cascade (Two-Stage)

Use one retriever as a first-stage filter and the other as a refiner. For example, run sparse retrieval first to find all documents containing the exact key terms (high recall on identifiers), then run dense similarity only on that filtered set. Or invert it: use dense retrieval to find top-100 semantically relevant documents, then re-score those 100 with BM25 for exact-term precision. The cascade approach reduces the search space for the second stage, lowering latency at the cost of potentially missing documents the first stage filtered out.

4. Re-Ranking with Cross-Encoders

Re-ranking is a two-stage retrieval pattern. In stage one, a fast bi-encoder (the standard embedding model) retrieves a broad set of top-K candidates (e.g., 50–100 documents). In stage two, a more accurate but slower cross-encoder scores each query–document pair jointly and re-orders the candidates.

Bi-Encoder vs. Cross-Encoder

The cross-encoder sees both the query and document simultaneously in a single forward pass, allowing it to model fine-grained token-level interactions (attention between query tokens and document tokens). This makes it significantly more accurate than a bi-encoder, which must compress each into a single vector before comparison. The trade-off is that cross-encoders cannot pre-compute document representations—every query requires a fresh forward pass for every candidate document.

Recommended Cross-Encoder Models (Local / Sovereign)

5. Query Transformation

Users are bad at writing search queries. They ask vague questions, use different vocabulary than the documents, combine multiple questions into one, or phrase things in ways no embedding model expects. Query transformation rewrites the user’s query before retrieval to improve what gets found. The LLM does the rewriting; the retrieval system runs on the improved query.

5.1 Sub-Query Decomposition

Complex multi-part questions are decomposed into simpler sub-queries. Each sub-query is run independently through the retriever, and results are merged. This dramatically improves recall for questions that span multiple documents or topics.

5.2 Query Expansion (Synonym Injection)

The LLM or a thesaurus adds synonyms and related terms to the query before retrieval. This is especially helpful for sparse retrieval, which depends on exact term matching. A query for “automobile insurance” can be expanded to include “car coverage,” “vehicle policy,” and “motor protection.”

5.3 Back-Translation (Paraphrasing)

Translate the query to another language (e.g., English → French → English) using the LLM or a translation model. The round-trip translation produces a paraphrase that uses different vocabulary, which may match documents the original query missed. This is a form of data augmentation that exploits the non-determinism of translation to generate alternative phrasings.

6. Query Expansion with HyDE

HyDE (Hypothetical Document Embeddings) is a clever query expansion technique that leverages the fact that embedding models work better on document-length text than on short queries. Instead of embedding the user’s query directly, you ask the LLM to generate a hypothetical answer to the question, then embed that hypothetical answer and use it for retrieval.

Why HyDE Works

Embedding models are trained on document-length text. A short query like “What are the termination penalties?” produces a sparse, low-information vector. A hypothetical answer like “The contract specifies that early termination incurs a penalty of 30% of the remaining contract value, payable within 30 days…” is rich, document-like text whose embedding lands much closer in vector space to real documents about termination penalties. The retrieval is more precise because the embedding space is being used the way it was designed to be used.

7. Step-Back Prompting

Step-back prompting is a retrieval strategy that first asks a broader, more general question to gather background context, then asks the specific question with that context in hand. It addresses the problem where a too-specific query retrieves narrow documents that lack the surrounding context needed to fully understand or answer the question.

How It Works

Given a specific user question, the LLM generates a “step-back” question—a more general version that retrieves foundational context. Both questions are run through the retriever, and their results are merged.

When Step-Back Helps

• Questions that need background: “Why did Q3 revenue drop?” benefits from first retrieving the broader “What happened in Q3?” context.
• Questions with implicit assumptions: “How fast is the new model?” benefits from first retrieving “What are the specifications of the new model?”
• Multi-hop reasoning: Questions that require connecting facts from different parts of a document set.

When Step-Back Doesn’t Help

• Simple factoid lookups where the specific question is already precise enough.
• Keyword/ID searches where broadening the query introduces noise.
• Time-sensitive questions where the broader context includes outdated information.

8. Multi-Vector Retrieval

Single-vector retrieval embeds each document chunk into one vector. Multi-vector retrieval stores multiple representations per document—for example, a summary embedding, a full-text embedding, and embeddings of key passages. At query time, the system retrieves from multiple indexes (or multiple vector fields in one index) and merges the results. This captures different aspects of relevance that a single embedding cannot.

Common Multi-Vector Strategies

The “Hypothetical Questions” Approach

An interesting multi-vector strategy: for each document, use the LLM to generate 3–5 questions that the document would answer. Embed those questions. At query time, compare the user’s query against these question embeddings. If the query is similar to a question the document answers, that document is likely relevant. This inverts the usual direction and can be very effective for Q&A corpora.

9. Contextual Compression

Contextual compression sits between retrieval and generation. After the retriever returns top-K documents, a compression step filters and extracts only the relevant portions before sending context to the LLM. This reduces token consumption, removes irrelevant noise that could distract the LLM, and improves answer quality by focusing the generation on only what matters.

Compression Approaches

1. LLM-Based Extraction

Send each retrieved document to the LLM with the instruction: “Extract only the parts relevant to this question.” The LLM returns the relevant sentences or paragraphs. This is accurate but expensive—one LLM call per retrieved document, before the main generation call.

2. Embedding-Based Filtering

Split each retrieved document into smaller units (sentences or paragraphs). Compute the embedding similarity of each unit to the query. Keep only units above a similarity threshold. No LLM call needed—just additional embedding comparisons. Faster but less nuanced than LLM extraction.

3. Document-Level Filtering

Use a lightweight classifier or the cross-encoder re-ranker score to drop entire documents below a relevance threshold. Simple, fast, and effective when the retriever returns a mix of relevant and irrelevant documents.

Tools and Implementations

10. Citation and Source Tracking

In legal, medical, and business RAG, the answer is not enough—you must be able to trace every claim back to the specific document and passage it came from. Citation and source tracking is the practice of maintaining a chain of custody from retrieved documents through to every sentence in the generated answer.

Why Citation Matters

• Legal: An attorney citing a contract clause must point to the exact section, page, and paragraph. A vague “according to the agreement” is not actionable in court.
• Medical: A clinical decision support system must cite the specific guideline, study, or drug label its recommendation is based on. Without citations, the recommendation is unreviewable and potentially dangerous.
• Business: An analyst summarizing financial documents must cite the specific filing, page, and table. Auditors require traceability.
• Trust: Users verify answers by checking sources. Without citations, the system is a black box.
• Compliance: Regulatory frameworks (EU AI Act, sector-specific regulations) increasingly require explainability and source attribution for AI-generated content.

Citation Strategies

Strategy 1: Chunk ID Tagging (Recommended)

Assign a unique ID to every chunk at ingestion time. When the retriever returns chunks, include their IDs in the context sent to the LLM. Instruct the LLM to cite the chunk ID after each claim. This is the most reliable approach because the LLM is working with explicit, structured identifiers.

Strategy 2: Post-Hoc Sentence Matching

After the LLM generates the answer, match each sentence in the answer back to the retrieved chunks using embedding similarity. The closest chunk is the cited source. This does not require the LLM to cooperate with citation instructions, but it is less precise—the matching is heuristic and can be wrong for claims that synthesize multiple sources.

Strategy 3: Span-Level Highlighting

For maximum precision, track which character spans in the original document each claim maps to. This requires the LLM or a separate model to identify the exact text span, but it enables UI features like “click to see the highlighted passage.” This is the gold standard for legal and medical applications.

What to Store for Full Traceability

11. Comparison Table

The following table summarizes every retrieval method across the dimensions that matter for production deployment. “Sovereign-compatible” means the method can be fully deployed on-premise with no external API calls.

Latency Budget by Pipeline Stage

12. Practical Recommendations

The right retrieval method depends on your data, your users, your latency budget, and your accuracy requirements. Below are concrete recommendations for common scenarios, all assuming sovereign, on-premise deployment.

Default Production Architecture

If you are building a new RAG system today and want the best balance of accuracy, latency, and simplicity, here is the recommended default:

This pipeline delivers excellent accuracy with reasonable latency (total retrieval + re-ranking in under 300ms on commodity hardware), uses only sovereign-compatible components, and provides full citation traceability. Add query transformation or HyDE if your users tend to ask vague or short questions. Add multi-vector if your documents are long and structurally diverse.

Conclusion

Retrieval method selection is the most impactful architectural decision in a RAG system. Dense vector retrieval handles semantics; sparse retrieval handles exact matches; hybrid search combines both. Cross-encoder re-ranking sharpens precision. Query transformation, HyDE, and step-back prompting improve what gets retrieved. Multi-vector captures multiple aspects of relevance. Contextual compression cleans up what reaches the LLM. Citation tracking ensures accountability.

For sovereign, on-premise deployments, every technique described here has a fully local implementation. The only tools to avoid are cloud-hosted retrieval and re-ranking APIs (Pinecone, Cohere, Voyage) that require sending your data to external servers. The open-source ecosystem now matches or exceeds these cloud services in quality, making fully sovereign RAG not just possible but practical.

Start with hybrid search and cross-encoder re-ranking. Add stages as your accuracy requirements demand. Measure everything. Cite everything. Build for the users who will rely on these answers in high-stakes decisions.