Embedding Models for Sovereign RAG: Local Over Cloud

1. What Embeddings Are

An embedding is a vector — an ordered list of floating-point numbers — that represents the semantic content of a piece of text, an image, or an audio clip. The key property is this: semantically similar inputs produce similar vectors. Two sentences about invoice processing produce vectors that point in roughly the same direction; a sentence about invoice processing and a sentence about baseball produce vectors that point in very different directions.

Concretely, the sentence “How do I refund a customer?” might embed to a 384-dimensional vector like [0.0231, -0.0844, 0.1021, ...], and the sentence “What is the refund process?” embeds to a vector whose 384 values sit very close to that first vector under cosine similarity. A sentence about quarterly tax filings embeds far away from both. The embedding model has learned, through training on massive text corpora, to arrange semantic meaning into geometric space.

This geometric arrangement is the entire foundation of RAG retrieval. When a user asks a question, you embed the question into a vector, then search your vector database for the document chunks whose vectors sit closest to the question vector. Those nearest neighbors are the chunks most likely to contain the answer. No embedding model, no RAG.

An embedding model takes raw text (or image pixels, or audio spectrograms) as input and produces a fixed-length vector as output. The dimension of that vector is a property of the model — 384, 768, 1024, 1536, or larger. The model is typically a transformer encoder (like BERT) with a pooling layer that collapses the per-token hidden states into a single sentence-level vector.

The math of similarity is almost always cosine similarity: the cosine of the angle between two vectors. Two identical vectors have cosine similarity 1.0; orthogonal vectors have similarity 0.0; opposite vectors have -1.0. For RAG retrieval you typically retrieve the top-k chunks above some similarity threshold — say, 0.75 or higher.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')

query = "How do I refund a customer?"
docs = [
    "To process a refund, open the customer record and click Refund.",
    "Quarterly tax filings are due on the 15th of April, June, September, January.",
    "Refunds can be issued within 30 days of the original purchase date."
]

query_vec = model.encode(query)
doc_vecs = model.encode(docs)

similarities = model.similarity(query_vec, doc_vecs)
# [0.82, 0.21, 0.79] -- first and third docs are semantically close

That is the entire mechanism. Everything else in this article is about which model produces the best vectors for your data, your hardware, and your sovereignty constraints.

2. Why Local Embeddings Matter

Here is the uncomfortable truth about cloud embedding APIs: when you call openai.embeddings.create() or cohere.embed(), you are sending your raw text over the internet to a third party's servers, where it is processed and returned as vectors. That text might be customer support tickets, internal HR policies, medical records, legal contracts, source code, financial filings, or anything else you are building RAG over. Every document you embed through a cloud API leaves your building.

For a hobby project or a marketing chatbot on public documentation, that trade-off is fine. For a hospital building a RAG system over patient records, a law firm building RAG over privileged client documents, a bank building RAG over confidential filings, or any organization subject to GDPR, HIPAA, SOC 2, or ITAR — that data leaving the building is a compliance violation, a security incident, or both.

Local embedding models close that leak. The model weights live on your server or GPU, the text never leaves your network, and the vectors are produced in your own infrastructure. The trade-offs are:

• Hardware cost: You need a machine with enough RAM or VRAM to hold the model. For small embeddings (all-MiniLM-L6-v2), that is 120 MB of RAM on a Raspberry Pi. For large embeddings (bge-large-en), that is ~1.3 GB of VRAM on a GPU. These are trivial requirements compared to running a 70B LLM.
• Latency: Local inference on a modest GPU is faster than a network round-trip to an API. On CPU only, small models embed in single-digit milliseconds per chunk. You are not paying for network latency.
• Throughput: A single mid-range GPU (RTX 4090, A10) can embed hundreds to thousands of chunks per second with a small model. Cloud APIs rate-limit you.
• Cost: Local is effectively free after the hardware. Cloud APIs charge per token, and re-embedding a million-document corpus after a model upgrade is a real bill.
• Reproducibility: A pinned local model produces identical vectors forever. Cloud APIs can silently update their model behind the same API name, breaking your stored vectors against new query vectors.

The last point is more important than people realize. If OpenAI updates text-embedding-3-small — which they can do at any time without notice — your stored document embeddings and your fresh query embeddings may come from different model versions, and retrieval quality silently degrades. With a local model pinned to a specific HuggingFace revision, this cannot happen.

3. Local Embedding Model Families

The open-source embedding landscape has matured dramatically since 2023. There are now several families of locally-runnable embedding models that match or exceed cloud API quality on standard benchmarks. Below is each family in depth.

3.1 sentence-transformers (SBERT Family)

The sentence-transformers library, built on top of HuggingFace Transformers, is the workhorse of local embeddings. The models in this family are collectively called SBERT (Sentence-BERT), after the original 2019 paper by Reimers & Gurevych. The library makes loading, encoding, and evaluating embedding models a one-liner.

The most widely used SBERT-family models are:

• all-MiniLM-L6-v2 — 23M parameters, 384 dimensions, 256-token context. The default choice for most projects. Embeds ~14,000 sentences per second on an RTX 3080. Good enough for most general-purpose RAG. The model you reach for when you do not have a specific reason to reach for something else.
• all-mpnet-base-v2 — 109M parameters, 768 dimensions, 384-token context. Better retrieval quality than MiniLM at the cost of roughly 4× the inference time. The model you reach for when MiniLM's recall is not good enough and you have the VRAM.
• multi-qa-mpnet-base-dot-v1 — specifically tuned for question-answer retrieval (asymmetric search: short query, longer answer). 109M parameters, 768 dimensions. Use this when your queries are questions and your corpus is answer-shaped documents.
• paraphrase-multilingual-MiniLM-L12-v2 — 117M parameters, 384 dimensions, supports 50+ languages. The multilingual workhorse if you need cross-lingual retrieval without a dedicated multilingual model.

from sentence_transformers import SentenceTransformer

# The default workhorse -- fast, lightweight, good enough for most projects
model = SentenceTransformer('all-MiniLM-L6-v2')

embeddings = model.encode([
    "The invoice is overdue by 14 days.",
    "Payment was received on March 3rd.",
    "The refund policy allows returns within 30 days."
], normalize_embeddings=True)

# embeddings.shape == (3, 384)
# Each row is a unit vector; cosine similarity is a simple dot product

The SBERT family is the most battle-tested. If you are starting a RAG project today and have no specific constraints, start with all-MiniLM-L6-v2 and upgrade only if retrieval quality is insufficient.

3.2 BGE (BAAI General Embedding)

BGE is developed by the Beijing Academy of Artificial Intelligence (BAAI) — the Chinese Academy of Sciences affiliated research institute. The BGE family has consistently topped the MTEB leaderboard since 2023 and is released under a permissive open license (MIT for the small/base models, custom-but-commercial-friendly for large).

The English BGE family comes in three sizes:

• bge-small-en-v1.5 — 33M parameters, 384 dimensions, 512-token context. Direct competitor to all-MiniLM-L6-v2. Slightly better MTEB scores in most categories. Same VRAM footprint.
• bge-base-en-v1.5 — 109M parameters, 768 dimensions, 512-token context. Direct competitor to all-mpnet-base-v2. Generally outperforms it on retrieval tasks.
• bge-large-en-v1.5 — 335M parameters, 1024 dimensions, 512-token context. The flagship. Highest MTEB retrieval scores among open models in its weight class. Requires ~1.3 GB VRAM.

BGE models require a specific query prefix for retrieval: queries must be prepended with the string "Represent this sentence for searching relevant passages: " while documents are embedded without the prefix. This is an instruction-tuned convention — forget it and your retrieval quality drops noticeably.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')

# IMPORTANT: BGE requires this query prefix for retrieval
query_prefix = "Represent this sentence for searching relevant passages: "
query = "How do I process a refund?"
query_vec = model.encode(query_prefix + query, normalize_embeddings=True)

# Documents are embedded WITHOUT the prefix
doc_vecs = model.encode(documents, normalize_embeddings=True)

BGE also offers bge-m3 — a multilingual, multi-granularity, multi-function model supporting 100+ languages with an 8192-token context window. This is the model to reach for if you need both long context and multilingual support in a single model.

3.3 E5 (EleutherAI / Microsoft)

The E5 family — originally from Microsoft Research, with the intfloat HuggingFace organization being the primary release channel — is trained with a contrastive learning objective that is particularly strong on cross-lingual retrieval. The family comes in three sizes:

• small — 33M parameters, 384 dimensions. The lightweight cross-lingual option.
• base — 109M parameters, 768 dimensions. Balanced.
• large — 335M parameters, 1024 dimensions. Strongest in the family.

Like BGE, E5 requires a query/document prefix convention. Queries are prefixed with "query: " and documents with "passage: ". Forgetting this convention degrades retrieval meaningfully — the model was trained to distinguish the two roles.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('intfloat/multilingual-e5-large')

# E5 convention: prefix queries and passages differently
query_vec = model.encode("query: " + user_question, normalize_embeddings=True)
doc_vecs = model.encode(["passage: " + doc for doc in documents], normalize_embeddings=True)

The standout in this family is multilingual-e5-large, which supports 100+ languages and consistently outperforms cloud multilingual embedding APIs on non-English retrieval. If your corpus is multilingual, this is your starting point.

3.4 Nomic Embed

Nomic Embed (nomic-embed-text-v1 and v1.5) is a fully open-source model from Nomic AI with a notable feature: an 8192-token context window. This is 16× the context length of standard SBERT/BGE/E5 models (512 tokens). For RAG over long documents — legal contracts, technical manuals, research papers — this means fewer chunks, less chunk-boundary information loss, and richer per-chunk context.

Nomic Embed v1.5 produces 768-dimensional vectors and is fully reproducible: the training data, training code, and model weights are all open. On MTEB it is competitive with OpenAI's text-embedding-3-small, which is remarkable given that it runs on a single consumer GPU with ~550 MB of VRAM.

A unique feature of Nomic Embed v1.5 is matryoshka embeddings: the 768-dimensional vector can be truncated to 512, 256, 128, or 64 dimensions with only modest quality loss. This lets you trade storage for quality at the vector database level without re-embedding your corpus — you store the full 768-dim vector and truncate at query time, or truncate at storage time to save space.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('nomic-ai/nomic-embed-text-v1.5', trust_remote_code=True)

# Nomic convention: prefix queries and documents
query_vec = model.encode("search_document: " + query, normalize_embeddings=True)
doc_vecs = model.encode(["search_document: " + d for d in documents], normalize_embeddings=True)

# Matryoshka truncation: truncate to 256 dimensions for smaller storage
truncated = doc_vecs[:, :256]
# Renormalize after truncation
import numpy as np
truncated = truncated / np.linalg.norm(truncated, axis=1, keepdims=True)

3.5 GTE (General Text Embeddings)

GTE is developed by the Alibaba DAMO Academy and released under a permissive license. The family — gte-small, gte-base, gte-large — mirrors the BGE/E5 sizing convention and is particularly strong on multilingual retrieval, with the multilingual variant (gte-multilingual) covering 70+ languages.

The English GTE models come in three sizes:

• gte-small — 33M parameters, 384 dimensions, 512-token context.
• gte-base — 109M parameters, 768 dimensions, 512-token context.
• gte-large — 335M parameters, 1024 dimensions, 512-token context.

GTE models do not require a query prefix — they are trained without the asymmetric query/passage distinction, which simplifies your pipeline. They are a strong drop-in alternative to BGE if you want to avoid the prefix convention.

3.6 Arctic Embed (Snowflake)

Arctic Embed is Snowflake's entry into the embedding model space, released under Apache 2.0. The family — arctic-embed-l, arctic-embed-m, arctic-embed-s — is specifically optimized for retrieval tasks rather than general-purpose semantic similarity. Snowflake trained these models on a large corpus of query-document pairs with a contrastive objective tuned for retrieval.

• arctic-embed-l — 335M parameters, 1024 dimensions, 512-token context. Top-tier MTEB retrieval scores.
• arctic-embed-m — 109M parameters, 768 dimensions. Balanced.
• arctic-embed-s — 33M parameters, 384 dimensions. The lightweight option.

Arctic Embed models are notable for punching above their weight on retrieval-specific MTEB tasks while being smaller than BGE-large. Apache 2.0 is the most permissive license in this list — no restrictions on commercial use, no attribution requirements beyond the standard notice.

3.7 Stella

Stella is a newer family of embedding models that has appeared on the MTEB leaderboard with competitive scores while being efficient at inference. The Stella models are particularly notable for achieving high retrieval quality at smaller parameter counts — the Stella_en_v5 family includes a 1.5B parameter variant that rivals much larger models, plus smaller variants for resource-constrained deployments.

Stella is a good choice when you want state-of-the-art MTEB scores but do not want the operational complexity of BGE's query prefix convention. The models support up to 8192-token context in the larger variants.

4. Model Size vs. Quality

Almost every embedding family ships in three sizes: small, base, and large. The trade-off is consistent across families — more parameters mean better retrieval quality at the cost of VRAM, inference latency, and storage.

The practical takeaway: small models are good enough for most RAG systems. The retrieval quality difference between small and large is real but modest — a few points of nDCG. Whether that matters depends on your corpus and your accuracy requirements. A small business Q&A bot does not need large embeddings; a medical literature search system might.

VRAM matters less than you might think for embeddings. A 335M-parameter model in FP16 is 1.3 GB — trivial on any modern GPU. You can run the entire BGE-large model on a 4 GB GPU with room to spare. The VRAM pressure in RAG comes from the LLM, not the embedding model.

If you are CPU-only, small models (33M parameters) embed in ~5–10 ms per chunk on a modern server CPU. Base models take ~20–40 ms. Large models take ~60–120 ms. For batch-embedding a million-document corpus, that is hours on CPU vs. minutes on GPU — but for real-time query embedding (one query at a time), even CPU is fine.

5. Dimensionality: Storage & Speed

The dimension of an embedding vector — 384, 768, 1024, 1536 — affects two things: storage in your vector database and retrieval speed at query time. Both scale linearly with dimension.

Storage impact

For a corpus of 1 million document chunks, each stored as a float32 vector:

• 384 dimensions: 1,000,000 × 384 × 4 bytes = 1.54 GB
• 768 dimensions: 1,000,000 × 768 × 4 bytes = 3.07 GB
• 1024 dimensions: 1,000,000 × 1024 × 4 bytes = 4.10 GB
• 1536 dimensions: 1,000,000 × 1536 × 4 bytes = 6.14 GB

These are just the vector bytes — your vector database adds index overhead (typically 1.5× to 3× the raw vector size for an HNSW index). At 10 million chunks, a 1536-dimensional corpus needs ~60 GB of raw vector storage plus ~120 GB of HNSW index. At 100 million chunks, you need a serious vector database infrastructure.

Retrieval speed

Cosine similarity is a dot product followed by a normalization. The dot product is O(d) where d is the dimension. A 1024-dim similarity is 2.67× slower than a 384-dim similarity. For brute-force (flat) search, retrieval time scales linearly with dimension. For HNSW or IVF indexes, the dimension affects the per-comparison cost but not the number of comparisons — so the practical speed difference is smaller than the storage difference but still meaningful at scale.

Quantization: half the storage for free

Most vector databases support binary quantization (1 bit per dimension) or scalar quantization (8-bit int per dimension). Binary quantization reduces storage 32× with a modest quality loss that can be recovered with a re-scoring pass over the top-k candidates using the full-precision vectors. For a 1024-dim corpus at 100M vectors, binary quantization takes you from ~400 GB to ~12.5 GB — the difference between “needs a dedicated vector database server” and “fits in memory on a normal machine.”

The practical recommendation: choose the dimension that matches your quality needs, then apply quantization and/or matryoshka truncation to fit your storage budget. Do not choose a smaller model just to save storage — choose the model with the quality you need, then optimize storage at the vector database layer.

6. Context Length: Why It Matters for RAG

The context length of an embedding model is the maximum number of tokens it can encode into a single vector. Text beyond this length is either truncated (losing the tail) or must be split into multiple chunks (creating multiple vectors). For RAG, context length directly affects your chunking strategy.

Why does longer context matter? Consider a 20-page legal contract (~6,000 words). With a 512-token model, you split it into ~15 chunks. Each chunk embeds independently, and the boundary between chunks loses cross-paragraph context. A clause on page 7 that references “the obligations described in Section 4.2” loses that reference if Section 4.2 is in a different chunk. With an 8192-token model, the entire contract is a single vector — retrieval can surface the whole document as one unit.

The trade-off: longer context means each vector covers more text, which can dilute the specificity of retrieval. A query about “indemnification clauses” might match a long chunk that contains one indemnification clause buried among 6,000 words of other content. For precise retrieval, shorter chunks often work better. The right answer is usually chunking with overlap — split into 512-token chunks with 50–100 tokens of overlap between adjacent chunks, regardless of your model's maximum context.

7. Specialized Domain Embeddings

General-purpose embedding models are trained on broad web corpora. If your RAG system covers a specialized domain — medical literature, legal contracts, source code — a domain-specific embedding model will often outperform a general model of the same size. The domain-specific model has seen the vocabulary, syntax, and semantic patterns of your corpus during training.

7.1 Code embeddings

For RAG over source code repositories, code-documentation, or technical Q&A:

• CodeBERT — Microsoft, 125M parameters. Trained on CodeSearchNet (6 programming languages). Produces embeddings that capture code semantics, not just syntax. Good for code-to-code and code-to-comment similarity.
• UniXcoder — Microsoft, 125M parameters. Multilingual code understanding. Supports code, comment, and identifier embedding in a unified space. Better than CodeBERT on most benchmarks.
• jina-embeddings-v2-code-en — Jina AI, 137M parameters, 768 dimensions. Specifically trained for code retrieval. Supports 8192-token context — long enough for entire source files.
• codeberta-base — HuggingFace, smaller and faster. Good for resource-constrained deployments.

from sentence_transformers import SentenceTransformer

# Code-specific embedding model
model = SentenceTransformer('jinaai/jina-embeddings-v2-base-code-en', trust_remote_code=True)

# Embed a function and a natural-language description of it
code = "def calculate_refund(order_total, days_since_purchase): return order_total * (1 - min(days_since_purchase / 30, 1))"
desc = "Calculate the refund amount based on how long since the purchase was made"

code_vec = model.encode(code)
desc_vec = model.encode(desc)
# These vectors will be closer together than with a general-purpose model

7.2 Medical embeddings

For RAG over medical literature, clinical notes, or biomedical databases:

• BioBERT — DMIS Lab, 110M parameters. Pre-trained on PubMed abstracts and PMC full-text articles. The foundation model for most medical embedding work.
• PubMedBERT — Microsoft, 110M parameters. Trained from scratch on PubMed (not fine-tuned from BERT). Better on biomedical entity recognition and relation extraction.
• BioBERT-based sentence-transformers — Several fine-tuned variants of BioBERT are available on HuggingFace for sentence embedding tasks. emilyalsentzer/Bio_ClinicalBERT is a clinical-note-specific variant.

Use these when your corpus is dominated by medical text with specialized terminology (“myocardial infarction,” “contraindicated,” “pharmacokinetics”) that general-purpose models embed poorly. The vocabulary mismatch between general models and medical text can cause dramatic retrieval failures.

7.3 Legal embeddings

For RAG over contracts, statutes, case law, or regulatory documents:

• LegalBERT — 110M parameters. Pre-trained on legal corpora (UK legislation, EU legislation, contracts). Better at legal vocabulary and long-sentence structure.
• InLegalBERT — trained on Indian legal corpora. Better for Commonwealth jurisdictions.
• CaseLawBERT — trained on US case law. Better for US jurisdictions.

Legal text has unique characteristics: extremely long sentences (200+ words), archaic vocabulary (“hereinafter,” “notwithstanding,” “in witness whereof”), and precise terminology where small word changes matter. A general embedding model often conflates “shall” and “may” or fails to distinguish “indemnify” from “hold harmless.” LegalBERT-based embeddings reduce these errors.

7.4 Multilingual embeddings

For RAG over corpora in multiple languages, or where query language differs from document language:

• LaBSE (Language-agnostic BERT Sentence Embedding) — Google, 471M parameters, 768 dimensions. Supports 109 languages. Strong cross-lingual alignment: a query in French retrieves relevant documents in English with high accuracy.
• distiluse-base-multilingual-cased-v2 — 135M parameters, 512 dimensions. Supports 50+ languages. Smaller and faster than LaBSE.
• paraphrase-multilingual-MiniLM-L12-v2 — 118M parameters, 384 dimensions. The lightweight multilingual option.
• multilingual-e5-large — 560M parameters, 1024 dimensions. The strongest multilingual retrieval model. 100+ languages.
• bge-m3 — 568M parameters, 1024 dimensions. 100+ languages, 8192-token context. The “do everything” multilingual model.

8. Image Embedding Models

For multi-modal RAG — systems that retrieve images alongside or instead of text — you need image embedding models. These produce vectors from image pixels that capture visual semantics: similar images produce similar vectors.

8.1 CLIP (OpenAI, open weights)

CLIP (Contrastive Language-Image Pre-training) is the foundational multi-modal embedding model. It produces embeddings in a shared text-image space: an image of a cat and the text “a photo of a cat” produce similar vectors. This makes CLIP uniquely useful for cross-modal retrieval — search images with text queries, or search text with image queries.

• CLIP ViT-B/32 — 151M parameters, 512 dimensions. The standard workhorse.
• CLIP ViT-L/14 — 428M parameters, 768 dimensions. Better quality, heavier.
• CLIP ViT-H/14 — 986M parameters, 1024 dimensions. Highest quality in the original CLIP family.

from transformers import CLIPModel, CLIPProcessor
from PIL import Image

model = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')

# Text and image in the SAME embedding space
texts = ["a photo of a cat", "a diagram of a neural network"]
images = [Image.open("cat.jpg"), Image.open("network_diagram.png")]

text_inputs = processor(text=texts, return_tensors="pt", padding=True)
image_inputs = processor(images=images, return_tensors="pt")

text_embeds = model.get_text_features(**text_inputs)
image_embeds = model.get_image_features(**image_inputs)

# cosine_similarity(text_embeds[0], image_embeds[0]) should be high -- both are "a cat"

8.2 OpenCLIP

OpenCLIP is the open-source reproduction and extension of CLIP, trained on larger and more diverse datasets (LAION-2B, LAION-400M). The OpenCLIP models generally outperform the original OpenAI CLIP models on standard benchmarks, especially on out-of-distribution images. Available in ViT-B/32, ViT-L/14, ViT-H/14, and ViT-G/14 sizes.

8.3 DINOv2 (Meta, vision-only)

DINOv2 is a self-supervised vision transformer from Meta that produces state-of-the-art image embeddings for pure-image similarity tasks. Unlike CLIP, DINOv2 does not have a text encoder — it is vision-only. But for image-to-image similarity retrieval (find images that look similar to this image), DINOv2 is the strongest open model available.

• ViT-S/14 — 22M parameters, 384 dimensions.
• ViT-B/14 — 86M parameters, 768 dimensions.
• ViT-L/14 — 300M parameters, 1024 dimensions.
• ViT-g/14 — 1.1B parameters, 1536 dimensions. The flagship.

For a multi-modal RAG system that retrieves both images and text, the typical architecture is: use CLIP for cross-modal retrieval (text-to-image and image-to-text) and DINOv2 for image-to-image similarity. Store both vectors per image and query against the appropriate index.

9. Audio Embedding Models

For RAG over audio content — podcasts, meeting recordings, customer support calls — you have two paths: (a) transcribe the audio with a speech-to-text model, then embed the transcript as text; or (b) embed the audio directly with an audio embedding model. Path (a) is simpler and uses the same text embedding pipeline you already have. Path (b) preserves audio-specific information (speaker identity, emotion, prosody) that transcription loses.

9.1 CLAP (Microsoft)

CLAP (Contrastive Language-Audio Pre-training) is the audio analog of CLIP. It produces embeddings in a shared text-audio space: an audio clip of a dog barking and the text “a dog barking” produce similar vectors. This enables text-to-audio and audio-to-text retrieval without transcription.

• clap-htsat-fused — 150M parameters, 512 dimensions. The standard model.
• clap-htsat-unfused — separate audio and text towers, slightly more flexible.

from transformers import ClapModel, ClapProcessor
import librosa

model = ClapModel.from_pretrained('laion/clap-htsat-fused')
processor = ClapProcessor.from_pretrained('laion/clap-htsat-fused')

# Text query retrieves matching audio clips
audio, sr = librosa.load("meeting_recording.wav", sr=48000)
text_query = "discussion about the Q3 budget"

audio_inputs = processor(audios=audio, return_tensors="pt", sampling_rate=sr)
text_inputs = processor(text=text_query, return_tensors="pt")

audio_embed = model.get_audio_features(**audio_inputs)
text_embed = model.get_text_features(**text_inputs)
# cosine_similarity(text_embed, audio_embed) tells you if this clip matches

9.2 Wav2Vec2 (Meta, speech features)

Wav2Vec2 is a self-supervised speech representation model that produces embeddings from raw audio. Unlike CLAP, Wav2Vec2 is not aligned with text — its embeddings capture speech features (phonemes, speaker characteristics, prosody) but not semantic content. Use Wav2Vec2 when you need to retrieve audio by audio similarity (find other clips with the same speaker, same accent, same emotional tone) rather than by semantic content.

For most RAG applications, the practical recommendation is: transcribe audio with Whisper (local, open-source), embed the transcript with a text embedding model. This gives you text-searchable audio RAG with the full power of text embedding models. Use CLAP or Wav2Vec2 only when you specifically need audio-native features.

10. Benchmarking: MTEB & Beyond

The Massive Text Embedding Benchmark (MTEB) is the standard evaluation framework for text embedding models. It evaluates models across 8 task categories in 58 datasets covering 112 languages:

• Retrieval — the most important category for RAG. Given a query, retrieve relevant documents from a large corpus.
• Reranking — re-order a candidate set of documents by relevance to a query.
• Classification — classify text into categories using embedding similarity.
• Clustering — cluster texts by semantic similarity.
• Pair classification — determine if two texts are semantically equivalent.
• STS (Semantic Textual Similarity) — score the similarity of two sentences.
• Summarization — rank summaries by quality relative to source text.
• Bitext mining — find translation pairs across languages.

For RAG, the Retrieval score is the one that matters most. The MTEB retrieval score is an average of nDCG@10 across multiple retrieval datasets (MS MARCO, TREC-COVID, NFCorpus, HotpotQA, FiQA, ArguAna, Quora, SCIDOCS, SciFact, and others). A model with a retrieval score of 65 is not “5% better” than one with a score of 60 — it is meaningfully better at the specific task your RAG system depends on.

The MTEB leaderboard is available at huggingface.co/spaces/mteb/leaderboard and is actively maintained. As of mid-2026, the top local models on retrieval are:

1. bge-large-en-v1.5 — ~64.2 retrieval
2. arctic-embed-l — ~64.0 retrieval
3. stella_en_v5 — ~63.8 retrieval
4. nomic-embed-text-v1.5 — ~63.0 retrieval
5. bge-base-en-v1.5 — ~63.0 retrieval
6. gte-large — ~62.6 retrieval
7. all-mpnet-base-v2 — ~57.9 retrieval
8. all-MiniLM-L6-v2 — ~56.3 retrieval

These numbers shift as new models are released, but the relative ordering of the model families has been stable for over a year.

Running your own benchmark

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics import ndcg_score

# Your labeled evaluation set: query -> list of (doc_id, relevance_score)
eval_set = [
    {"query": "How do I process a refund?", "relevant": [12, 47, 89], "irrelevant": [3, 22, 156]},
    {"query": "What is the late payment penalty?", "relevant": [34, 78], "irrelevant": [12, 91]},
    # ... 100-500 more
]

candidate_models = [
    "all-MiniLM-L6-v2",
    "BAAI/bge-base-en-v1.5",
    "BAAI/bge-large-en-v1.5",
    "nomic-ai/nomic-embed-text-v1.5",
    "Snowflake/snowflake-arctic-embed-l-v2.0",
]

for model_name in candidate_models:
    model = SentenceTransformer(model_name, trust_remote_code=True)
    
    # Embed all documents in your corpus
    doc_ids = list(all_document_ids)
    doc_texts = [documents[did] for did in doc_ids]
    doc_vecs = model.encode(doc_texts, normalize_embeddings=True)
    
    ndcg_scores = []
    for item in eval_set:
        query_vec = model.encode(item["query"], normalize_embeddings=True)
        similarities = (doc_vecs @ query_vec).tolist()
        
        # Build relevance labels: 3 for relevant, 0 for irrelevant
        relevance = []
        for i, did in enumerate(doc_ids):
            if did in item["relevant"]:
                relevance.append(3)
            elif did in item["irrelevant"]:
                relevance.append(0)
            else:
                relevance.append(1)  # unlabeled, assume neutral
        
        ndcg = ndcg_score([relevance], [similarities], k=10)
        ndcg_scores.append(ndcg)
    
    print(f"{model_name}: nDCG@10 = {np.mean(ndcg_scores):.3f}")

This is the only evaluation that tells you which model to use. Everything else is a proxy.

11. Master Comparison Table

The following table covers every model discussed in this article. All VRAM figures are FP16 inference; CPU RAM is roughly 2×. MTEB retrieval scores are approximate averages across MTEB's English retrieval subset and shift over time — treat them as comparative indicators, not absolute ground truth. License abbreviations: Apache 2.0 (Apache), MIT (MIT), custom-but-commercial-friendly (Custom-CC).

12. Practical Recommendations

Below are concrete model recommendations by use case. These are starting points — always validate on your own data as described in the benchmarking section.

12.1 Small business general Q&A

Use: all-MiniLM-L6-v2

Why: For a RAG system over a few thousand documents — internal wiki, product catalog, FAQ — the retrieval quality of MiniLM is more than sufficient. It runs on CPU, costs nothing, and embeds fast enough for real-time query. You almost certainly do not need a larger model.

Upgrade to: bge-base-en-v1.5 if you notice retrieval misses on queries that use different vocabulary than your documents (e.g., users search “cancel subscription” but documents say “terminate recurring billing”).

12.2 Medical RAG

Use: BioBERT-based sentence-transformer, or bge-large-en-v1.5 fine-tuned on medical text

Why: Medical vocabulary is specialized enough that general models miss common medical semantic relationships. BioBERT has seen PubMed and understands that “myocardial infarction” and “heart attack” refer to the same concept, while general models may not align them as strongly.

For clinical notes specifically: Bio_ClinicalBERT, which is further fine-tuned on MIMIC-III clinical notes.

12.3 Legal RAG

Use: LegalBERT-based sentence-transformer, or bge-large-en-v1.5 with long-context chunking

Why: Legal text has unique structure — long sentences, precise terminology, cross-references within and across documents. LegalBERT handles the vocabulary; long-context chunking (nomic-embed-text-v1.5 or bge-m3 with 8192-token context) preserves cross-clause references that 512-token chunking loses.

For US case law: CaseLawBERT, trained specifically on US judicial opinions.

12.4 Multilingual RAG

Use: multilingual-e5-large or bge-m3

Why: Both models support 100+ languages with strong cross-lingual alignment. A query in Japanese retrieves documents in English; a query in Arabic retrieves documents in Spanish. bge-m3 has the advantage of 8192-token context; multilingual-e5-large is slightly higher quality on retrieval benchmarks.

For lightweight multilingual: paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions, runs on CPU).

12.5 Code RAG

Use: jina-embeddings-v2-base-code-en

Why: Code-specific models understand that def calculate_refund(total, days) and function computeRefund(amount, daysSincePurchase) are semantically similar, while general models may embed them as dissimilar due to syntax differences. Jina's code model also supports 8192-token context — long enough for entire source files.

Alternative: UniXcoder for multilingual code (Python, Java, JavaScript, Go, Ruby, PHP).

12.6 Multi-modal RAG (text + images)

Use: CLIP ViT-L/14 for cross-modal retrieval, DINOv2 ViT-B/14 for image-to-image similarity

Why: CLIP lets users search images with text queries (“find the diagram showing the network topology”) and text with image queries. DINOv2 is stronger for pure visual similarity (find images that look like this one). Store both vectors per image and query against both indexes.

For resource-constrained deployments: CLIP ViT-B/32 — half the parameters, 80% of the quality.

12.7 Audio RAG

Use: Whisper (local transcription) + text embedding model

Why: For 95% of audio RAG use cases (meeting transcripts, podcast search, call center QA), the simplest path is best: transcribe with Whisper, embed the transcript with your text embedding model. This gives you text-searchable audio with the full power of text retrieval.

Use CLAP only if: you need to retrieve audio by acoustic features (find clips with the same speaker, same emotional tone, same background music) that transcription discards.

12.8 Edge / low-resource deployment

Use: all-MiniLM-L6-v2 (text), CLIP ViT-B/32 (image)

Why: Both models run on CPU at acceptable speeds. all-MiniLM-L6-v2 embeds ~100 sentences per second on a Raspberry Pi 4. CLIP ViT-B/32 encodes images in ~200 ms on the same hardware. For a RAG system on an edge device (industrial sensor gateway, offline assistant, kiosk), these are the models that fit.

For the smallest possible footprint: arctic-embed-s (22M parameters, 88 MB VRAM) — smaller than MiniLM with comparable retrieval quality.

12.9 Maximum quality, no constraints

Use: stella_en_v5_1.5b (text), OpenCLIP ViT-G/14 (image), DINOv2 ViT-g/14 (image similarity)

Why: If you have a 24 GB+ GPU and want the absolute best retrieval quality available locally, these are the state-of-the-art models. The Stella 1.5B model rivals proprietary cloud embedding APIs. OpenCLIP ViT-G/14 is the strongest open cross-modal model. DINOv2 ViT-g/14 is the strongest open vision-only model.

12.10 Long documents (research papers, manuals, contracts)

Use: nomic-embed-text-v1.5 or bge-m3

Why: 8192-token context means fewer chunks, less boundary information loss, and the ability to retrieve entire documents as single units when appropriate. Use matryoshka truncation on Nomic to trade dimension for storage at the vector database layer.

13. Running Embeddings Locally

There are three primary ways to run embedding models locally. Each has different trade-offs in terms of ease of setup, throughput, and operational complexity.

13.1 Ollama (easiest)

Ollama supports embedding models natively. It is the fastest path from zero to working embeddings — one command downloads and serves the model on localhost.

# Pull an embedding model
ollama pull nomic-embed-text

# Embed a single text
ollama embed nomic-embed-text "How do I process a refund?"

# Embed multiple texts in one call
ollama embed nomic-embed-text '["Process a refund", "Cancel a subscription", "Update payment method"]'

# The model is now served at http://localhost:11434
# Use it from any language via the REST API:
curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "How do I process a refund?"
}'

import requests

response = requests.post('http://localhost:11434/api/embeddings', json={
    'model': 'nomic-embed-text',
    'prompt': 'How do I process a refund?'
})
embedding = response.json()['embedding']
# embedding is a list of 768 floats

Ollama supports several embedding models out of the box:

• nomic-embed-text — Nomic Embed v1.5, 768 dimensions, 8192 context
• bge-m3 — BGE multilingual, 1024 dimensions, 8192 context
• mxbai-embed-large — MixedBreed.ai large, 1024 dimensions
• all-minilm — all-MiniLM-L6-v2, 384 dimensions
• snowflake-arctic-embed — Arctic Embed, 1024 dimensions

Ollama is the recommended path for teams that want embeddings running in 5 minutes without writing Python. The trade-off: Ollama is optimized for LLM serving, not high-throughput batch embedding. For initial corpus ingestion of millions of documents, sentence-transformers or a dedicated embedding server will be faster.

13.2 sentence-transformers (most flexible)

The sentence-transformers Python library is the most flexible path. It supports every model on HuggingFace, runs on CPU or GPU, and gives you direct control over batching, pooling, and normalization.

from sentence_transformers import SentenceTransformer
import torch

# Auto-detect GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer('BAAI/bge-large-en-v1.5', device=device)

# Batch embed a corpus
documents = [
    "The refund policy allows returns within 30 days.",
    "Late payments incur a 5% penalty per month.",
    # ... thousands more
]

embeddings = model.encode(
    documents,
    batch_size=64,               # tune for your GPU VRAM
    normalize_embeddings=True,   # unit vectors for cosine similarity
    show_progress_bar=True,
    convert_to_numpy=True
)

# embeddings.shape == (len(documents), 1024)
# Store in your vector database (pgvector, Qdrant, Milvus, etc.)

# Real-time query embedding (single query, low latency)
query_embedding = model.encode(
    "How do I process a refund?",
    normalize_embeddings=True
)

# Retrieve top-5 from your vector database
# (pseudo-code -- adapt to your vector DB client)
results = vector_db.search(query_embedding, top_k=5)

For batch corpus ingestion, sentence-transformers on a single RTX 4090 embeds ~1,800 documents per second with bge-large-en-v1.5 (1024 dimensions, 512 context). A 1-million-document corpus takes ~9 minutes. With all-MiniLM-L6-v2, the same GPU embeds ~14,000 documents per second — the million-document corpus takes ~71 seconds.

13.3 HuggingFace Transformers (low-level)

If you need direct access to the model's hidden states (for custom pooling strategies, layer-wise analysis, or research), use HuggingFace Transformers directly. This is more code but maximum flexibility.

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-large-en-v1.5')
model = AutoModel.from_pretrained('BAAI/bge-large-en-v1.5')
model.eval()

def mean_pooling(token_embeddings, attention_mask):
    """Collapse per-token embeddings into a single sentence vector."""
    mask = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    summed = torch.sum(token_embeddings * mask, dim=1)
    counts = torch.clamp(mask.sum(dim=1), min=1e-9)
    return summed / counts

def embed(texts):
    encoded = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt')
    with torch.no_grad():
        outputs = model(**encoded)
    sentence_embeddings = mean_pooling(outputs.last_hidden_state, encoded['attention_mask'])
    return F.normalize(sentence_embeddings, p=2, dim=1)

vectors = embed(["How do I process a refund?", "What is the late payment penalty?"])
# vectors.shape == (2, 1024)

Use this path only if sentence-transformers does not give you what you need. For 99% of RAG systems, sentence-transformers is the right abstraction layer.

13.4 Infinity (dedicated embedding server)

Infinity is an open-source embedding server from Michael Feil that exposes a REST API compatible with the OpenAI embeddings endpoint. This means any code already written for OpenAI's embedding API can point at your local Infinity server by changing one URL — zero code changes, full sovereignty.

# Install and run Infinity with BGE-large
pip install infinity-emb[all]

infinity_emb v2 --model-id BAAI/bge-large-en-v1.5 --port 7997

# Now use it exactly like the OpenAI API:
curl http://localhost:7997/v1/embeddings -H "Content-Type: application/json" -d '{
  "model": "BAAI/bge-large-en-v1.5",
  "input": "How do I process a refund?"
}'

from openai import OpenAI

# Point the OpenAI client at your local Infinity server
client = OpenAI(base_url='http://localhost:7997/v1', api_key='not-needed')

response = client.embeddings.create(
    model='BAAI/bge-large-en-v1.5',
    input='How do I process a refund?'
)
embedding = response.data[0].embedding
# 1024-dimensional vector, produced locally, no data left your network

This is the recommended path for teams migrating from cloud embedding APIs to local embeddings: deploy Infinity, change one environment variable, and you are sovereign.

14. Docker & Production Deployment

For production RAG systems, your embedding service should be a containerized microservice with health checks, metrics, and horizontal scalability. Below are two Docker Compose configurations: one using Infinity (OpenAI-compatible API) and one using Ollama.

14.1 Infinity in Docker

# docker-compose.yml -- Infinity embedding service
version: '3.8'

services:
  embeddings:
    image: michaelf34/infinity:latest
    command: v2 --model-id BAAI/bge-large-en-v1.5 --port 7997
    ports:
      - "7997:7997"
    volumes:
      - infinity-cache:/app/.cache/huggingface
    environment:
      - INFINITY_BATCH_SIZE=64
      - INFINITY_ENGINE=torch
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7997/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

volumes:
  infinity-cache:

# Start the service
docker compose up -d

# Verify it is healthy
curl http://localhost:7997/health
# {"status":"ok","model":"BAAI/bge-large-en-v1.5"}

# Embed text through the OpenAI-compatible API
curl http://localhost:7997/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"BAAI/bge-large-en-v1.5","input":"How do I process a refund?"}'

14.2 Ollama in Docker

# docker-compose.yml -- Ollama embedding service
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

volumes:
  ollama-models:

# Start the service
docker compose up -d

# Pull the embedding model inside the container
docker exec -it ollama-embeddings ollama pull nomic-embed-text

# Verify
curl http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"test"}'

14.3 Python embedding service with FastAPI

For full control over the embedding pipeline (custom pooling, post-processing, caching), a thin FastAPI service on top of sentence-transformers is the right architecture.

# embedding_service.py
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
import torch
import os

app = FastAPI(title="Sovereign Embedding Service")

MODEL_NAME = os.getenv('EMBEDDING_MODEL', 'BAAI/bge-large-en-v1.5')
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer(MODEL_NAME, device=device)

class EmbedRequest(BaseModel):
    texts: list[str]
    normalize: bool = True

class EmbedResponse(BaseModel):
    embeddings: list[list[float]]
    model: str
    dimensions: int

@app.post("/embed", response_model=EmbedResponse)
def embed(request: EmbedRequest):
    vectors = model.encode(
        request.texts,
        normalize_embeddings=request.normalize,
        batch_size=32,
        convert_to_numpy=True
    )
    return EmbedResponse(
        embeddings=vectors.tolist(),
        model=MODEL_NAME,
        dimensions=vectors.shape[1]
    )

@app.get("/health")
def health():
    return {"status": "ok", "model": MODEL_NAME, "device": DEVICE}

# Run with: uvicorn embedding_service:app --host 0.0.0.0 --port 8000

# Dockerfile
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

CMD ["uvicorn", "embedding_service:app", "--host", "0.0.0.0", "--port", "8000"]

# requirements.txt
fastapi==0.115.0
uvicorn[standard]==0.30.0
sentence-transformers==3.0.0
torch==2.4.0
numpy==1.26.0

14.4 Scaling embeddings horizontally

For high-throughput production systems (millions of documents, thousands of queries per second), run multiple embedding service instances behind a load balancer. Embedding models are stateless — any instance can embed any request — so horizontal scaling is trivial.

# docker-compose.yml -- scaled Infinity service
version: '3.8'

services:
  embeddings:
    image: michaelf34/infinity:latest
    command: v2 --model-id BAAI/bge-large-en-v1.5 --port 7997
    deploy:
      replicas: 4    # 4 instances across 4 GPUs
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7997/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  nginx:
    image: nginx:latest
    ports:
      - "7997:7997"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - embeddings
    restart: unless-stopped

# nginx.conf -- round-robin load balancing across embedding instances
upstream embedding_backend {
    least_conn;
    server embeddings_1:7997;
    server embeddings_2:7997;
    server embeddings_3:7997;
    server embeddings_4:7997;
}

server {
    listen 7997;

    location / {
        proxy_pass http://embedding_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /health {
        access_log off;
        return 200 "ok";
    }
}

With 4 instances of bge-large-en-v1.5 on 4 GPUs, your aggregate throughput is ~7,200 documents per second — enough to embed a 10-million-document corpus in ~23 minutes, or serve ~7,000 concurrent queries per second.

14.5 Embedding cache

In production, cache your embeddings. The same document embedded twice produces identical vectors — there is no reason to recompute. A simple Redis cache keyed on hash(model_name + document_text) eliminates redundant embedding work and makes re-embedding after model upgrades the only cache-miss event.

import hashlib
import redis
import json
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
cache = redis.Redis(host='localhost', port=6379, db=0)

def embed_cached(texts, model_name='BAAI/bge-large-en-v1.5'):
    results = [None] * len(texts)
    to_embed = []
    to_embed_indices = []
    
    for i, text in enumerate(texts):
        key = hashlib.sha256(f"{model_name}:{text}".encode()).hexdigest()
        cached = cache.get(key)
        if cached is not None:
            results[i] = json.loads(cached)
        else:
            to_embed.append(text)
            to_embed_indices.append(i)
    
    if to_embed:
        new_vectors = model.encode(to_embed, normalize_embeddings=True, convert_to_numpy=True)
        for idx, text, vec in zip(to_embed_indices, to_embed, new_vectors):
            key = hashlib.sha256(f"{model_name}:{text}".encode()).hexdigest()
            cache.setex(key, 86400 * 30, json.dumps(vec.tolist()))  # 30-day TTL
            results[idx] = vec.tolist()
    
    return results

15. Conclusion

Embeddings are the foundation of RAG retrieval — and for sovereign RAG, they must be local. The good news is that the open-source embedding landscape in 2026 is mature: there are local models at every size tier that match or exceed cloud embedding APIs on retrieval quality, run on trivial hardware, and keep your data inside your network.

The practical path for most teams:

1. Start with all-MiniLM-L6-v2 via Ollama or sentence-transformers. It is fast, free, and good enough for 80% of use cases.
2. Measure retrieval quality on your actual data with a hand-labeled evaluation set of 100–500 query-document pairs.
3. Upgrade to bge-base-en-v1.5, arctic-embed-m, or nomic-embed-text-v1.5 only if measurement shows MiniLM is missing relevant documents.
4. Specialize with BioBERT, LegalBERT, UniXcoder, or multilingual-e5-large if your domain has vocabulary that general models embed poorly.
5. Scale with Infinity or a custom FastAPI service behind a load balancer when throughput demands it.
6. Cache embeddings in Redis to avoid redundant work and make model upgrades the only cache-miss event.

The model you choose matters less than the discipline of measuring on your own data and the consistency of using one model for both indexing and querying. A team that benchmarks all-MiniLM-L6-v2 on their corpus and sticks with it will outperform a team that switches to bge-large-en-v1.5 without measuring, hits a query-prefix bug, and silently degrades retrieval for six months before noticing.

Sovereign RAG starts with sovereign embeddings. The models are ready. The infrastructure is ready. The only question is whether your text leaves your building — and with the models in this article, it does not have to.