Multi-Modal RAG: Audio, Video, and Image Retrieval

Multi-Modal RAG is Retrieval-Augmented Generation extended beyond text. A standard RAG system ingests documents, splits them into text chunks, embeds those chunks, stores the embeddings in a vector database, and at query time retrieves the closest chunks to ground an LLM's answer. A multi-modal RAG system does the same thing -- but the chunks are not limited to text. They can be passages from a PDF, segments of an audio recording with timestamps, frames extracted from a video with generated captions, or images with CLIP embeddings and natural-language descriptions. The retrieval layer can match a user's text query against any of these, and the generation layer can receive text, images, or audio references as context.

The key insight is that every modality -- audio, video, image -- must be reduced to a form that a text-oriented retriever and LLM can use. Audio becomes a transcript with timestamps. Video becomes a transcript plus a sequence of captioned key frames. Images become captions plus visual-feature embeddings. The original media files are never embedded directly into the LLM's context window; instead, the system stores textual and vector representations of them, retrieves those representations, and then optionally surfaces the original media to the user as a citation. This is what makes multi-modal RAG tractable on local hardware: you do not need a model that ingests raw video; you need a pipeline that converts video into retrievable text and vectors.

The architecture therefore adds two new stages to the standard RAG pipeline. Before chunking and embedding, there is a transcription and captioning stage that converts non-text media into text. Before retrieval, there is an optional cross-modal embedding stage that places different modalities into a shared vector space (CLIP for text+image, CLAP for text+audio, ImageBind for text+image+audio+video and more). These two additions -- transcription/captioning and cross-modal embedding -- are the entire difference between standard RAG and multi-modal RAG. Everything else (chunking, vector storage, retrieval, LLM generation) is reused from the text pipeline.

The Unified Retrieval Surface

A well-built multi-modal RAG system presents a single retrieval surface to the user. You ask a question in natural language -- "What did the cardiologist say about the irregular rhythm in my last visit?" -- and the system can return a text passage from the visit notes, a 12-second audio clip of the dictation, a frame from an echocardiogram video with a caption, and a DICOM image with a generated description. All four artifacts are relevant to the query, all four are retrieved by the same text query, and all four can be surfaced to the user as citations. The user does not need to know whether the answer came from text, audio, video, or images. The system retrieves across all modalities transparently.

This unified surface is the goal. Achieving it requires that every modality be reduced, at ingest time, to a text representation (for the LLM) and a vector representation (for the retriever), with a back-reference to the original media file and a timestamp or frame index so the system can play, display, or cite the original artifact at query time. The rest of this page walks through each modality's ingestion pipeline in detail, then covers the embedding models, storage architecture, and the sovereign stack that ties it all together.

Real-world data is not just text. A medical practice has dictated notes (audio), training videos for staff (video), DICOM images from imaging studies (images), and PDF lab reports (text). A business has recorded Zoom meetings (video+audio), training videos (video), slide decks (images+text), and contracts (text). A school has lecture recordings (video+audio), student presentation videos (video), scanned worksheets (images), and syllabi (text). A family has voice memos (audio), home videos (video), photos (images), and financial documents (text). A RAG system that can only ingest text is blind to the majority of the information these organizations and people actually hold.

The practical consequence is that a text-only RAG system silently discards most of your knowledge base. If your cardiologist dictated a 3-minute note about an irregular rhythm, that note exists only as audio. A text-only RAG system cannot answer "what did the cardiologist say about my rhythm?" because the answer is not in any text document -- it is in a WAV file. If your company recorded a 90-minute strategy meeting, the decision to pivot a product line exists only in that recording. A text-only RAG system cannot retrieve it. The information is in your possession, but your AI cannot see it.

Why Sovereign Multi-Modal Matters Even More

Multi-modal data raises the stakes of sovereignty. Audio recordings of patient visits are PHI under HIPAA. Video recordings of business strategy meetings are proprietary. Medical images are among the most sensitive data in any organization. Family photos and voice memos are the most personal artifacts in a household. Sending any of these to a cloud transcription API, a cloud image-captioning API, or a cloud embedding API is a far more severe exposure than sending text. You are transmitting voice, faces, and medical imagery to a third party whose logging, retention, and training practices you cannot audit.

A sovereign multi-modal RAG system transcribes audio with a local Whisper model, captions images with a local LLaVA or BLIP model, extracts visual features with a local CLIP model, and stores everything on hardware you control. No audio file leaves your network. No video frame is uploaded. No medical image is sent to a cloud captioning service. The entire ingestion and retrieval pipeline runs on-premise, which is the only architecture that makes multi-modal RAG viable for regulated and personal data.

Text ingestion is the standard RAG pipeline and the foundation on which multi-modal RAG is built. The supported formats are the ones you would expect: PDF, DOCX, TXT, Markdown, HTML, and email (EML/PST/MBOX). Each format has a parser that extracts plain text. PDFs may require OCR if they are scanned images rather than text-native; a local OCR engine (Tesseract or PaddleOCR) converts scanned pages to text before chunking. Email ingestion extracts the body, subject, sender, recipient, and attachments -- the attachments are then routed to the appropriate parser (a PDF attachment goes through the PDF parser, an audio attachment goes through the audio pipeline described in the next section).

Once plain text is extracted, the pipeline is the same across all text formats: chunk the text using your chosen chunking strategy (fixed-size, recursive, semantic, or sentence-aware -- see the dedicated chunking strategies page), generate an embedding for each chunk using a local embedding model (BGE, E5, Nomic, or SBERT), store the embedding in the vector database (Qdrant, ChromaDB, pgvector) alongside the chunk text and metadata, and at query time embed the user's question, retrieve the top-k closest chunks, and pass them to the local LLM as context.

The metadata stored with each text chunk is what links it back to the source. At minimum, every chunk stores: the source file path, the page number or section, the character offset within the source, and the chunk's position in the document. This metadata is what allows the system to cite the exact passage the answer came from -- "page 4, paragraph 2 of the cardiology consult dated March 12, 2026." The same metadata pattern (source reference + position) is reused for audio timestamps, video frame indices, and image file references, which is what makes the unified retrieval surface possible.

Audio ingestion converts a recording -- a patient dictation, a meeting recording, a lecture, a voice memo -- into a transcript with timestamps, chunks the transcript for retrieval, and stores a back-reference to the original audio file so that any retrieved text chunk can be played back as audio. The pipeline has four steps, plus two optional enhancements (speaker diarization and audio embeddings) that significantly improve retrieval quality in specific scenarios.

Speaker Diarization: Who Spoke When

Transcription alone produces a wall of text without any indication of who said what. For meetings, medical dictations with multiple speakers (e.g., a physician and a patient), interviews, and any scenario where attribution matters, the pipeline adds a diarization step. Diarization segments the audio by speaker, labeling each segment with a speaker ID ("SPEAKER_00", "SPEAKER_01") and optionally mapping those IDs to real names if a speaker profile is available.

The standard local tool for diarization is pyannote.audio, an open-source speaker diarization toolkit that runs on GPU. It performs speaker segmentation (detecting speaker changes) and optional speaker embedding (clustering segments by voice similarity). When integrated with Whisper, the pipeline becomes: pyannote identifies speaker boundaries, Whisper transcribes each segment, and the final transcript is formatted as "[00:01:42] SPEAKER_00: The rhythm looks irregular on the ECG. [00:01:55] SPEAKER_01: How long has that been present?" This enriched transcript is what gets chunked and embedded -- and because each chunk now carries speaker labels, retrieval can be filtered or weighted by speaker. A query "what did the doctor say about the medication?" can be routed preferentially to chunks labeled SPEAKER_00 if SPEAKER_00 is the physician.

Speaker identification (mapping speaker IDs to real names) requires a speaker profile -- a short voice sample per known speaker, converted to a speaker embedding. The system compares pyannote's speaker embeddings against the stored profiles and assigns names where the cosine similarity exceeds a threshold. This is fully local: no voice print is ever uploaded. The trade-off is that identification accuracy degrades with short utterances, noisy recordings, and voices that are similar (e.g., family members). For high-stakes attribution, a human review step is recommended; for retrieval purposes, speaker IDs alone are usually sufficient.

Audio Embedding Models: CLAP for Audio-to-Text Similarity

Text-only retrieval over transcripts works well when the transcript is accurate, but it has a subtle weakness: the transcript is a lossy representation of the audio. Tone, emphasis, ambient sounds, music, and non-speech audio events are lost in transcription. A meeting transcript may say "the CFO expressed confidence in the forecast," but the audio carries the hesitation in the voice that a human listener would immediately recognize. For retrieval scenarios where the audio itself carries information beyond the words -- sound effects, music, ambient environment, emotional tone -- a text-only index is insufficient.

CLAP (Contrastive Language-Audio Pretraining) addresses this by embedding audio and text into a shared vector space. A CLAP model takes an audio clip and produces a vector; it takes a text description and produces a vector in the same space. When the two vectors are close, the audio clip matches the description. This enables retrieval that text transcription cannot: "find the part of the recording where someone was speaking angrily," "find the segment with background music," or "find the section with a siren in the background." In a sovereign pipeline, CLAP runs locally -- the audio is never uploaded -- and the CLAP embeddings are stored alongside the transcript-chunk embeddings in the vector database.

The practical architecture is a hybrid: transcript chunks are embedded with a text embedding model (for precise word-level retrieval), and the same audio segments are embedded with CLAP (for acoustic-level retrieval). At query time, the user's text query is embedded with both the text embedder (to match transcripts) and the CLAP text encoder (to match audio embeddings). Results from both are merged and reranked. This hybrid is more expensive -- it doubles the embedding work -- but it catches acoustic context that text-only retrieval misses. For most applications, transcript-only retrieval is sufficient, and CLAP is an optional enhancement for domains where acoustic content matters (medical auscultation recordings, environmental monitoring, music archives).

Video is the richest modality because it contains both audio and images, and the ingestion pipeline must handle both. A video file (MP4, MKV, MOV, AVI) is processed by splitting it into two parallel tracks: an audio track (transcribed as in the audio pipeline) and a sequence of key frames (extracted, captioned, and embedded as images). The two tracks are then linked by timestamp, so that a retrieval hit on a transcript chunk can also surface the frame that was visible at that moment, and a retrieval hit on a frame caption can also surface the words being spoken at that moment.

Scene Detection: PySceneDetect for Automatic Boundaries

PySceneDetect is an open-source Python library that detects scene boundaries in a video by analyzing frame-to-frame differences. It identifies the timestamps at which the visual content changes significantly -- a cut, a slide transition, a camera pan to a new subject. These boundaries are superior to fixed-interval frame extraction for content with distinct visual segments (lectures with slide changes, training videos with chapter transitions, surgical videos with procedural phases). By extracting a frame only at scene boundaries, the pipeline avoids redundant frames of static content and captures each distinct visual moment exactly once.

The integration is straightforward: PySceneDetect processes the video, outputs a list of scene boundary timestamps, and ffmpeg extracts a frame at each boundary. The frame is then captioned and embedded as described above. PySceneDetect runs on CPU (it does not require a GPU) and processes video at roughly 2-5x realtime on a modern machine, making it suitable for batch ingestion. The two detection algorithms -- ContentDetector (threshold on frame-to-frame content difference) and AdaptiveDetector (adaptive threshold that handles gradual transitions) -- cover most video types. For medical and training video, ContentDetector with a tuned threshold is the usual choice.

Image ingestion converts a picture -- a photograph, a diagram, a scanned document, a medical image -- into a caption (for text retrieval), a CLIP embedding (for visual-similarity retrieval), and a file reference (for display). Images are the modality where cross-modal embedding adds the most value, because images can be retrieved by either a text description or by another image. The pipeline has four steps.

DICOM Support for Medical Imaging

DICOM (Digital Imaging and Communications in Medicine) is the standard format for medical imaging -- MRIs, CTs, X-rays, ultrasounds. A DICOM file contains both the pixel data and a rich metadata header: patient ID, study date, modality, body part, slice thickness, radiology report text, and more. A sovereign multi-modal RAG system with DICOM support extracts this metadata, generates a natural-language description from it ("Axial T2-weighted MRI of the brain, study dated 2026-03-12, 5mm slices, 24 slices"), and stores it as the caption. The pixel data is converted to a standard image format (PNG or JPEG) for CLIP embedding and for display.

The radiology report, when present in the DICOM header or in a linked file, is one of the highest-value text artifacts in a medical RAG system. It is stored as a text chunk (embedded and retrievable like any text) and linked to the corresponding DICOM image by study ID. A query like "show me the MRI where the radiologist noted a possible meningioma" retrieves the report text chunk, which links to the DICOM image, which the system can display. The user sees the report text, the image, and the metadata -- all from a single text query, all from local storage.

The technical tool for DICOM parsing is pydicom, an open-source Python library that reads DICOM headers and extracts pixel data. For rendering, the pixel data is converted to a NumPy array and then to a displayable image. For 3D volumes (CT and MRI stacks), individual slices are extracted and captioned separately, with the slice index stored in metadata so the system can display the exact slice that a retrieved caption refers to. Orthanc is an open-source DICOM server that can be deployed alongside the RAG system for PACS-level storage and query (DIMSE/DIMDIR), providing a standards-compliant layer between the filesystem and the RAG pipeline.

The embedding models are the heart of multi-modal RAG. They determine what can be retrieved and how. A text-only embedding model (BGE, E5) places text chunks in a vector space but cannot place images or audio in that same space. Multi-modal embedding models place multiple modalities in a shared space, which is what enables cross-modal retrieval (a text query finding an image, an audio clip finding a text description). The four models below are the standard choices for a sovereign stack; all run locally.

CLIP: Text and Image in One Space

CLIP (Contrastive Language-Image Pretraining) was trained on 400 million image-text pairs to learn a shared embedding space where a caption and its image map to nearby vectors. In RAG, CLIP serves two roles: (a) it embeds images so that text queries can find them ("find images of a fractured wrist" returns images whose CLIP vectors are near the text's CLIP vector), and (b) it embeds images so that similar images can be found (image-to-image similarity). CLIP does not generate text -- it produces only vectors -- so it is always paired with a captioning model (LLaVA or BLIP) that provides the textual representation. The ViT-B/32 variant is sufficient for most retrieval tasks; ViT-L/14 improves accuracy at the cost of more VRAM and slower embedding.

CLAP: Text and Audio in One Space

CLAP (Contrastive Language-Audio Pretraining) is the audio analog of CLIP. It was trained on audio-text pairs (audio clips with descriptions) to learn a shared space where an audio clip and its textual description map to nearby vectors. In RAG, CLAP enables retrieval based on acoustic content that transcends spoken words: background sounds, music, ambient environment, emotional tone. A text query "someone speaking with background traffic noise" retrieves audio segments whose CLAP embeddings match that description. CLAP is most valuable when the audio carries meaning beyond the transcript -- environmental recordings, auscultation sounds, music. For pure speech, transcript-based retrieval is usually sufficient and CLAP is a complement, not a replacement.

ImageBind: Six Modalities in One Space

ImageBind (Meta AI) is the most ambitious of the multi-modal embedders. It binds six modalities -- text, image, audio, video, depth maps, thermal images, and IMU (inertial sensor) data -- into a single shared embedding space, anchored on images. The key property is that modalities that were never directly paired in training (e.g., audio and depth) are still close in the embedding space if they are both close to a shared image. In practice, this means a text query can retrieve audio, video, images, depth maps, and thermal images from a single vector collection. For a sovereign RAG system that ingests diverse sensor data (a robotics dataset, a medical imaging archive with depth and thermal, an industrial monitoring system), ImageBind is the unified embedder that makes cross-modal retrieval possible without a separate model per modality. The trade-off is size (~1.2B parameters, ~4-6GB VRAM) and the fact that per-modality accuracy is slightly lower than dedicated models (CLIP alone is better for image-only tasks).

LLaVA: Visual Question Answering

LLaVA is not an embedding model in the CLIP/CLAP sense -- it does not produce a shared-space vector. It is a vision-language model that takes an image and a text prompt and produces a text answer. In RAG, it serves as the captioning and VQA layer: given an image, it generates the caption that becomes the image's textual representation. Given an image and a question ("What abnormality is visible in this X-ray?"), it produces an answer that can be stored as an enriched caption or used directly as context for the LLM. LLaVA's advantage over BLIP is the ability to produce detailed, prompted, multi-sentence descriptions and to answer questions about an image rather than just describing it. Its disadvantage is size (7B+) and slower inference. For a sovereign stack with a 16GB+ GPU, LLaVA is the recommended captioner; for a constrained 8GB GPU, BLIP-large is the fallback.

Multi-modal RAG separates storage into three layers: the vector database (embeddings and metadata), the filesystem (media files), and the metadata layer (links between vectors and media). This separation is not optional -- it is a consequence of the fact that embeddings are small and need fast similarity search, while media files are large and need cheap bulk storage. Trying to store video files inside a vector database is both expensive and slow; trying to do similarity search over files on disk is impractical. The three layers serve different purposes and are optimized differently.

Layer 1: Vector Database (Embeddings + Metadata)

The vector database (Qdrant, ChromaDB, pgvector, or Milvus) stores every embedding -- text chunk embeddings, CLIP image embeddings, CLAP audio embeddings, frame-caption embeddings, transcript-chunk embeddings -- in one or more collections. Each vector record carries a payload: the modality, the source file path, the timestamp or frame index, the caption or chunk text (for display and for the LLM), and any access-control labels. The vector database is optimized for fast approximate nearest-neighbor search (HNSW is the default index) and for metadata filtering (so retrieval can be scoped to a modality, a source, or an access level). It stores vectors and metadata -- never the media itself.

Layer 2: File System (Media Files)

The filesystem stores the original media: PDFs, audio files (WAV, MP3), video files (MP4, MKV), images (JPG, PNG), DICOM files, and extracted video frames. The organization is a directory structure indexed by source and date: /media/audio/2026/03/consult_20260312.wav, /media/video/2026/03/training_session.mp4, /media/images/2026/03/scan_001.dcm, /media/frames/training_session/frame_0083.jpg. The filesystem is cheap bulk storage -- a 1TB NVMe or SATA SSD holds hundreds of hours of video and tens of thousands of images. For larger archives, a RAID array or a NAS provides capacity and redundancy. The filesystem is accessed by the application layer to serve media to the user at retrieval time (play this audio segment, display this frame, show this image).

Layer 3: Metadata Linking

The metadata that links vectors to media lives in the vector database payloads and, optionally, in a relational database (PostgreSQL) that stores richer provenance: source ingestion batch, ingestion date, original filename, content hash, access-control list, retention policy, and audit log. The vector database payload stores the minimal link (file path + timestamp + modality); the relational database stores the full provenance record. At retrieval time, the vector database returns the payload, the application reads the file path, fetches the media from the filesystem, and optionally enriches the response with provenance from the relational database. This three-layer design keeps each store focused on what it does best: vectors for search, filesystem for bulk media, relational DB for provenance and access control.

The Ingestion-to-Retrieval Flow

The full sovereign flow, end to end: (1) a media file arrives in the ingestion queue (a user uploads an audio file, a video is dropped in a watched directory, DICOM files are received from a modality). (2) The router identifies the modality and dispatches to the appropriate pipeline. (3) The pipeline converts the media to text (Whisper for audio, Whisper+LLaVA for video, LLaVA/BLIP for images) and to vectors (text embedder for captions/transcripts, CLIP for images, CLAP for audio, optionally ImageBind for unified embeddings). (4) The vectors and their metadata payloads are written to the vector database. (5) The original media file is moved to its permanent filesystem location. (6) At query time, the user's text question is embedded (text embedder + CLIP text encoder + CLAP text encoder for hybrid retrieval), the top-k vectors are retrieved, their payloads are read, and the application layer surfaces the results: text citations, playable audio segments, displayable frames, and viewable images. The LLM receives the text representations as context and generates an answer. The user sees the answer, the citations, and the original media. Everything local.

The table below summarizes how each modality is ingested, embedded, stored, and retrieved in a sovereign multi-modal RAG system. The pattern is consistent across modalities: convert to text, embed, store the vector with a media back-reference, retrieve by text query, and surface the original media as a citation. The differences are in the conversion tool and the embedding model.

The value of multi-modal RAG is clearest in concrete scenarios. The four use cases below illustrate how the pipelines described in this page combine to make non-text data retrievable and cite-able, and why sovereign (on-premise) execution is a requirement rather than a preference for each one.

Medical: DICOM Images and Voice Dictations

A cardiology practice has three types of data: structured EHR records (text), dictated visit notes (audio), and imaging studies (DICOM). A sovereign multi-modal RAG system ingests all three. The dictations are transcribed with Whisper large-v3-turbo locally, diarized with pyannote to separate physician from patient, and the transcript chunks are embedded and stored with audio timestamps. The DICOM studies are parsed with pydicom: the metadata and radiology report become text chunks, the pixel data is converted to images and captioned by LLaVA, and the images are embedded with CLIP. A clinician asks "What did Dr. Patel say about the irregular rhythm, and is there a prior echo that shows the same finding?" The system retrieves the relevant transcript chunk (with a playable audio clip of the dictation), the radiology report text, and the prior echocardiogram frame whose CLIP embedding matches the finding description. All of it is surfaced together. No PHI left the network at any point -- no cloud transcription, no cloud captioning, no cloud embedding.

Business: Training Videos and Meeting Recordings

A company has a library of training videos (software demonstrations, safety procedures, onboarding sessions) and recorded Zoom meetings (strategy discussions, product reviews, customer calls). A sovereign multi-modal RAG system ingests both. Each video is demuxed with ffmpeg, the audio is transcribed with Whisper (diarized for meetings, to attribute statements to speakers), and key frames are extracted by PySceneDetect and captioned by LLaVA ("The instructor is demonstrating the failover procedure on the admin dashboard"). The transcript chunks and frame captions are embedded and stored together. An employee asks "How do I configure the backup retention policy, and was this discussed in last quarter's infrastructure meeting?" The system retrieves the training video segment (with a playable clip showing the configuration screen) and the meeting transcript chunk where the topic was discussed (with the speaker identified and the timestamp for playback). The strategy meeting's proprietary content never went to a cloud transcription or captioning service.

Education: Lecture Videos and Slides

A university department records lectures (video + audio) and distributes slide decks (PDF, which may be image-based). A sovereign multi-modal RAG system ingests the lecture recordings: audio transcribed with Whisper, scene-detected frames captioned ("Slide showing the equation for Bayes' theorem with an example calculation"), and the slide PDFs are parsed (OCR'd if scanned). A student asks "When did the professor explain the difference between precision and recall, and which slide had the confusion matrix example?" The system retrieves the lecture transcript chunk (with a playable video segment of the professor explaining the concept), the frame caption for the confusion-matrix slide (with the slide image displayed), and the PDF page from the slide deck. The lecture content and the student's query history stay on the university's hardware, compliant with FERPA, never sent to a cloud lecture-transcription service that could retain or train on student-facing content.

Personal: Family Photos and Voice Memos

A family has a home server storing years of photos, home videos, and voice memos. A sovereign multi-modal RAG system ingests them all: photos are captioned by LLaVA ("A family of four standing at the Grand Canyon at sunset") and embedded with CLIP; voice memos are transcribed with Whisper; home videos are processed through the full video pipeline. A parent asks "Find the voice memo where I reminded myself to call the insurance agent, and show me the photos from the Grand Canyon trip." The system retrieves the voice memo transcript chunk (with a playable audio clip) and the Grand Canyon photos (displayed as a gallery, retrieved via their CLIP embeddings and LLaVA captions). Everything runs on the home server. No family photo, no voice memo, no transcript ever leaves the house. This is the personal AI assistant trajectory described in the RAG philosophy page -- and multi-modal ingestion is what makes it see and hear, not just read.