Conversational RAG: The Future Interface for Sovereign AI

Traditional RAG is query-response. You type a query. The system retrieves documents. It generates an answer. The interaction ends. The next query is independent -- the system has no memory of what you asked before, no understanding that your new question might be a follow-up, no ability to resolve pronouns, references, or implicit context. Each query is a standalone transaction. This is search with a language model bolted on. It is useful, but it is not conversation.

Conversational RAG is a different paradigm. It is ongoing dialogue. The system remembers context, maintains conversation history across turns, and can ask clarifying questions when your intent is ambiguous. It does not just answer -- it participates. When you say "what about the kids?" it knows you were asking about dentist appointments, so it retrieves the kids' dental appointments rather than searching literally for the word "kids" in your documents. When you ask "what did her last report card say?" it resolves "her" to the person you were discussing three turns ago. When your question is vague -- "how about the insurance?" -- it can ask: "Are you asking about your medical, dental, or auto insurance?" instead of guessing and retrieving the wrong documents.

The difference is not cosmetic. It is the difference between using a tool and talking to someone who knows you. A search bar is a tool. You formulate a precise query, you get a result, and you reformulate if the result is wrong. A conversation is a relationship. You speak naturally, you reference things you said earlier, you expect the other party to track the thread. Conversational RAG brings RAG from the first paradigm into the second. The technical cost is significant -- it requires session state, conversation history, context window management, reference resolution, and retrieval that adapts to conversational context rather than treating each query as independent. But the user experience cost of not doing it is higher: without conversation, RAG is just a search engine that talks.

Clarifying Questions: When the System Asks Back

One of the most powerful capabilities of conversational RAG -- and one that traditional RAG entirely lacks -- is the ability to ask clarifying questions. When a query is ambiguous, a traditional RAG system guesses. It retrieves whatever seems most relevant based on the embedding similarity of the literal query, and it generates an answer from whatever it retrieved. If the user asked "what about the insurance?" and the system retrieved auto insurance documents when the user meant medical insurance, the answer is wrong and the user has to start over. A conversational system recognizes the ambiguity and asks: "Are you asking about your medical insurance or your auto insurance?" The user answers in one word. The system retrieves correctly. The conversation continues.

This requires the LLM to assess its own retrieval confidence. If the top retrieved documents span multiple distinct topics, or if the query is vague relative to the document store, the system can be configured to ask for clarification rather than guessing. The threshold is tunable: too aggressive, and the system asks too many questions, becoming annoying; too passive, and it guesses wrong too often. The sweet spot is asking only when the cost of a wrong answer is high and the ambiguity is genuine. This is a judgment call that the LLM makes on each turn, informed by the conversation context and the distribution of retrieved results.

Notice the flow. The system did not guess. It identified genuine ambiguity (two scans, one query), asked a minimal clarifying question, and the user resolved it in three words. The entire exchange took maybe six seconds. A traditional RAG system would have retrieved one of the two reports, generated an answer, and the user would have had to say "no, the other one" and re-query. Clarifying questions prevent wrong answers before they happen. This is only possible in a conversational system, because it requires the system to hold the turn open, ask, and wait for a response -- something a query-response system cannot do.

The defining interface of conversational RAG is voice. Not a chat box. Not a search bar. You speak. The system hears. It answers in voice. This is the interface that makes conversational RAG usable from the couch, from the kitchen, from across the room -- without a screen, without a phone, without touching anything. The voice pipeline is the technical core of the experience, and its latency determines whether the system feels like an assistant or a slow search engine.

The full voice pipeline is five stages, each running locally:

For a conversational voice interface to feel natural, the total round-trip latency -- from the moment you stop speaking to the moment the system starts answering -- must be under 2 seconds. Above 2 seconds, the interaction starts to feel like waiting. Above 4 seconds, it feels like a slow search engine. Above 6 seconds, users stop using voice and go back to typing. The 2-second target is not arbitrary -- it is the threshold below which a voice interaction feels like a conversation and above which it feels like a query. Hitting it on local hardware is the engineering challenge that defines real-time conversational RAG.

The budget above totals roughly 1,795 milliseconds -- under the 2,000-millisecond target. Each stage is fighting for milliseconds, and the dominant cost is LLM generation. A 7B/8B model generating a concise 50-word answer at 80 tokens per second on a GPU takes about 875 milliseconds for generation alone. Every other stage combined must fit in the remaining 1,125 milliseconds. This is achievable on a well-configured GPU system, but it requires careful optimization at every stage.

Optimization Strategies for Hitting the Budget

Several techniques can shave milliseconds off the pipeline to bring the total under 2 seconds. First, streaming generation: the LLM can begin streaming tokens to the TTS engine before the full answer is generated. The first sentence can be synthesized and start playing while the LLM is still generating the second sentence. This "first-audio-out" optimization can cut perceived latency by 500-800 milliseconds, because the user hears the beginning of the answer while the rest is still being generated. This is the single most impactful optimization for voice latency.

Second, model quantization: a 7B/8B model in 4-bit quantization (Q4_K_M) runs roughly 50% faster than the same model in 8-bit, with minimal quality loss for RAG answers. Ollama supports this out of the box. Third, speculative decoding: the LLM uses a smaller draft model to propose tokens that the larger model verifies, accelerating generation by 1.5-2x for certain workloads. Fourth, prompt caching: if the system prompt and conversation history are the same across turns (which they are, within a session), the KV cache can be reused, skipping the prefill stage and saving 100-300 milliseconds. Ollama supports this natively.

Fifth, STT model selection: Whisper-base is a good default, but for latency-critical use, whisper.cpp with the tiny.en or base.en model (English-only, smaller) can be 2-3x faster than the multilingual base model. If you only need English, use the English-only models. Sixth, parallel pipelining: on a system with a GPU, STT and query rewriting can overlap with retrieval preparation. The GPU runs STT while the CPU handles the VAD post-processing and prepares the retrieval query. These are small gains, but at the millisecond scale of the 2-second budget, they add up.

The hardest problem in conversational RAG is not retrieval. It is context. Consider this exchange:

"What about the kids?" is not a standalone query. It only makes sense if the system remembers that the previous question was about dentist appointments. The literal query -- "what about the kids" -- will retrieve documents about children in general: school records, photos, birth certificates, pediatrician visits. None of that is what the user wants. The user wants to know if the kids have dentist appointments. The system can only know this if it maintains conversation history and uses it to resolve the reference.

The Conversation Buffer

The mechanism is a conversation buffer -- a rolling window of the last N turns of dialogue, plus the retrieved chunks and the generated answer from each turn. When a new query arrives, the system passes the conversation buffer to the LLM (or a smaller, faster model) along with the new query. The LLM's job is to rewrite the new query into a self-contained retrieval query that incorporates the context. "What about the kids?" in the context of a dentist appointment query becomes "What dentist appointments do the children have?" This rewritten query is what gets embedded and sent to the vector database. The retrieval is now correct.

This query rewriting step is the single most important technique in conversational RAG. Without it, follow-up questions fail. With it, the system can handle pronoun resolution ("her" = Emma, from two turns ago), topic continuation ("what about the kids?" = kids' appointments, not kids in general), and ellipsis ("and the insurance?" = the insurance related to the dental visit). The rewriting model does not need to be the full 7B/8B answer-generation model. A smaller, faster model (Phi-3.5, Qwen 2.5 3B) can handle query rewriting in 100-200 milliseconds, keeping the latency budget intact.

Context Window Management

The conversation buffer cannot grow indefinitely. Every turn adds tokens -- the user's query, the retrieved chunks, the LLM's answer. After 10-15 turns, the buffer may exceed the LLM's context window, or at least become so large that it slows down generation and degrades quality (the LLM starts ignoring early turns in a long context). The system needs a strategy for managing the context window as conversations grow.

The simplest strategy is a fixed-size rolling window: keep only the last N turns (typically 5-10), discarding older turns. This works for short conversations but loses context in long ones. A more sophisticated strategy is conversation summarization: when the buffer exceeds a threshold (say, 8 turns), the system uses the LLM to summarize the older turns into a compact paragraph, then keeps the summary plus the last 3-4 verbatim turns. The summary preserves the thread ("The user asked about their dentist appointment, then about the kids' appointments, then set a reminder for Emma's permission slip") while the recent turns preserve the details that might be referenced in follow-ups.

Conversation Summarization in Practice

Conversation summarization is triggered when the buffer exceeds a token threshold. The system takes the oldest portion of the conversation (say, turns 1-7) and passes it to the LLM with a prompt like: "Summarize the following conversation, preserving all factual details, decisions made, and entities discussed. Keep it under 200 tokens." The LLM produces a compact summary. The new buffer becomes: [summary of turns 1-7] + [verbatim turns 8-10]. The summary is much smaller than the original turns, but it preserves the information the LLM needs to resolve references in future turns. This can be done transparently, in the background, without the user noticing.

Session Persistence

Conversations do not always end cleanly. You might ask about your dentist appointment, get the answer, and then twenty minutes later say "oh, and what time did I need to arrive?" If the session has been reset (because of the silence timeout), the system has no context and treats this as a new query about "arriving" -- which retrieves nothing useful. Session persistence solves this. The conversation state -- the buffer, the summary, the retrieved chunks -- is stored in a local database (SQLite for home systems, Postgres for larger deployments) and can be resumed within a configurable window (say, 30 minutes of inactivity). If you come back within the window, the system restores the session and you can ask follow-ups as if you never left. After the window expires, the session is archived and a new one begins.

Session persistence also enables cross-session continuity. If you ask "did I ever follow up on that roof repair?" a week after the original conversation about the roofer, the system can search its archived sessions, find the one about the roof, and retrieve the relevant context. This blurs the line between conversation history and long-term memory, which we cover in the Memory & Personalization section below.

Conversational RAG does not have to be text-in, text-out. In a sovereign system, the conversation can be multi-modal: you ask a question by voice, the system retrieves not just text documents but images -- medical scans, photographs, diagrams, charts -- and answers about what it sees. You can ask follow-up questions about the image. The system describes what it observes, answers questions about specific features, and can compare images across time. This is multi-modal conversational RAG, and it is one of the most powerful -- and most sensitive -- capabilities of a sovereign AI system.

The canonical use case is medical imaging. A patient has a chest X-ray, an MRI, or a CT scan stored in their personal document store. They ask by voice: "What did the doctor note about my last X-ray?" The system retrieves the radiology report (text) and the image itself. The LLM reads the report and answers. But the patient follows up: "Can you show me where the issue is?" or "Does it look different from the one last year?" To answer these, the system needs to actually look at the images. This is where vision-language models come in.

This conversation requires a vision-language model (VLM) that can look at the MRI image and answer questions about it. The leading open-source, locally-runnable VLMs in 2026 are LLaVA (Large Language and Vision Assistant) and CogVLM. Both can be run locally via Ollama or directly with their model weights, and both can answer natural-language questions about images. LLaVA combines a vision encoder (CLIP) with a language model (typically Llama or Vicuna) and can describe images, answer questions about image content, and reason about visual features. CogVLM is a similarly capable model with strong performance on visual question answering benchmarks.

LLaVA and CogVLM for Local Visual Question Answering

LLaVA is the most accessible option for local deployment. The LLaVA-7B and LLaVA-13B variants run through Ollama, requiring 6-8 GB of VRAM for the 7B model and 12-16 GB for the 13B model. The model takes an image and a text question as input and generates a text answer. For conversational RAG, the flow is: the user's voice query triggers retrieval of both the text report and the image. The text report goes to the standard LLM. If the query requires visual reasoning (describing the image, comparing images, identifying features), the image and the question are sent to LLaVA. LLaVA generates a description or answer, which is then spoken back via TTS. The user can ask follow-up questions about the same image without re-retrieving it, because the image is held in the conversation context.

CogVLM is a more research-oriented alternative with strong performance on medical and technical imagery. It is particularly good at fine-grained visual reasoning -- identifying small features in complex images, which matters for medical scans. The trade-off is that CogVLM is heavier and slower than LLaVA, making it less suitable for real-time voice interaction. For a home or small office system, LLaVA is the practical choice. For a medical practice or a system that needs high-accuracy visual analysis of clinical images, CogVLM may be worth the additional compute. Both run locally. Neither sends images to the cloud.

Security Layers for Medical Images

Multi-modal conversation with medical images introduces security concerns that go beyond text retrieval. Medical images are protected health information (PHI) under HIPAA, and even in a sovereign system -- especially in a sovereign system -- they require explicit access controls. The system implements layered security: the images themselves are stored in an encrypted, access-controlled store. Retrieval of image documents is gated by the same role-based access control that governs text documents -- a child asking about "the MRI" does not retrieve a parent's medical images. The VLM (LLaVA or CogVLM) only receives images that the current speaker is authorized to access. The access control is enforced at the retrieval layer, before the image reaches the model, so the model cannot leak images the speaker cannot see.

Beyond medical imaging, multi-modal conversation extends to any image in the document store. You can ask about a photograph ("who is in this picture?"), a diagram ("what does this wiring diagram show?"), or a chart ("what does this lab result graph indicate?"). The VLM handles all of these. The key architectural principle is that the image retrieval and the VLM query are triggered by the conversational context -- the system decides, based on the query and the retrieved documents, whether to invoke the VLM. Not every query needs image analysis. The system routes to the VLM only when the query requires visual reasoning, keeping latency low for text-only questions. For a deeper dive on multi-modal RAG architecture, see our Multi-Modal RAG page.

Traditional RAG is reactive. You ask. It answers. It never speaks unless spoken to. Proactive RAG breaks this pattern. The system does not just answer questions -- it can proactively surface relevant information based on the document store, the current time, and the user's context. It can bring things to your attention before you ask, because it knows what is in your documents and it knows what is coming up.

This interaction was not triggered by a query. The system proactively checked the document store, found an appointment scheduled for the next day, and brought it to the user's attention. This is proactive RAG -- the system as an agent that monitors your document store and surfaces relevant information at the right time, rather than waiting to be asked.

How Proactive RAG Works

The proactive layer runs as a background process -- a scheduled job that periodically scans the document store for time-sensitive items. It looks for: appointments with upcoming dates, deadlines (bills due, permission slips to return, renewals expiring), time-sensitive instructions from documents ("take medication for 7 days" -- and it is day 6), and items flagged for follow-up in previous conversations. When it finds something relevant and time-appropriate, it queues a proactive notification. The notification fires when the user is next present (detected by wake word activity, motion, or a scheduled time like morning or evening). The system speaks the notification, the user responds, and the conversation continues.

The proactive layer requires a few components that reactive RAG does not. First, a scheduler (cron or a custom event loop) that triggers the scan at regular intervals -- every hour, or every morning at 7am, or on a configurable schedule. Second, a date extraction pipeline that identifies time-sensitive information in documents at ingestion time -- not just the text content, but the dates, deadlines, and time references within the text. This can be done with a lightweight NLP model or even a regex-based extractor for common date formats. Third, a notification policy that determines what is worth surfacing proactively -- not every document with a date is worth a notification. The policy should be conservative by default and tunable by the user. Too many proactive notifications and the system becomes an annoyance. Too few and it is not doing its job.

Proactive vs. Reactive: The Right Balance

The ideal system is mostly reactive (it answers when you ask) with a thin layer of proactive surfacing for genuinely time-sensitive items. The proactive layer should feel like a helpful family member who says "hey, don't forget your appointment tomorrow" -- not like a notification machine that pings you every five minutes. The technical implementation is a background scanner with a conservative policy, a notification queue, and a user-configurable threshold. The result is a system that is not just a search engine or a chatbot, but an assistant that actively helps you stay on top of your life -- within boundaries you control.

Proactive RAG also extends to contextual awareness. If the system knows you are cooking (because you have been asking recipe questions), it can proactively set a timer when you ask "how long does the dough need to rise?" If it knows you are reviewing medical documents, it can proactively offer to track a new medication. The proactive layer uses the current conversation context, not just the document store, to decide what to surface. This requires the system to maintain a lightweight model of what the user is currently doing -- derived from recent queries and conversation topics -- and to use that model to make proactive suggestions. This is the beginning of the shift from reactive assistant to ambient intelligence, which we discuss in the final section.

A conversational RAG system that forgets everything between sessions is a chatbot with a search engine. A conversational RAG system that remembers -- your preferences, your schedule patterns, your frequent queries, your habits -- is a personal assistant. Memory is what separates a tool from an assistant. And in a sovereign system, that memory is stored locally, on your hardware, under your control. No corporation is building a profile on you from your AI conversations. The profile is yours, it lives on your disk, and you can inspect, edit, or delete it at any time.

Three Tiers of Memory

The memory system has three tiers, each serving a different time horizon:

• Working Memory The current conversation -- the last 5-10 turns, the retrieved chunks, the entities discussed. This is the conversation buffer from the Context Management section. It exists in RAM during the active session and is discarded (or persisted to short-term storage) when the session ends. Time horizon: minutes to hours.
• Session Memory The full conversation history for a session, persisted in a local database (SQLite or Postgres). Includes the summarized context from the Context Management section. This allows session resumption within a window (30 minutes to 24 hours) and provides the raw material for cross-session queries ("did I ever ask about the roof repair?"). Time horizon: hours to days.
• Long-Term Memory Persistent facts, preferences, and patterns learned over time. Stored in a structured local database. This includes: the user's name and family members' names, their doctor's name and appointment patterns, their medication schedule, their preferences ("I prefer morning appointments"), their frequent query topics, and corrections they have made ("no, I said Thursday, not Tuesday"). Time horizon: weeks to years. This is the personalization layer.

What Long-Term Memory Stores

Long-term memory is not a dump of every conversation. It is a structured, curated store of facts and preferences that the system has extracted from conversations over time. The extraction happens in the background: after each session, the system runs a lightweight LLM pass over the conversation to identify durable facts -- things that will be true tomorrow, not just today. "The user's dentist is Dr. Reyes at Camelback Dental" is a durable fact. "The user asked about the dishwasher warranty" is not (unless the warranty itself is stored in the document store, which it is). The extraction model identifies facts, preferences, and patterns, and stores them in a structured format that can be queried and injected into future conversations.

How Memory Differs from Cloud AI Profiling

This is where the sovereignty requirement becomes not just a privacy feature but a fundamental difference in what the system is for. Cloud AI systems -- ChatGPT, Google Assistant, Alexa, Siri -- also build profiles on you. They learn your preferences, your patterns, your relationships. The difference is what they do with that profile. A cloud AI builds a profile on you to serve ads, to train future models, to improve engagement metrics, to sell insights to third parties, and to retain you as a user. The profile serves the company's interests, not yours. You cannot inspect it. You cannot edit it. You cannot delete it (not really). You do not even know what is in it.

A sovereign conversational RAG system builds a profile on you to serve you. The memory exists to make the system better for you -- to remember your preferences so it does not have to ask twice, to learn your schedule so it can proactively help, to correct its own mistakes. The profile serves your interests. You can inspect it (it is a local database, queryable through the admin interface). You can edit it (delete a preference, correct a fact). You can delete it (wipe the memory database and start fresh). And no one else ever sees it, because it never leaves your hardware.

Memory Injection at Query Time

When a query arrives, the system does not just retrieve from the vector database. It also queries the long-term memory store for relevant facts. If the user asks "when is my next appointment?" the system retrieves appointment documents from the vector database, but it also retrieves from memory: "The user's dentist is Dr. Reyes. The user prefers morning appointments." These facts are injected into the LLM prompt alongside the retrieved chunks, giving the LLM context to personalize the answer. The answer might be: "Your next appointment is with Dr. Reyes on Thursday at 9am. I know you prefer mornings, so this one is right in your preferred window." The personalization is not cosmetic -- it comes from stored memory and it makes the answer more useful.

The memory injection must be careful. Too much injected memory and the prompt becomes bloated, slowing generation and diluting the retrieved context. The system should inject only the facts relevant to the current query, selected by a lightweight similarity match between the query and the memory store. If the query is about appointments, inject appointment-related memories. If the query is about a recipe, do not inject the dentist's name. The memory store should be queryable by topic, not just a flat dump. This is a design decision that affects the quality of personalization and the latency of the system.

Everything described above -- the voice interface, the conversation history, the multi-modal medical image queries, the proactive notifications, the long-term memory of your preferences and schedule -- all of it runs locally. This is not a feature. It is a requirement. The sovereignty of conversational RAG is not optional, and it is not a "privacy mode" toggle. It is the architectural foundation. Without it, the system is just another cloud AI with a voice interface, and everything that makes conversational RAG valuable -- the intimacy of the conversation, the sensitivity of the data, the personalization of the memory -- becomes a liability instead of an asset.

Consider what a conversational RAG system knows about you. It has your voice -- your actual voice, captured by a microphone in your home, transcribed and stored. It has your questions -- every query you have asked, which reveals what you are thinking about, what you are worried about, what you are planning. It has your documents -- your medical records, your financial records, your children's school records. It has your conversation history -- the full thread of your dialogue, including follow-up questions that reveal your thought process. It has your memory profile -- your preferences, your patterns, your habits, your relationships. This is the most detailed picture of your personal life that any single system could hold. Now consider where this data lives.

The sovereignty requirement means every component of the conversational RAG pipeline runs on hardware you control. The STT (Whisper) runs locally -- no audio is sent to a cloud transcription service. The query understanding runs locally -- no query is sent to a cloud LLM. The retrieval runs locally -- your documents are in a local vector database. The generation runs locally -- the LLM is Ollama on your hardware. The TTS runs locally -- Piper or another local TTS engine synthesizes the voice. The memory is stored locally -- SQLite or Postgres on your encrypted disk. The session state is stored locally. The proactive scheduler runs locally. The VLM (LLaVA, CogVLM) runs locally -- no image is sent to a cloud vision API. No component of the pipeline touches the internet during normal operation.

This is why edge-tts -- Microsoft's text-to-speech service -- is listed in the TTS section below as "avoid for sovereign." It produces high-quality voices, but it sends your text to Microsoft's servers for synthesis. In a sovereign system, that is unacceptable. The TTS must run locally, even if the voice quality is slightly lower. The same applies to any cloud-based STT, any cloud-based LLM API, any cloud-based embedding service. The sovereignty requirement is absolute: if a component sends data to the internet, it is not part of the pipeline. For a deeper discussion of the philosophy behind this, see our RAG Philosophy: Sovereign Retrieval page.

The technical architecture of a conversational RAG system is more complex than a query-response RAG system because it must manage state across turns, sessions, and time. The components below are the building blocks of a production conversational RAG system running entirely on local hardware.

Session State Management

Every conversation is a session. A session has a unique ID, a start time, a user identity (determined by voice biometrics or explicit login), a conversation buffer, and a status (active, paused, archived). The session state is the central data structure that ties the pipeline together. When a query arrives, the system loads the session state, uses the conversation buffer for context, processes the query through the pipeline, updates the buffer with the new turn, and saves the state. The session state is stored in a local database so it can be resumed if the process restarts or the user returns after a pause.

Session management has three critical operations: create (when a new wake word event or explicit query starts a conversation), resume (when the user returns within the session window and continues the conversation), and archive (when the session expires or the user explicitly ends it). Archived sessions are not deleted -- they become searchable history, enabling cross-session queries ("what did I ask about last week?"). The archive is also the input to the long-term memory extraction process, which runs as a background job after each session ends.

Conversation History in a Local Database

The conversation history is persisted in a local database. For a home system or small office, SQLite is sufficient -- it is a single-file database, embedded in the application, requiring no separate server process, and capable of handling hundreds of thousands of conversation turns without performance issues. For a larger deployment (a medical practice, a business with many users), Postgres is the better choice -- it handles concurrent sessions, larger history, and more complex queries. Both run locally. Both store the data on the same encrypted disk as everything else.

The schema for conversation history is straightforward: a sessions table (session ID, user, start time, end time, status), a turns table (turn ID, session ID, turn number, user query, rewritten query, retrieved chunk IDs, LLM response, timestamp), and a memory table (fact ID, user, fact type, fact content, source session, created time, last updated). The turns table is the core -- it records every interaction in a way that can be replayed, audited, and searched. The retrieved chunk IDs link each turn to the specific document chunks that grounded the answer, which is essential for citation, debugging, and the "show me the source" capability.

Context Window Compression for Long Conversations

As a conversation grows, the context window fills. A 7B/8B model typically has an 8K-32K token context window. With 500 tokens per turn (query + retrieved chunks + answer), an 8K window fills in 16 turns. A 32K window fills in 64 turns. Long conversations -- a medical consultation, a complex research session, a multi-topic planning discussion -- can exceed these limits. The system must compress the context to stay within the window without losing the thread.

The compression strategy, as described in the Context Management section, is summarization. When the buffer exceeds a threshold (say, 75% of the context window), the oldest turns are summarized by the LLM into a compact paragraph, and the verbatim turns are replaced by the summary. The summary preserves entities, facts, and decisions, while discarding the verbose retrieved chunks and the full text of the answers. A 2,000-token conversation can be compressed to a 200-token summary with minimal information loss. This can be done iteratively -- as the conversation continues, older summaries can be re-summarized into even more compact form. The result is a conversation that can run for hours without exceeding the context window, while the LLM retains the essential thread throughout.

Voice Activity Detection (VAD)

Voice activity detection is the component that determines when the user starts speaking and when they stop. It is critical for natural conversation flow. Without VAD, the system either records continuously (picking up background noise and other people) or requires a button press to start and stop recording (which is not hands-free). VAD solves this by continuously analyzing the audio stream and detecting speech segments -- when speech starts, recording begins; when silence is detected for a configurable duration (typically 500-800 milliseconds), recording stops and the utterance is sent to STT.

The leading open-source VAD models are Silero VAD and WebRTC VAD. Silero VAD is a neural network-based detector that runs on CPU with very low latency (under 50 milliseconds per frame) and high accuracy. It distinguishes speech from silence, background noise, and music with high reliability. WebRTC VAD is a simpler, faster, signal-processing-based detector that is less accurate but even lower latency. For most conversational RAG systems, Silero VAD is the right choice -- it is accurate enough to avoid false triggers (the system starting to record when no one is speaking) and sensitive enough to catch quiet speech. The VAD runs continuously, listening to the microphone, and only activates the STT pipeline when it detects a speech segment.

Wake Word Detection for Hands-Free Operation

Wake word detection is the component that makes the system hands-free. A small, always-on model listens to the audio stream for a specific wake word -- "Computer," "Hey Assistant," "Home," or any custom word or phrase. When the wake word is detected, the system activates VAD and begins recording the query. The wake word model is tiny (under 10 MB) and runs on CPU with negligible power and compute cost. It runs continuously, alongside the VAD, without measurable impact on the system.

The leading open-source wake word engines are openWakeWord and Porcupine (Picovoice's open-license tier). openWakeWord is fully open-source, trains custom wake words from a short audio sample, and runs on CPU with under 5% of a single core. Porcupine offers higher accuracy and lower false-positive rates but has a more restrictive license for the free tier. For a sovereign system, openWakeWord is the recommended choice because it is fully open and can be trained on any word without licensing concerns. The wake word is a critical configuration decision: it should be distinctive enough to avoid false triggers (not a common word), short enough to speak naturally (2-3 syllables), and memorable enough that the household uses it consistently.

Text-to-speech is the final stage of the voice pipeline -- the component that turns the LLM's text answer into speech the user hears. The choice of TTS engine is a trade-off between quality, latency, expressiveness, and sovereignty. The system needs a voice that sounds natural (not robotic), synthesizes quickly (under 1 second for a typical answer), runs locally (no cloud API), and ideally supports streaming (so the first sentence starts playing while the rest is being generated).

Piper: The Recommended Choice for Speed

Piper is the recommended TTS engine for real-time conversational RAG. It is fast (5-20x real-time on CPU, meaning a 5-second answer is synthesized in 250-500 milliseconds), runs entirely locally, supports streaming synthesis (the first sentence can start playing while subsequent sentences are being generated), and produces natural-sounding speech. The voice quality is good -- clear, intelligible, and natural -- though it lacks the emotional expressiveness of more computationally intensive models. For a voice assistant that needs to answer questions quickly and clearly, Piper is the right trade-off. It is the TTS equivalent of a 7B LLM: not the absolute best quality, but fast enough for real-time interaction and good enough that users do not complain.

Coqui TTS: The Quality Choice

Coqui TTS (the open-source successor to Mozilla TTS) offers higher quality voices with better prosody -- the rhythm and intonation of speech. It supports multiple model architectures (Tacotron, VITS, XTTS) and can produce very natural, warm-sounding speech. The trade-off is speed: Coqui is slower than Piper, typically 1-3x real-time on CPU, meaning a 5-second answer takes 1.5-5 seconds to synthesize. On a GPU, Coqui is significantly faster and can approach Piper's speed while maintaining higher quality. For a system with a GPU, Coqui is a viable alternative to Piper. For a CPU-only system, Coqui may be too slow for real-time voice interaction. The XTTS model in Coqui also supports voice cloning -- you can provide a short sample of a voice you want the system to use, and it will synthesize speech in that voice. This is a powerful feature for personalization (the system can speak in a voice the family finds pleasant), but it should be used carefully, as voice cloning has ethical implications.

Bark: The Expressive Choice (With a Cost)

Bark is a generative TTS model that produces highly expressive, emotionally rich speech. It can laugh, sigh, convey excitement, add emphasis, and even produce non-speech sounds like music or ambient noise. The quality is remarkable -- Bark-generated speech can sound indistinguishable from a human recording in some samples. The cost is latency: Bark is computationally expensive, generating slower than real-time on CPU (a 5-second answer takes 10-25 seconds to synthesize on CPU). On a high-end GPU, Bark is faster but still slower than Piper or Coqui. Bark also does not support streaming -- it generates the full audio clip before returning any of it. For these reasons, Bark is not recommended as the primary TTS for real-time conversational RAG, but it could be used for specific scenarios where expressiveness matters more than speed: reading a story to a child, delivering a proactive notification with appropriate emotional tone, or generating audio for offline consumption.

Quality vs. Latency: The Trade-Off

The TTS choice is a quality-latency trade-off that must be made in the context of the overall pipeline budget. Piper buys you 500 milliseconds of latency budget for the TTS stage, leaving more budget for LLM generation. Coqui costs you 1,500-2,000 milliseconds, eating into the LLM budget and potentially pushing the total over 2 seconds. Bark costs you 5,000+ milliseconds on CPU, making real-time conversation impossible. On a GPU system with a fast LLM, Coqui is viable and the quality improvement is noticeable. On a CPU system, Piper is the only choice that keeps the pipeline under 2 seconds. The system can be configured to use different TTS engines for different scenarios: Piper for quick answers (where speed matters), Coqui for longer responses (where quality matters and the user is already listening), and Bark for offline-generated content (where latency does not matter).

Ollama with Voice Models: The Emerging Option

The Ollama ecosystem is beginning to support voice models natively, which would simplify the architecture by running both the LLM and the TTS through the same inference engine. This is an emerging capability in 2026 and is not yet as mature or fast as dedicated TTS engines like Piper. The potential advantage is architectural simplicity -- one model server for both text generation and speech synthesis, with a unified API. The current disadvantage is that Ollama voice models are slower than Piper and the quality is variable. This is a space to watch, as the Ollama ecosystem is moving quickly and voice support is a natural extension of its model-serving architecture.

Real-time conversational RAG -- voice in, voice out, under 2 seconds -- has specific hardware requirements that go beyond a standard RAG system. The bottleneck is the LLM generation stage and the STT stage, both of which are dramatically accelerated by a GPU. A CPU-only system can run conversational RAG, but it cannot hit the 2-second target with a 7B/8B model. The hardware recommendations below are for systems that aim for real-time voice interaction.

The GPU Requirement Explained

The GPU is not a luxury for real-time conversational RAG. It is a requirement. The reason is the LLM generation stage. A 7B/8B model on CPU generates at 10-20 tokens per second. A 50-word answer (70 tokens) takes 3.5-7 seconds. That alone exceeds the 2-second budget, before adding STT, retrieval, and TTS. The same model on an RTX 4060 GPU generates at 40-80 tokens per second, producing the same answer in 0.9-1.8 seconds. The GPU turns a 6-second pipeline into a 1.8-second pipeline. The 8GB VRAM of the RTX 4060 is sufficient for a 7B/8B model in 4-bit quantization (which requires 4-5 GB) plus Whisper-base (which requires under 1 GB) plus the embedding model and the VAD model. If you want to run LLaVA 7B for multi-modal conversation, you need the VLM in VRAM as well, which pushes you toward a 12-16 GB VRAM card (RTX 4070 or 4080).

The GPU also accelerates Whisper STT. Whisper-base on CPU takes 1-2 seconds for a 5-second utterance. On GPU, it takes 200-400 milliseconds. This is the second-largest latency saving from the GPU after the LLM. For a system that aims for under 2 seconds total, the GPU acceleration of both STT and LLM generation is what makes the target achievable. Without it, you are fighting for milliseconds in stages that are already fast (embedding, retrieval) while losing seconds in the two stages that dominate the budget.

Low-Latency STT: Whisper Model Selection

For the STT stage, the model choice is a trade-off between accuracy and speed. Whisper-tiny (39M parameters) is the fastest but least accurate. Whisper-base (74M) is a good balance -- fast enough for real-time on a GPU, accurate enough for most home and office use. Whisper-small (244M) is more accurate but 3x slower. Whisper-medium (769M) and Whisper-large (1.5B) are the most accurate but too slow for real-time conversational use, even on a GPU. For a real-time conversational system, Whisper-base (or the English-only variant base.en, which is slightly faster) is the recommended choice. If accuracy is critical (medical dictation, technical terminology), Whisper-small on a GPU is viable, adding roughly 600 milliseconds to the pipeline.

Fast LLM: The 7B/8B Class

For sub-second response (the LLM generation stage), the 7B/8B model class is the sweet spot. Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B, and Phi-3.5 are all capable models that generate at 40-80 tokens per second on a consumer GPU (RTX 4060 or equivalent). A 50-word answer (70 tokens) at 60 tokens per second takes 1.17 seconds. At 80 tokens per second (an RTX 4070 with 4-bit quantization), it takes 0.875 seconds. These are the models that make the 2-second target achievable. Larger models (14B, 32B, 70B) are more capable but too slow for real-time voice interaction -- a 14B model on an RTX 4060 generates at roughly 20-30 tokens per second, pushing a 50-word answer to 2.5-3.5 seconds. For conversational RAG, where the answers are grounded in retrieved documents (not requiring the full reasoning power of a larger model), the 7B/8B class is the right trade-off of quality and speed.

TTS: Piper for Speed

As covered in the TTS section, Piper is the recommended choice for the real-time pipeline. It runs on CPU (freeing the GPU for the LLM and STT), synthesizes at 5-20x real-time (under 500ms for a typical answer), and supports streaming. The combination of a GPU-accelerated LLM and a CPU-based Piper TTS is the architecture that hits the 2-second target most reliably. If you switch to Coqui or Bark for higher quality, you need to account for the additional latency, either by accepting a longer total pipeline or by investing in a faster GPU.

The trajectory of conversational RAG leads to ambient AI. Not an app you open. Not a website you visit. Not a device you pick up. A presence in your home or office that you talk to naturally, the way you talk to a family member or a colleague who is in the room. You do not "launch" ambient AI. You do not "activate" it (beyond a wake word, which itself may fade as speaker identification improves). You simply speak, and it responds. It is there. It is always there. And it is yours.

Ambient AI is the endpoint of the conversational RAG pipeline described on this page. It is what happens when every component -- voice in, context management, multi-modal understanding, proactive surfacing, long-term memory, voice out -- works together seamlessly, at conversational speed, on local hardware. The user experience is not "using an AI tool." The user experience is "having an AI in the room." You ask it questions. It answers. It remembers what you talked about. It proactively tells you about appointments and deadlines. It can look at your medical images and describe what it sees. It learns your preferences over time. And all of it -- every word, every image, every preference, every conversation -- stays on hardware you own.

That entire interaction -- the proactive appointment reminder, the prescription refill, the image retrieval, the visual question answering, the conversational follow-up -- happened by voice, from the couch, in under 30 seconds, with no screen, no phone, no app. The system was proactive (it brought up the prescription), context-aware (it connected the dentist visit and the pharmacy trip), multi-modal (it looked at the MRI and described it), and sovereign (every byte stayed on the local server). This is ambient AI. It is not a prototype. It is not a mockup. It is an integration of existing, open-source, locally-runnable components. The technology is here. The packaging is the work that remains.

The Shift from App to Presence

The shift from app to presence is the defining transition of ambient AI. Today, AI is an app: you open ChatGPT, you open Claude, you open a search bar, you type a query, you close the app. The interaction is bounded by a session, a screen, and an intention to "use AI." Ambient AI dissolves those boundaries. The AI is not something you use. It is something that is there. You do not decide to interact with it any more than you decide to interact with the air. You speak. It hears. It answers. The interaction is unbounded by sessions (the memory persists), by screens (the interface is voice), and by intention (the system can speak first). This is a fundamentally different relationship with technology, and it is only possible when the AI is local, persistent, and private. A cloud AI cannot be ambient in the same way, because every interaction requires a network round-trip, a session, and the implicit acknowledgment that you are sending your data to a server. A local AI is ambient because it is always there, always listening (for the wake word), always ready, and always private.

The Sovereignty of Ambient AI

Ambient AI is the most intimate form of artificial intelligence. It is in your home. It hears your voice. It knows your schedule. It sees your medical images. It remembers your conversations. It learns your preferences. If any of this data were in the cloud, the privacy implications would be severe -- far more severe than the already-concerning implications of cloud chatbots. Ambient AI without sovereignty is surveillance. Ambient AI with sovereignty is a personal assistant that you own. This is why the sovereignty requirement is not a feature of conversational RAG -- it is the precondition for it. Without sovereignty, ambient AI is a microphone in your home that sends everything to a corporation. With sovereignty, it is a computer in your home that processes everything locally and keeps it yours.

The future of RAG is not a better search engine. It is not a smarter chatbot. It is a presence in your environment that you talk to naturally, that knows your life, that helps you proactively, and that is entirely, verifiably, and permanently yours. Every technical page in this RAG library -- the fundamentals, the chunking strategies, the embedding models, the vector databases, the retrieval methods, the multi-modal pipeline, the graph RAG -- exists to build toward this. The technology is ready. The architecture is known. The hardware is affordable. The sovereignty is non-negotiable. The work is integration. The future is ambient, and it is sovereign.