A Complete Guide to Splitting Documents for Retrieval -- From Fixed-Size to Agentic, With Trade-Offs, Code Examples, and Production Recommendations for Sovereign, On-Premise Systems
The Art and Science of Splitting Documents for Retrieval-Augmented Generation
Chunking is the process of splitting documents into smaller, retrievable units before embedding them in a vector database. It is the single most underappreciated lever in RAG system quality. A perfectly tuned embedding model and a state-of-the-art LLM will still produce wrong answers if the chunks fed to the retriever cut a sentence in half, bury a key clause inside an unrelated paragraph, or isolate a table from its header row. This guide covers every major chunking approach in depth -- fixed-size, sentence-aware, paragraph-aware, recursive, semantic, document-aware, and agentic -- along with overlap strategies, chunk-size guidance, multi-modal chunking for audio/video/images, practical recommendations by document type, a full comparison table, and readable code examples. Every recommendation is framed for sovereign, on-premise RAG deployments where data never leaves your infrastructure.
You cannot feed an entire document into a vector database as a single embedding and expect useful retrieval. A 200-page contract embedded as one vector produces a single point in vector space -- a vague average of everything the document says. A query about "termination clause" and a query about "payment terms" both match that same point equally well (or equally poorly), and the retriever has no way to pull out the specific paragraph that answers either question. The LLM receives the entire document or nothing, and for long documents, "entire document" exceeds the context window anyway.
Chunking solves this by splitting documents into smaller units -- paragraphs, sentences, sections, or topically coherent passages -- each of which gets its own embedding. When a user asks "What is the termination notice period?", the retriever can pull the specific chunk containing the termination clause and feed only that to the LLM. The answer is precise, sourced, and cheap (fewer tokens to generate over).
Chunking sits at the very start of the RAG pipeline, and every downstream stage depends on it:
Bad chunking at step one compounds through every subsequent stage. A chunk that cuts a sentence in half produces a broken embedding -- the vector represents a fragment of meaning, not a complete thought. When retrieved, it gives the LLM an incomplete picture. The LLM, lacking the missing half, either hallucinates the rest or says "I don't know." Either way, the user gets a worse answer than they should have. Bad chunking equals bad retrieval equals bad answers. This is not a theoretical concern -- it is the most common cause of RAG quality problems in production, more common than weak embedding models or small LLMs.
Modern LLMs support context windows of 128K, 200K, even 1M tokens. Why chunk at all? Why not dump the entire document into the prompt? Three reasons. First, cost: generation pricing is per input token. Feeding 100K tokens when 500 tokens would answer the question is a 200x cost penalty. Second, latency: time-to-first-token scales with input length. A 100K-token prompt is slower than a 2K-token prompt, every time. Third, attention dilution: LLMs suffer from the "lost in the middle" phenomenon -- information in the middle of a long context is retrieved and used less accurately than information at the start or end. Precise chunking plus targeted retrieval puts only the relevant 500 tokens in the prompt, maximizing accuracy, minimizing cost, and minimizing latency. Chunking is not a workaround for small context windows; it is an optimization for quality, speed, and cost at any context size.
Fixed-size chunking is the simplest strategy: split the document into chunks of exactly N characters or N tokens, where N is a constant. If the document is 10,000 characters and N is 500, you get 20 chunks of 500 characters each. That is the entire algorithm. There is no linguistic analysis, no structure awareness, no semantic reasoning. The splitter counts characters (or tokens) and cuts.
You choose a chunk size (e.g., 500 characters or 128 tokens) and optionally an overlap (e.g., 50 characters). The splitter walks through the text and emits a chunk every N units. With overlap, the last O characters of chunk K are repeated as the first O characters of chunk K+1, creating a sliding window.
Fixed-size chunking is appropriate for uniform documents where structure is either absent or irrelevant -- log files, transcripts without speaker labels, homogeneous text streams, and quick prototyping where you need a baseline to compare smarter strategies against. It is also a reasonable default for very large corpora where the cost of smarter chunking (embedding model inference for semantic chunking, LLM calls for agentic chunking) is prohibitive. For a 10-million-document ingestion pipeline running on limited hardware, fixed-size chunking at 512 tokens with 10% overlap is a perfectly defensible production choice -- the retrieval quality loss is small compared to the cost of semantic chunking at that scale.
Sentence-aware chunking splits text at sentence boundaries instead of arbitrary character counts. The algorithm first detects sentences using an NLP library (spaCy, NLTK, or a simpler regex-based sentence tokenizer), then groups sentences together until the target chunk size is reached. The result: every chunk starts at the beginning of a sentence and ends at the end of a sentence. No chunks are cut mid-sentence.
The sentence tokenizer walks the text and identifies sentence boundaries -- periods followed by spaces and capital letters, but handling abbreviations ("Dr.", "Inc.", "U.S.") and edge cases ("What about 3.14?"). Once sentences are separated, a greedy accumulator adds sentences to the current chunk until adding the next sentence would exceed the target size. At that point, the chunk is emitted and a new one begins.
nltk.sent_tokenize() uses Punkt, a pre-trained unsupervised tokenizer. Lightweight and reliable for well-formed English text.[.!?]\s+[A-Z]. Fast but fragile -- mis-handles abbreviations, decimal numbers, and ellipses.Sentence-aware chunking is the right step up from fixed-size for prose documents: articles, reports, narrative documents, blog posts, emails, and any text where individual sentences carry complete meaning. It is a strong default for general-purpose RAG and is the strategy most "simple" RAG tutorials should actually use instead of fixed-size. The upgrade in retrieval quality over fixed-size is noticeable and immediate, while the added cost (a spaCy model load) is negligible.
Paragraph-aware chunking splits at paragraph boundaries instead of sentence boundaries. A paragraph is a natural unit of thought: the writer grouped those sentences together because they discuss a single subtopic. Respecting that boundary produces chunks that are topically more coherent than sentence-grouped chunks, which may accidentally merge two paragraphs discussing different aspects of a topic.
The splitter detects paragraph boundaries (typically double newlines, or indentation in formatted documents) and treats each paragraph as an atomic unit. Paragraphs are accumulated into chunks until the target size is reached. If a single paragraph exceeds the target size, the splitter falls back to sentence-level splitting within that paragraph, and if a single sentence exceeds the target, it falls back to character splitting. This cascade ensures no chunk exceeds the limit while respecting the highest available structural boundary.
Paragraph-aware chunking is ideal for structured documents with clear sections: technical reports, policy documents, manuals, legal briefs with well-formed paragraphs, and any prose where the author's paragraph structure reflects topical organization. It is strictly better than sentence-aware chunking when paragraphs are clean and consistently formatted. It is the recommended strategy for most general-purpose RAG deployments that handle prose documents and do not want the complexity of semantic or document-aware chunking.
Recursive chunking is the strategy popularized by LangChain's RecursiveCharacterTextSplitter and it is the default most practitioners should reach for. The idea is elegant: define a hierarchy of separators, from most-structural to least-structural, and try each in turn. First, try splitting by paragraphs. If the resulting chunks fit within the target size, you are done. If a chunk is still too large, recursively apply the next separator (sentences) to just that chunk. If still too large, try words. If still too large, try characters. The splitter descends through the hierarchy until every chunk fits.
The default hierarchy in LangChain for plain text is:
The algorithm tries the first separator. If any resulting piece exceeds the chunk size, it recursively applies the second separator to just that piece. This continues down the hierarchy. The result is chunks that respect the highest available structural boundary that still fits within the size limit. A document with clean paragraph breaks produces paragraph-aligned chunks. A document with one giant paragraph gets sentence-split. A document with one giant sentence gets word-split. The splitter adapts to whatever structure the document actually has.
chunk_size is in characters, not tokens. 512 characters is roughly 100-130 tokens for English, not 512 tokens. You must pass a token-based length function or use a tokenizer-aware splitter to get true token-bounded chunks.Recursive chunking is the best default for mixed-format document collections. If you are building a RAG system over a knowledge base containing PDFs, Markdown files, plain text, and Word documents, recursive chunking handles all of them without per-document-type tuning. It is the strategy most production RAG systems should start with before considering any of the more advanced approaches below. The cost-to-quality ratio is exceptionally good: near-zero compute cost, substantial improvement over fixed-size, and it degrades gracefully on messy documents.
Semantic chunking uses embeddings to detect topic shifts and splits at those boundaries. The insight: a document is not a linear sequence of paragraphs -- it is a sequence of topics, and topic boundaries do not always align with paragraph breaks. By embedding each sentence (or small unit) and measuring the semantic distance between consecutive units, the splitter can identify where the topic changes and cut there. The resulting chunks are topically coherent: each chunk discusses one topic, not a mix of two.
Semantic chunking shines on long technical documents, research papers, legal contracts, and any prose where topics span multiple paragraphs or shift within them. It is the first strategy in this list that actually understands what the text is about, and the retrieval precision improvement over recursive chunking is often the difference between a RAG system that feels "pretty good" and one that feels "surprisingly accurate." For on-premise deployments, the embedding model runs locally (no API calls, no data egress), and the per-document cost is seconds of CPU time -- an excellent trade for better retrieval.
Document-aware chunking respects the actual structure of the document: headers, sections, tables, code blocks, lists, footnotes, and captions. Instead of treating the document as a flat string of text, it parses the document's structure tree and uses that tree to define chunk boundaries. A Markdown H2 section becomes a chunk boundary. An HTML table stays together as one chunk (or is split at row boundaries if too large). A code block is never split mid-function. The result is chunks that match the document's authored structure.
Markdown is the easiest format for document-aware chunking because its structure is explicit in the syntax. Headers (#, ##, ###) define sections. Code fences (```) define code blocks. Tables have pipe-delimited rows. A Markdown-aware splitter walks the document's header tree and creates one chunk per section (or per subsection, depending on target size). The section's header is prepended to each chunk so the embedding and the LLM both know what section they are reading.
For HTML documents, an HTML-aware splitter parses the DOM tree and splits at block-level element boundaries (<div>, <section>, <article>, <table>, <pre>). It preserves table structure, does not split inside a <pre> block (code), and can carry tag-based metadata (e.g., "this chunk came from a <table>") into the vector database for filtered retrieval.
For source code, a code-aware splitter parses the abstract syntax tree (AST) and splits at function, class, and method boundaries. It never splits inside a function body. The chunk includes the function signature and docstring. LangChain's RecursiveCharacterTextSplitter.from_language() supports Python, JavaScript, TypeScript, Go, Rust, Java, C++, and other languages.
PDFs are the hardest format for document-aware chunking because PDF structure is visual, not semantic. The PDF format specifies where glyphs appear on a page, not whether they are a header, body text, or caption. Tools like unstructured.io, PyMuPDF, pdfplumber, and marker attempt to reconstruct document structure from layout analysis: detecting headers by font size, identifying tables by ruled lines or alignment, and separating columns. The quality varies enormously by PDF type. Born-digital PDFs (exported from Word or LaTeX) preserve structure better than scanned PDFs (which require OCR first).
Document-aware chunking is the right choice for technical documentation, code repositories, API documentation, wikis, manuals, and any corpus where documents have meaningful structure. If your users will ask questions like "How do I configure the Docker installation?" and the document has a section literally titled "Docker Installation," document-aware chunking ensures that section is a single retrievable chunk -- not scattered across arbitrary 512-token windows. For on-premise deployments over internal documentation (API docs, runbooks, SOPs), this is often the highest-ROI chunking upgrade available.
Agentic chunking is the most expensive and most intelligent approach: an LLM reads the document and decides the optimal chunk boundaries itself. Instead of applying a fixed algorithm, the LLM is prompted to analyze the document's content, identify natural topical and structural boundaries, and propose chunks that maximize each chunk's internal coherence and retrievability. This is the chunking strategy that most closely mimics what a human expert would do if asked to split a document for a Q&A system.
The document is fed to the LLM with a prompt like: "You are a RAG chunking expert. Read the following document and split it into chunks optimized for retrieval. Each chunk should be topically coherent, self-contained enough to understand without surrounding context, and between 200 and 800 tokens. Output the chunks as a JSON array of strings." The LLM reads the document, identifies natural boundaries, and returns the chunks.
Agentic chunking is reserved for high-value documents where retrieval accuracy is critical and the cost of a wrong answer (or a missed answer) is high: legal contracts in a law firm RAG system, clinical guidelines in a medical RAG system, regulatory documents in a compliance system, and executive briefings. It is also appropriate for small, high-value corpora (a few hundred critical documents) where the one-time ingestion cost is acceptable and the corpus does not change often. For large, frequently-updated corpora (millions of documents, daily ingestion), agentic chunking is usually prohibitively expensive.
Overlap is the practice of repeating a small portion of each chunk at the beginning of the next chunk. If chunk K ends with the last 50 tokens of a paragraph, chunk K+1 begins with those same 50 tokens before continuing. This creates a sliding window where content near a boundary appears in both chunks, ensuring that information at the edge of a chunk is not lost from either side's context.
Consider a sentence that straddles a chunk boundary: "The effective date of this agreement is January 15, 2026, and it shall remain in force for..." -- chunk K ends at "January 15, 2026," and chunk K+1 starts at "and it shall remain in force for..." Without overlap, the embedding for chunk K captures "the effective date is January 15, 2026" but not the duration. The embedding for chunk K+1 captures "shall remain in force for [duration]" but not the date. A query about "When does the agreement start and how long does it last?" matches neither chunk strongly.
With 50-token overlap, chunk K+1 begins with the last 50 tokens of chunk K, including "The effective date of this agreement is January 15, 2026," followed by "and it shall remain in force for..." Now chunk K+1's embedding captures both the date and the duration. The query matches strongly. Overlap ensures that information near a boundary is represented in at least one complete chunk.
The standard recommendation is 10-20% of the chunk size. For a 512-token chunk, that is 50-100 tokens of overlap. For a 256-token chunk, 25-50 tokens. For a 1024-token chunk, 100-200 tokens. The trade-off:
Chunk size is the single most important parameter in any chunking strategy, and it is not one-size-fits-all. The right size depends on what you are optimizing for: retrieval precision (small chunks), generation context (large chunks), or a balance of both. Here is the guidance, from small to large.
Small chunks produce focused embeddings. A 256-token chunk covers roughly one paragraph or a few sentences. The embedding captures a specific, narrow meaning. When the user asks a specific question ("What is the penalty for late payment under Section 4?"), the retriever can pull the exact paragraph that answers it, without surrounding noise. The LLM gets a precise, relevant context.
The trade-off: small chunks may lack the context the LLM needs to understand the retrieved passage. A chunk that says "The penalty is 1.5% per month" is only useful if the LLM knows it is about late payments under Section 4 -- which may require context from a chunk that was not retrieved. Small chunks work best when each chunk is self-contained (agentic or document-aware chunking with section headers prepended) and when you retrieve multiple chunks (top-k=5 or more) to provide surrounding context.
512 tokens is the most commonly recommended default and for good reason. It is roughly 350-400 words, about one to two paragraphs. Large enough to contain a complete idea with its surrounding context (the topic sentence, the explanation, and the conclusion). Small enough that the embedding is focused and the retriever can pinpoint relevant passages without too much noise. For most RAG systems handling prose documents, 512 tokens with 50-100 tokens of overlap is the sweet spot.
Larger chunks carry more context per retrieval. A 1024-token chunk might contain an entire section of a document, including background, main point, and examples. When retrieved, the LLM gets enough context to understand the passage without needing surrounding chunks. This reduces the need for high top-k retrieval (you can retrieve fewer chunks and still have enough context).
The trade-off: larger chunks have more diffuse embeddings. A 1024-token chunk covering two subtopics will match queries about either subtopic weakly, rather than one subtopic strongly. Retrieval precision drops. Larger chunks also mean fewer chunks per document, which means coarser retrieval granularity -- you cannot retrieve just the relevant paragraph, you retrieve the whole section it sits in.
With modern LLMs supporting 128K+ token context windows, there is a temptation to use very large chunks (2048, 4096, or even 8192 tokens). The argument: if the LLM can handle long context, give it long context. This works when the document is highly topically focused (a single long technical section that should not be split) and when the embedding model can represent long passages meaningfully (some embedding models degrade on very long inputs -- check your model's optimal input length).
The trade-off is severe: very large chunks have very diffuse embeddings, retrieval precision drops significantly, and you are feeding the LLM a lot of potentially-irrelevant context (which hurts accuracy via attention dilution and increases cost). Use 2048+ token chunks only when the document's structure demands it (a single long section that cannot be meaningfully split) and combine with a re-ranking step to ensure the retrieved large chunk is actually relevant.
| Chunk Size | ~Word Count | Retrieval Precision | Context Per Chunk | Best For |
|---|---|---|---|---|
| 256 tokens | ~180 words | High | Low | Factoid Q&A, specific clause lookup, self-contained chunks |
| 512 tokens | ~375 words | Balanced | Balanced | General-purpose RAG, most prose documents |
| 1024 tokens | ~750 words | Moderate | High | Technical docs, multi-paragraph reasoning, low top-k retrieval |
| 2048+ tokens | ~1500+ words | Low (without re-ranking) | Very High | Long-context models, highly focused sections, with re-ranking |
RAG systems are increasingly multi-modal: users ask questions about audio recordings, video footage, and images, not just text. Chunking strategies for these modalities differ fundamentally from text chunking because the "natural boundaries" are different. You cannot split an audio file at "sentence boundaries" the same way you split text -- you split at silence, speaker changes, or topic shifts detected from a transcript. This section covers the major multi-modal chunking approaches. For a deeper treatment, see our multi-modal RAG page.
Audio (meeting recordings, interviews, podcasts, lectures) is chunked by temporal boundaries that align with natural pauses or speaker changes:
pydub, librosa, webrtcvad (Voice Activity Detection).pyannote.audio, whisper-x with diarization, resemblyzer.Video combines visual and audio tracks. Chunking must consider both:
PySceneDetect, OpenCV with histogram differencing.Images do not have natural sequential boundaries like text or audio. Chunking an image means deciding what constitutes a retrievable unit:
Different document types have different structures, and the best chunking strategy depends on the document. Here are concrete recommendations for common document types in enterprise RAG systems.
Recommended: Document-aware chunking by section/clause, with 512-token chunks and 64-token overlap. Legal contracts are highly structured: numbered sections, lettered subsections, defined terms. Split at section boundaries. Prepend the section number and title to each chunk ("Section 4.2: Termination for Convenience"). Do not split a clause across chunks -- a single clause should be one chunk, even if it exceeds the target size. For high-value contracts (M&A agreements, IP licenses), upgrade to agentic chunking so the LLM can recognize that a defined term in Section 12 refers back to the definition in Section 1 and ensure that context is preserved.
Recommended: Document-aware chunking by encounter/section, with 256-512 token chunks and 50-token overlap. Medical records are structured by encounter (visit, lab result, imaging report, medication change). Split at encounter boundaries. Each chunk should carry metadata: patient ID (or de-identified ID), encounter type, date, provider. Never split a single encounter across chunks. For clinical notes with narrative sections (HPI, Assessment, Plan), use paragraph-aware chunking within the encounter. For on-premise deployments in HIPAA-regulated environments, all chunking must happen on-premise with no API calls -- use local spaCy + document-aware splitters, not cloud-based agentic chunking.
Recommended: Code-aware AST-based chunking by function/class/method, with 512-1024 token chunks and zero overlap (code units do not benefit from overlap the way prose does). Each chunk should include the function signature, docstring, and full body. Never split inside a function. For files with multiple small functions, group related functions (same class, same module) into one chunk if they fit. Carry metadata: file path, language, class name, function name. This enables queries like "show me the authentication logic" to retrieve the relevant function, not a random 512-token slice that might contain half a function.
Recommended: Document-aware (Markdown/HTML) chunking by section, with 512-token chunks and 64-token overlap. Manuals are structured by chapters and sections. Split at header boundaries. Prepend the section breadcrumb ("Installation > Docker > Configuration") to each chunk. Keep code examples together with their explanation -- do not split the "Run this command:" sentence from the command itself. For API documentation, keep the endpoint definition, parameters, and example response as one chunk.
Recommended: One email per chunk, or paragraph-aware chunking for long emails, with 256-512 token chunks. Most emails are short enough to be a single chunk. The email metadata (sender, recipient, date, subject) should be prepended to the chunk text so the embedding and the LLM both have context. For long email threads, split at message boundaries (each email in the thread is a chunk) and carry the thread subject as metadata. Overlap is usually unnecessary since each email is a natural unit.
Recommended: Transcript-then-chunk approach. Transcribe with a local Whisper model (on-premise for confidentiality), then apply semantic or sentence-aware chunking to the transcript. Each chunk is linked to its audio segment via timestamps. For meetings, split at speaker/turn boundaries and carry speaker labels as metadata. For interviews, keep each question-answer pair as one chunk if it fits the size limit. Chunk size: 512 tokens of transcript, 64-token overlap. Store both the text embedding (for retrieval) and the audio segment (for playback to the user).
Recommended: Scene-based chunking with transcript alignment. Detect scene boundaries, extract keyframes, caption keyframes with a local vision-language model (LLaVA, Qwen-VL), transcribe audio with Whisper, and apply semantic chunking to the transcript. Each chunk has: text (transcript segment), image (keyframe), and timestamps. Embed the text for retrieval. For training videos, chunk at topic boundaries (often aligns with slide changes). For surveillance, chunk by minute or by event detection (motion, person appearance).
| Document Type | Recommended Strategy | Chunk Size | Overlap | Key Metadata |
|---|---|---|---|---|
| Legal Contracts | Document-aware (by clause/section) | 512 tokens | 64 tokens | Section number, clause type, document title |
| Medical Records | Document-aware (by encounter) | 256-512 tokens | 50 tokens | Patient ID, encounter type, date, provider |
| Source Code | Code-aware (AST, by function) | 512-1024 tokens | 0 | File path, language, class, function name |
| Technical Manuals | Document-aware (Markdown/HTML) | 512 tokens | 64 tokens | Section breadcrumb, document title |
| Emails | One-per-chunk / paragraph-aware | 256-512 tokens | 0-32 tokens | Sender, recipient, date, subject, thread ID |
| Audio Recordings | Transcript-then-chunk + semantic | 512 tokens (transcript) | 64 tokens | Speaker, timestamp, audio file offset |
| Video | Scene-based + transcript-then-chunk | 512 tokens (transcript) | 64 tokens | Scene ID, keyframe, timestamp, duration |
The following table summarizes every chunking strategy covered in this guide, ranked roughly from simplest to most sophisticated. Use it as a quick reference when deciding which strategy to evaluate for your use case.
| Strategy | Complexity | Context Preservation | Retrieval Precision | Best For | Tool / Library |
|---|---|---|---|---|---|
| Fixed-Size | Very Low | Poor (cuts mid-sentence) | Low | Uniform logs, prototyping, baseline | Native Python, LangChain CharacterTextSplitter |
| Sentence-Aware | Low | Fair (complete sentences) | Fair | Articles, reports, narrative prose | spaCy, NLTK, pySBD, Stanza |
| Paragraph-Aware | Low | Good (paragraph boundaries) | Good | Structured prose, manuals, reports | Native Python, LlamaIndex SentenceSplitter |
| Recursive | Low-Moderate | Good (best available boundary) | Good | Mixed-format corpora, general default | LangChain RecursiveCharacterTextSplitter |
| Semantic | Moderate | Very Good (topic-coherent) | High | Long technical docs, research papers | LangChain SemanticChunker, LlamaIndex SemanticSplitter, custom |
| Document-Aware | Moderate-High | Very Good (structure-aware) | High | Technical docs, code, Markdown, HTML | LangChain MarkdownHeaderTextSplitter, HTMLSplitter, unstructured.io |
| Agentic | Very High | Excellent (LLM-decided) | Excellent | High-value docs, critical retrieval accuracy | Custom (LLM + prompt engineering), proposition-based systems |
Source: Avondale.AI Technical Documentation Library
Category: RAG Engineering
Type: Technical Reference Guide
License: CC BY 4.0
Suggested Citation:
Chunking Strategies for RAG: Complete Guide to Splitting Documents for Retrieval. Avondale.AI Technical Documentation Library. Accessed June 2026. https://avondale.ai/technical/rag/chunking-strategies.html