RAG Chunking Strategies in 2026: A Practical Field Guide

A chunking strategy is the rule a RAG pipeline uses to divide source documents into the units that get embedded, indexed, and retrieved. Picking one comes down to three questions: how much structure your documents carry, how much you can spend at ingest and at query time, and whether your retrieval failures come from chunks too small to answer from or too diluted to match against. Six approaches cover the practical space in 2026, from fixed-size windows you can ship in an afternoon to LLM-rewritten propositions that cost real money per document.

The decision deserves more care than it usually gets, because it is expensive to revisit. Changing strategy invalidates every vector in the index, and re-embedding a large corpus is a migration project with a bill attached. Chunk-strategy mistakes also rank among the five failure modes that kill RAG systems in production, and they are among the hardest to see: the pipeline runs, the index fills, and recall quietly sits fifteen points below where it could be.

What is a chunking strategy?

A chunk is the unit of retrieval. The embedding model never sees your documents. It sees chunks, similarity search happens at chunk granularity, and everything downstream inherits the choice.

Two pressures pull in opposite directions. Small chunks embed crisply: a 200-token passage about one topic produces a vector with a clear semantic address, and queries about that topic land on it reliably. But a small chunk often lacks the material needed to answer, with the definition in one chunk and the exception in the next. Large chunks carry enough to answer from, and they embed muddily. A 2,000-token chunk spanning four topics produces a vector pulled toward the average of all four, matching none of them well.

Every approach below is a position on that tension, plus a view on how much document structure to respect and how much compute to spend recovering it.

How do fixed-size and recursive splitters differ?

Fixed-size splitting cuts every document into windows of N tokens or characters, usually with some overlap between neighbors. It is trivial to implement, produces predictable output sizes, and is indifferent to content. On homogeneous plain prose it performs better than its reputation suggests. Its weakness is the boundary: it cuts mid-sentence, mid-table, and mid-argument, because it has no idea those things exist.

Recursive, structure-aware splitting fixes the worst of this at nearly the same price. LangChain's RecursiveCharacterTextSplitter, probably the most widely deployed chunker in production, tries to split on paragraph breaks first, then line breaks, then whitespace, falling back a level only when a piece still exceeds the size cap. Format-specific variants go further. Markdown and HTML splitters cut at headings and keep the heading path attached as metadata. Code splitters cut at function and class boundaries using language grammars. LlamaIndex ships the whole family as node parsers: SentenceSplitter, MarkdownNodeParser, CodeSplitter, and a HierarchicalNodeParser that builds parent-child trees.

For documents with real markup, recursive splitting at a 256-to-512-token target with heading metadata is the default to beat. It respects the boundaries an author already drew, and authors are decent chunkers.

When does semantic splitting earn its cost?

Semantic chunking drops the assumption that formatting reveals structure. Embed each sentence or small sentence group, walk the document in order, and split wherever cosine similarity between neighbors falls below a threshold. Topic shifts become boundaries even when the text is one undifferentiated block. Greg Kamradt's 2023 notebooks popularized the technique, and LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser are the packaged versions.

The costs are real: an embedding call per sentence at ingest, a threshold that needs tuning per corpus, and variable output sizes that complicate context budgeting. The published evidence is mixed in an instructive way. Chroma's 2024 technical report on chunking evaluation found recall gaps of several points between strategies on some corpora and near-parity on others, with no single winner across document types. Where semantic splitting reliably earns its keep is text with no usable markup: meeting transcripts, OCR output whose headings were flattened, scraped pages reduced to plain text. Where markup survives, structure-aware splitting usually matches it at a fraction of the ingest cost.

What does late chunking change?

Conventional pipelines split first and embed each piece in isolation, and isolation is the problem. A chunk reading "the company raised the figure again in Q3" embeds without knowing which company or which figure, because that context lived two paragraphs up. Late chunking, introduced by Jina AI in 2024, inverts the order. Run the entire document through a long-context embedding model once, then mean-pool the token-level vectors within each chunk's span. Each chunk vector is computed with the whole document in the attention window, so pronouns, abbreviations, and running themes stay resolved.

Storage cost is unchanged at one vector per chunk. Ingest cost rises with document length, and the technique needs an embedding model that exposes token-level output over long contexts, which in practice means a narrow set of model families, Jina's own v3 and v4 lines among them. That coupling matters if you have read about what an embedding-model migration actually involves, because late chunking deepens the tie to one vendor. Anthropic's contextual retrieval, published the same year, chases a similar effect from another angle: an LLM writes a short document-context preamble for each chunk before embedding. It works with any embedding model and pays per chunk in generation tokens instead.

How does parent-document retrieval work?

Parent-document retrieval, often called small-to-big, refuses to make one chunk do two jobs. Index small chunks for matching precision. When one matches, hand the model its parent: the enclosing section, page, or full document. LangChain's ParentDocumentRetriever and LlamaIndex's hierarchical nodes with auto-merging retrieval are the standard implementations, and the auto-merging variant swaps a set of matched siblings for their shared parent automatically.

This dissolves most of the size dilemma rather than balancing it. The match unit can be a tight 128-token passage while the read unit is a 1,500-token section. The price is plumbing: a document store alongside the vector store, parent-child bookkeeping, and dedup when three children all point at the same parent. The characteristic failure is the oversized parent, where a single match drags a 6,000-token section into context and crowds out every other result.

Do LLM-derived propositions pay off?

Propositional chunking rewrites documents before indexing them. An LLM decomposes the text into atomic, standalone statements, resolving pronouns and attaching context, so "It doubled the following year" becomes "Acme's cloud revenue doubled in 2024." The Dense X Retrieval paper (Chen et al., 2023) formalized the approach and found propositions outperforming sentence and passage units on fact-seeking benchmarks.

The appeal for fact-lookup workloads is obvious: every indexed unit is self-contained and precisely addressable. The costs are equally obvious. A full LLM pass over the corpus at ingest, the same cost again on every document update, and a new risk class, because the rewriting model can subtly distort a claim, and the distortion becomes what your system retrieves. Provenance links back to source spans are mandatory, and audits should sample them regularly. Reserve this for stable, high-value corpora where per-document spend is justified. Skip it for anything that churns daily.

How do size, overlap, and metadata trade off?

Three dials sit underneath every strategy.

Size trades match precision against context cost. Smaller chunks give retrieval a finer address space, and they force the system to assemble answers from more pieces, each retrieval slot carrying less material. Larger chunks deliver more per slot and blur the vectors. Rerankers changed this calculus. Over-retrieving 30 to 50 small chunks is cheap, a cross-encoder reranker can reorder them accurately because it reads the full query-chunk pair, and the final context gets the best 5 to 10. With a reranking stage in place, chunk granularity can optimize for recall and leave precision to the reranker.

Overlap is boundary insurance. For fixed-size windows, 10 to 20 percent is the common range, enough that a sentence severed by one boundary appears intact in a neighbor. Structure-aware splitters need less, since their cuts already fall on natural edges. Past roughly 25 percent, overlap mostly mints near-duplicate vectors that occupy multiple top-k slots with the same content.

Metadata is the cheapest upgrade on this list. Attach the heading path, document title, source, and date to every chunk. Filters on those fields shrink the search space before similarity runs, and prepending the heading path to the chunk text before embedding measurably helps queries that name a section or topic. Filter support and its performance characteristics vary across the 2026 vector store field, so check that your store handles the filter shapes you need at your scale.

How does the embedding model constrain the choice?

Every size decision is bounded by the embedding model. The hard bound is the sequence limit, and many hosted models truncate silently rather than raising an error, so oversized chunks lose their tails without any log line. The soft bound is effective length: a model trained mostly on short passages degrades well before its window fills, and leaderboard scores computed on 512-token benchmark passages say little about how the same model handles 3,000-token chunks. Test at the sizes you intend to use.

The pairing is also sticky in both directions. A new splitting strategy forces re-embedding, and a new embedding model forces it too, so bundle the two changes into one migration and one evaluation pass rather than paying for the corpus twice.

Which strategy fits which documents?

By document type, the decision usually resolves quickly.

Clean prose with markup (docs sites, wikis, internal knowledge bases): recursive structure-aware splitting with heading metadata, 256 to 512 tokens, minimal overlap. Add parent-document expansion when sections run long and queries span them.

Tables and PDFs: chunking is the second problem here, because document parsing quality bounds everything downstream. A table sliced mid-row is unrecoverable at query time. Extract tables as units, keep them whole where they fit, split by row groups with the header repeated where they do not, and chunk the surrounding prose separately.

Code: split at symbol boundaries with a grammar-aware splitter, never fixed windows, and attach file path and symbol names as metadata. A function cut in half retrieves as noise.

Transcripts: no markup to respect, so semantic splitting or sentence windows grouped by speaker turn, with timestamps and speaker identity as metadata. Boundaries at speaker changes beat boundaries at token counts.

Whatever you pick, build the held-out evaluation set first: a few hundred query and expected-document pairs from real usage, scored at recall@10. Run it before the change and after. Chunking choices argued from intuition tend to lose to chunking choices argued from that table.

Diagest is the data-for-AI pipeline we build at ixprt. It ingests, cleans, dedupes, and prepares source material for retrieval, so the splitting layer downstream receives documents whose structure survived the trip.