Every RAG system has two halves: a retriever that decides which text the model sees, and a generator that answers from it. Most teams iterate endlessly on the generator — better prompts, bigger models, careful instructions — while the retriever’s most consequential parameter goes unexamined: how documents were cut into chunks in the first place. Chunking determines what a single unit of retrieval can possibly contain. If the answer to a user’s question is split across two chunks, or if a chunk blends three unrelated topics into one diluted embedding, no amount of prompt engineering recovers it.
This post walks through the chunking strategies that matter in 2026 — fixed-size splitting, recursive splitting, semantic chunking, document-structure chunking, late chunking, and contextual retrieval — with what benchmarks actually show about each. The headline finding from the current research: benchmarks disagree, which is itself the most useful signal. The right strategy depends on your documents and your queries, and the only defensible way to pick is to measure recall on your own data.
Why Chunking Controls Retrieval Quality
An embedding model compresses a text span into a fixed-size vector. Everything in that span — every topic, every caveat, every aside — averages into a single point in vector space. Split a long product manual into 200-token slices and the paragraph describing “refund policy for EU customers” competes with fragments of unrelated sections; make chunks 4,000 tokens and each embedding is so diluted that it matches nothing precisely. This is the fundamental tension: chunks small enough to be topically pure lose the surrounding context that makes them interpretable, and chunks large enough to be self-contained embed poorly.
Two additional constraints shape the choice. Embedding models have token limits — a span longer than the model’s context gets truncated, silently dropping content. And whatever you retrieve gets pasted into the generator’s context window, where longer contexts degrade answer quality: research on long-context degradation has repeatedly shown retrieval accuracy falling as context grows even on trivial tasks, and a 2026 systematic analysis identified a practical quality cliff around 2,500 tokens of retrieved context. Bigger chunks are not just diluted; they also eat the budget where quality actually lives.
Fixed-Size and Recursive Splitting: The Working Defaults
Fixed-size chunking cuts text every N tokens, usually with a small overlap so a sentence split at a boundary appears whole in at least one chunk. It is trivially cheap and produces uniform chunks. Recursive character splitting improves on it with a hierarchy of separators: try to split on paragraph breaks first, fall back to sentences, then to words, so cuts land on natural boundaries wherever possible while still targeting the size budget. A minimal implementation makes the mechanics concrete:
def recursive_split(text: str, max_chars: int = 2000,
separators: list[str] | None = None) -> list[str]:
seps = separators if separators is not None else ["\n\n", "\n", ". ", " ", ""]
if len(text) <= max_chars:
return [text.strip()] if text.strip() else []
for sep in seps:
parts = text.split(sep)
if len(parts) == 1:
continue # separator not present; try the next finer one
chunks, current = [], ""
for part in parts:
candidate = (current + sep + part) if current else part
if len(candidate) <= max_chars:
current = candidate
else:
if current:
chunks.append(current.strip())
if len(part) > max_chars: # still too big: recurse with finer separators
chunks.extend(recursive_split(part, max_chars, seps[seps.index(sep)+1:]))
else:
current = part
if current:
chunks.append(current.strip())
return [c for c in chunks if c]
return [text[i:i+max_chars] for i in range(0, len(text), max_chars)]The same logic backs the standard LlamaIndex and LangChain splitters. Across the published benchmarks, recursive splitting in the 400–512 token range is the most consistently strong default: one 2026 benchmark of seven strategies over 50 academic papers put recursive 512-token splitting first at 69% accuracy, and earlier tests placed it at 85–90% recall. The commonly cited starting recipe — 400–512 tokens, 10–20% overlap — is a reasonable default, though the overlap assumption deserves scrutiny: a January 2026 systematic analysis using SPLADE retrieval on Natural Questions found overlap provided no measurable benefit while increasing index size. Overlap is a hypothesis to test, not a law.
Semantic Chunking: Better Boundaries, Real Costs
Semantic chunking embeds each sentence, computes similarity between adjacent sentences, and cuts where similarity drops — a topic shift. The appeal is obvious: boundaries land where meaning changes rather than where a token counter happens to trip. The costs are equally real: you embed every sentence at indexing time (several times the embedding cost of the final chunks), you tune a similarity threshold, and results are not reliably better. One frequently cited figure puts semantic chunking’s recall improvement at up to 9% over simpler methods; other evaluations found it underperforming — the same 50-paper benchmark saw it land at 54% behind recursive splitting’s 69%, producing fragmented chunks averaging just 43 tokens, and HotpotQA tests with ColBERT-style embeddings found plain sentence splitting ahead of semantic approaches.
The pattern across studies: semantic chunking wins when documents have long, internally-coherent sections with sharp topic shifts, and loses when its threshold mis-fires on dense technical text where every sentence shares vocabulary. Try it third, after fixed-size and recursive, and only with a retrieval metric in hand.
Structure-Aware Chunking: Let the Document Decide
The strongest chunk boundaries usually already exist in the document. Markdown headings, HTML sections, PDF pages, code ASTs — structural boundaries are semantic boundaries that came free. Three flavors matter:
- Page-level chunking treats each page as a chunk for paginated documents. NVIDIA’s widely referenced 2024 comparison of seven strategies across five datasets found page-level chunking won with 0.648 accuracy and the lowest variance — with the caveat that it only applies where pagination is meaningful.
- Header-based chunking splits markdown or HTML on heading structure, keeping each section intact. For documentation and wikis this is usually the right default, since a section under “Refund Policy” arrives at the generator with its topic already coherent.
- Parent-document retrieval decouples the two sizes deliberately: embed small chunks for precise matching, but return their larger parent (section or page) to the generator. You get small-chunk embedding quality with big-chunk context, at the cost of storing two granularities and mapping between them.
For code, AST-based splitting (cutting at function and class boundaries rather than mid-identifier) is the structural equivalent, and research on syntax-aware chunking shows meaningful gains over character counting on source files.
Late Chunking: Context at the Embedding Level
Late chunking, introduced by Jina AI researchers in 2024, inverts the pipeline. Instead of chunking first and embedding each piece separately, you run the entire document through a long-context embedding model and pool token-level embeddings into chunk vectors only afterward. Because every token’s embedding was computed with the full document in its attention window, each chunk vector carries document-wide context — pronouns resolve, domain shorthand makes sense — without any per-chunk preprocessing:
import torch
from transformers import AutoModel, AutoTokenizer
model_name = "jinaai/jina-embeddings-v3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
def late_chunk(document: str, chunk_size: int = 512) -> list[list[float]]:
inputs = tokenizer(document, return_tensors="pt", truncation=True,
max_length=8192)
with torch.no_grad():
hidden = model(**inputs).last_hidden_state[0] # (tokens, dim)
ids = inputs["input_ids"][0]
vectors = []
for start in range(0, len(ids), chunk_size):
end = min(start + chunk_size, len(ids))
vectors.append(hidden[start:end].mean(dim=0).tolist())
return vectorsThe requirements: an embedding model whose architecture exposes token-level outputs (encoder models like BERT-family work; autoregressive decoders do not fit this pattern), and a context window long enough for your whole document — chunks beyond the model’s window lose the benefit. When those hold, the trade is attractive: one forward pass per document instead of one per chunk, and context without an LLM in the indexing loop.
Contextual Retrieval: Prepending Document Context
Anthropic’s contextual retrieval takes a different route to the same problem: use an LLM at indexing time to write a short situating paragraph for each chunk — what document it comes from, what section, what it’s about — and prepend that context before embedding. A chunk that says only “the fee is waived for accounts over $10,000” now arrives with “From the Acme Bank fee schedule, section on premium checking accounts” attached. Anthropic reported retrieval failure rates dropping substantially with the technique, compounded further when combined with reranking and BM25 hybrid search.
The cost profile differs sharply from late chunking: one LLM call per chunk at indexing time, which is why prompt caching makes it economical. Late chunking moves context into the embedding step cheaply but constrains model choice; contextual retrieval costs LLM tokens per chunk but works with any embedding model. Hybrid retrieval — adding classic sparse BM25 scoring alongside vectors — is orthogonal to both and compounds with either.
What the Benchmarks Actually Agree On
Line up the major studies and the disagreements are loud — NVIDIA’s winner (page-level) can’t even run on non-paginated corpora; semantic chunking is someone’s +9% and someone else’s last place; a clinical-domain study found structure-aligned adaptive chunking at 87% accuracy versus 13% for fixed-size baselines, a gap no generic benchmark reproduces. But three conclusions survive the noise:
- Recursive splitting at 400–512 tokens is the right default. It is cheap, robust, and at or near the top in most evaluations. Start there.
- Query type changes the optimum. Factoid queries favor smaller chunks (256–512 tokens); analytical queries that need synthesis favor larger ones (1,024+). If your product serves both, that’s an argument for parent-document retrieval, not a compromise size.
- Domain structure beats generic cleverness. Where your documents have real structure — pages, headers, sections — exploiting it outperforms threshold-tuned semantic methods.
Measuring Before You Choose
Chunking decisions should be driven by a retrieval evaluation set: a few dozen real queries with known relevant passages. The core metric is recall@k — of the passages a human says answer the query, what fraction appear in the top k retrieved chunks. Track precision (retrieved chunks that are actually relevant), plus a ranking measure like MRR or NDCG to catch the case where the right chunk is retrieved but buried at position 8. The evaluation loop is deliberately small:
def recall_at_k(results: list[list[str]], relevant: list[set[str]], k: int = 5) -> float:
hits = sum(
1 for retrieved, rel in zip(results, relevant) if rel & set(retrieved[:k])
)
return hits / len(relevant)
def mrr(results: list[list[str]], relevant: list[set[str]]) -> float:
total = 0.0
for retrieved, rel in zip(results, relevant):
for rank, chunk_id in enumerate(retrieved, start=1):
if chunk_id in rel:
total += 1.0 / rank
break
return total / len(relevant)Run it over two or three candidate strategies on your actual documents. An afternoon of evaluation on your data settles a question that no amount of benchmark-reading can. When recall is high but answers are still wrong, the bottleneck has moved downstream — that’s the moment to reach for reranking (a cross-encoder rescoring the top candidates) or contextual retrieval, not a re-chunk.
Wrapping Up
Chunking is where RAG quality is decided, and it rewards boring discipline over fashionable complexity: recursive splitting at 400–512 tokens as the default, document structure exploited wherever it exists, parent-document retrieval when queries span factoid and analytical modes, and late chunking or contextual retrieval as targeted upgrades when measured recall says you need them. The benchmarks disagree with each other, so treat every published number — including the ones here — as a hypothesis to check against your own retrieval set. Build the small evaluation harness first; every chunking decision after that becomes an experiment instead of a guess.