Every RAG pipeline starts with the same piece of code: split documents into chunks, embed them, store them. It looks like plumbing. It is not. The choice of how you split text can move retrieval quality more than swapping embedding models, and the research on which strategy wins is messier — and more useful — than most tutorials suggest. Two findings keep colliding in public evaluations: sophisticated semantic chunking frequently loses to a dumb fixed-size splitter, and the techniques that reliably help do not change where you cut at all. They change what each chunk knows about its surroundings.
This post walks through what the benchmark literature actually measures, why the obvious baseline keeps winning, and how the two context-preserving techniques worth adopting — late chunking and Contextual Retrieval — attack the same failure from opposite directions. Then it gives you a minimal evaluation harness so you can stop arguing about chunking and start measuring it on your own corpus.
Why Standard Benchmarks Cannot Rank Chunkers
The benchmarks people reach for first — MTEB, BEIR, MSMARCO — evaluate retrieval at the document level. They measure whether a system surfaces the right whole document, ranked by nDCG@10. Chunking never enters the picture, because these benchmarks were built for classical information retrieval, where the unit of return is a document. If you chunk a corpus and retrieve chunks, you are playing a different game than the one those benchmarks score.
A technical report from Chroma made this concrete by evaluating retrieval at the token level instead. The idea: use an LLM to generate queries plus the exact text spans that answer them, then score retrieved chunks by how well their tokens overlap the relevant tokens — recall, precision, and intersection-over-union (IoU) on tokens rather than documents. Think of chunks as bounding boxes and relevant spans as ground truth. The full evaluation code is available on GitHub, so you can regenerate the same kind of dataset for your own domain.
The reframing matters because RAG systems do not care much about ranking order. An LLM answering from twenty retrieved chunks is remarkably insensitive to where the relevant text sits in the prompt — what matters is whether the relevant tokens arrived at all. A strategy that returns a chunk 90% filled with irrelevant padding can rank well on document-level metrics while starving the model of useful context.
The Uncomfortable Baseline: Fixed-Size Splitting Holds Up
The Chroma evaluation’s most quoted conclusion is deflating for anyone who assumed semantic chunking was strictly better: heuristic strategies like the ubiquitous RecursiveCharacterTextSplitter often perform well when parameterized appropriately, and default settings for some popular strategies perform surprisingly poorly. Semantic chunking, which embeds sliding windows of sentences and cuts where similarity drops, is sensitive to its threshold — set it wrong and you get shattering over-fragmentation, where no chunk is long enough to carry a self-contained idea. That failure mode is worse than it sounds because the fragments still consume a vector, a row, and a slot in the prompt.
This does not mean semantic chunking is useless. It means the evaluation order matters: establish the fixed-size baseline first, with token-accurate counting and sensible overlap, and only reach for fancier splitters when your own metrics show a gap. One practical note from the same report — chunk size around 200–400 tokens with no overlap was competitive across their domains, and larger retrieval windows (top-10 rather than top-5) reliably raised recall. The Pinecone chunking guide covers the semantic variant’s mechanics in detail if you do want to implement it, but treat it as an experiment to be measured, not a default.
Late Chunking: Move the Split After the Transformer
The first technique that genuinely improves on naive splitting does not change the boundaries at all. Late chunking, published by Jina AI, flips the standard pipeline’s order. Normally you chunk first, then embed each chunk independently — which is why a chunk saying “The company’s revenue grew by 3%” cannot be retrieved for a query about a specific company. With late chunking, you run the entire document through a long-context embedding model first, then pool the resulting token embeddings into chunk-sized vectors afterward. Every chunk’s embedding now carries information from the whole document, because every token attended to every other token before the pooling step happened.
The reported gains grow with document length, which tracks the intuition: longer documents have more cross-references to lose. The technique needs an embedding model with a genuinely long context window — several thousand tokens minimum — and a pooling step you control, which rules out closed embedding APIs that only accept pre-cut text. The reference implementation lives in the late-chunking repository. If your corpus is long contracts, filings, or manuals full of anaphoric references (“its obligations”, “the aforementioned terms”), this is the technique most likely to pay for itself.
Contextual Retrieval: Fix the Chunks, Not the Splitter
The second technique keeps chunking exactly as it is and repairs each chunk afterward. Contextual Retrieval prepends a 50–100 token LLM-generated summary — situating the chunk within its document — before embedding and before building the BM25 index. The pipeline numbers are worth knowing precisely: on their evaluation, contextual embeddings alone cut the top-20 retrieval failure rate by 35% (5.7% to 3.7%), adding contextual BM25 cut failures by 49% (to 2.9%), and adding a reranker on top cut failures by 67% (to 1.9%). The components stack because they fix different failures: the context restores referents, BM25 catches exact identifiers like error codes, and the reranker re-scores the merged candidate list.
The cost objection usually collapses once prompt caching enters the picture. Generating context per chunk requires passing the whole document with each chunk, but with prompt caching the document is loaded once and only the chunk varies between calls; the published one-time cost was $1.02 per million document tokens. Their cookbook has the full implementation. One more boundary condition from the same write-up: if your entire knowledge base fits in roughly 200,000 tokens, skip chunking entirely and put the document in the prompt with caching — retrieval only earns its complexity past that scale.
A Minimal Harness You Can Run This Week
You do not need a benchmark lab to make this decision. You need a few dozen queries with annotated relevant spans and a token-level scorer. Here is a compact, self-contained version of the Chroma-style evaluation — character offsets stand in for tokens, which keeps the code dependency-free:
from dataclasses import dataclass
@dataclass
class Chunk:
text: str
start: int # character offset in the source document
end: int
def fixed_chunks(doc: str, size: int = 800, overlap: int = 100) -> list[Chunk]:
step = max(size - overlap, 1)
chunks = []
for start in range(0, len(doc), step):
chunks.append(Chunk(doc[start:start + size], start, min(start + size, len(doc))))
if start + size >= len(doc):
break
return chunks
def span_tokens(spans):
tokens = set()
for start, end in spans:
tokens.update(range(start, end))
return tokens
def token_metrics(gold_spans, retrieved_spans):
gold = span_tokens(gold_spans)
got = span_tokens(retrieved_spans)
if not gold:
return {"recall": 0.0, "precision": 0.0, "iou": 0.0}
recall = len(gold & got) / len(gold)
precision = len(gold & got) / len(got) if got else 0.0
iou = len(gold & got) / len(gold | got)
return {"recall": round(recall, 3),
"precision": round(precision, 3),
"iou": round(iou, 3)}To rank strategies, swap the chunker and keep everything else fixed. A lexical scorer stands in for your real retriever here — in production you would plug in your embedding model:
def rank_by_similarity(query: str, chunks: list[Chunk], k: int = 5) -> list[Chunk]:
q_terms = set(query.lower().split())
scored = []
for c in chunks:
terms = set(c.text.lower().split())
scored.append((len(q_terms & terms), c))
scored.sort(key=lambda pair: pair[0], reverse=True)
return [c for _, c in scored[:k]]
gold_spans = [(4100, 4620)] # annotated relevant excerpt from your doc
query = "What was ACME's Q2 revenue growth?"
for name, chunks in [
("fixed-800/100", fixed_chunks(document, 800, 100)),
("fixed-400/0", fixed_chunks(document, 400, 0)),
]:
top = rank_by_similarity(query, chunks, k=10)
spans = [(c.start, c.end) for c in top]
print(name, token_metrics(gold_spans, spans))Run this over 30–50 real queries and the winner for your corpus will usually be obvious — and it is frequently the boring option. If you adopt Contextual Retrieval, the context-generation step slots in before embedding and needs nothing but a prompt and a cacheable document:
CONTEXT_PROMPT = """<document>
{document}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the
overall document for the purposes of improving search retrieval of the
chunk. Answer only with the succinct context and nothing else."""
def contextualize(chunk: str, document: str, complete) -> str:
"""complete(prompt) calls your LLM of choice. Cache the document
portion so repeated calls only pay for the chunk, not the doc."""
prompt = CONTEXT_PROMPT.format(document=document, chunk=chunk)
context = complete(prompt)
return context + " " + chunkWhat I Would Actually Do
- Start with recursive or fixed-size splitting at 200–400 tokens, token-accurate counting, and retrieve top-10 rather than top-5.
- Build the token-level harness above and score the baseline on 30+ real queries before changing anything.
- If your documents are long and cross-referential, try late chunking with a long-context embedding model — it is the cheapest structural upgrade once you control the pooling step.
- If retrieval misses hinge on lost referents (“which company?”, “which quarter?”), add Contextual Retrieval with prompt caching, plus BM25 and a reranker — that combination is where the 67% failure reduction came from.
- If the whole corpus fits in about 200K tokens, delete the chunker and use prompt caching instead.
Chunking is the cheapest lever in a RAG system precisely because it sits upstream of everything: the embeddings, the index, and the prompt all inherit its mistakes. The benchmark literature of the last two years points the same direction — measure at the token level, trust the boring baseline until proven otherwise, and spend your complexity budget on giving chunks context rather than on cleverer scissors.