The setting nobody revisits
Imagine packing for a trip. You have a big pile of clothes, documents, and electronics, and you need to decide how to organize it all in your suitcase. Do you throw everything into one giant compartment, or do you use smaller pouches and packing cubes? The right answer depends on what you'll need to find later. Your passport goes in a small, easy-to-reach pouch — not buried at the bottom of the main bag. But your sweaters get packed together in one big cube, because you'll grab the whole stack at once.
Building a system that lets an AI answer questions by searching your documents — a setup called Retrieval-Augmented Generation, or RAG — involves the same kind of packing decision. Before an AI can search through a pile of documents, those documents have to be cut up into smaller pieces, called "chunks." Think of chunking as deciding how big to make each pouch or cube. Many developers pick one chunk size and use it for everything, the way you might decide to just use medium-sized pouches for your whole suitcase. This is a mistake.
New research shows that the best chunk size actually depends on the question being asked. If someone wants a specific fact — like grabbing a passport — small chunks work best, because the exact answer isn't buried in surrounding clutter. If someone wants to understand a broad topic — like surveying your wardrobe options — bigger chunks work better, because they keep related ideas together instead of scattering them into disconnected fragments.
The lesson: chunk size isn't a setting you pick once and forget. It's a dial you should keep adjusting based on what you're building and who's asking the questions. Getting this right — rather than defaulting to a single one-size-fits-all chunk size — can be the difference between an AI assistant that actually helps and one that frustrates you, and it can improve how often the system finds the right information (a measure called "recall") by up to 9%.
How it works
For years, chunking has been treated as a rote preprocessing step, a piece of boilerplate code in a utils.py file that you write once and never touch again. You import a text splitter, pick a size like 1000 and an overlap of 200, and run your documents through it. But this "set it and forget it" approach is a primary cause of poor RAG performance. The choice of chunking strategy is not a preprocessing decision; it's a retrieval tuning parameter with a measurable impact on recall.
The Default That's Costing You Recall
The most common method is fixed-size chunking, often implemented with a recursive character splitter. This approach walks through a document and splits it every N characters or tokens, with a small overlap to preserve some context across the break. A widely recommended starting point is a chunk size between 400 and 512 tokens, with a 10-20% overlap [2, 4].
Here's a simplified look at how a library like LangChain might implement this:
from langchain.text_splitter import RecursiveCharacterTextSplitter
with open('my_long_document.txt') as f:
long_text = f.read()
# A common but potentially suboptimal default
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
length_function=len,
)
chunks = text_splitter.create_documents([long_text])The trade-off here is simplicity at the expense of effectiveness. This method is fast, computationally cheap, and easy to implement. However, it is completely unaware of the document's structure or meaning. It will happily slice a sentence in half, break a table mid-row, or separate a question from its answer if they fall on either side of a 512-token boundary. This semantic fragmentation is a disaster for retrieval. The embedding model receives a mangled piece of text, creating a vector that doesn't fully represent the original meaning. When a user asks a question, the query vector fails to find a strong match because the relevant context was shattered during indexing.
The Mismatch: Chunk Size vs. Question Shape
The core problem is that a single chunk size cannot serve all types of queries. A mid-2025 study from the Fraunhofer IAIS systematically analyzed this relationship and found a clear pattern: the optimal chunk size is a function of the information density required by the query [1].
For concise, fact-based queries, smaller chunks of 64 to 128 tokens perform best. Think of questions like:
"What was the company's revenue in Q2 2024?"
"When was the project's deadline?"
"What is the specific dosage for this medication?"
The answer to these questions is a single, dense piece of information. A small chunk containing just this fact (and minimal noise) has a much more precise and distinct vector representation. When a user asks a factual question, their query vector is also sharp and specific. In a vector space, this leads to a high-similarity match. If that same fact is buried in a larger 1024-token chunk full of unrelated context, its vector representation is diluted, making it harder for the retriever to find.
For broad, contextual queries, larger chunks of 512 to 1024 tokens are more effective [1]. These are questions that require synthesis and understanding of a wider narrative:
"Summarize the main risks outlined in the annual report."
"What was the market sentiment regarding the product launch?"
"Explain the relationship between characters A and B in the first three chapters."
Here, the answer isn't a single sentence but is spread across multiple paragraphs or sections. A small chunk would only capture a fragment of the necessary information, forcing the LLM to stitch together multiple, potentially disjointed retrieval results. A larger chunk preserves the narrative flow and the relationships between concepts, providing the LLM with the rich context it needs to generate a comprehensive answer.
The same Fraunhofer IAIS study also revealed that embedding models themselves have different sensitivities. A model named Stella, for example, benefited from the global context in larger chunks, while a model named Snowflake excelled at fine-grained matching with smaller chunks [1]. This shows a deep interplay between your data, your query patterns, and your choice of embedding model.
Beyond Fixed Size: Structural and Semantic Chunking
If a single fixed size is suboptimal, what are the alternatives? More advanced strategies try to align chunk boundaries with the document's inherent structure or meaning.
Structural Chunking: This method uses the document's layout as a guide.
Page-Level Chunking: For documents like PDFs, you can treat each page as a single chunk. This strategy performed exceptionally well in NVIDIA's 2024 benchmarks, achieving the highest accuracy (0.648) and lowest variance [2]. Its power comes from preserving the visual and logical grouping of information that the document's author created. The trade-off is obvious: it only works for paginated documents. It's useless for a long HTML page or a plain text file.
Paragraph/Section Chunking: A more universal approach is to split along paragraph breaks or section headers (e.g., Markdown
##). This is a significant improvement over fixed-size chunking because paragraphs are natural units of thought. It's much less likely to sever a key idea mid-stream.
Semantic Chunking: This is the most sophisticated approach. Instead of looking at character counts or paragraph breaks, it looks at the meaning of the text itself. One common technique involves embedding every sentence in the document and then grouping adjacent sentences that are semantically similar. A chunk break is created when the similarity between consecutive sentences drops below a certain threshold, indicating a topic shift.
This method is powerful. By aligning chunks with the flow of ideas, it creates highly coherent and contextually rich segments for the retriever. It can lead to recall improvements of up to 9% over simpler methods [2]. The trade-off is cost and complexity. You must perform an embedding operation on every single sentence in your document corpus just to decide on the chunk boundaries. For millions of documents, this preprocessing cost can be substantial.
How to Test, Not Guess
Given these trade-offs, how do you choose? You don't. You test. Arguing in a conference room about whether to use 256-token chunks or 1024-token chunks is a waste of time. The correct answer depends on your specific documents and, most importantly, the questions your users will ask.
Here is a simple, effective framework for making an evidence-based decision:
Create a Golden Set: Curate a representative set of 50-100 questions your users would ask. For each question, identify the exact source text from your documents that contains the correct answer. This is your evaluation benchmark.
Create Multiple Indexes: Ingest a sample of your documents (e.g., a few thousand) multiple times, creating a separate vector index for each chunking strategy you want to test.
index_fixed_small: Fixed-size chunks (e.g., 128 tokens)index_fixed_large: Fixed-size chunks (e.g., 1024 tokens)index_semantic: Semantically chunkedindex_structural: Paragraph-chunked
Run and Evaluate: For each index, run all the questions from your golden set through your retrieval system. For each question, retrieve the top K documents (e.g., K=5).
Measure Recall: Check if the known-good source text is present in the retrieved chunks. The percentage of questions for which the correct context was successfully retrieved is your recall score.
This process transforms the abstract debate about chunking into a concrete experiment. You might find that index_fixed_small gets 95% recall on factual questions but only 60% on summary questions, while index_fixed_large shows the opposite profile. This data allows you to make an informed decision, whether it's choosing one "best average" strategy or building a more advanced system that routes queries to different indexes based on question type.
What this means in practice
Shifting your perspective on chunking from a static setting to a dynamic parameter has immediate, practical consequences for how you build, maintain, and use RAG systems.
For the people building these systems, the chunk_size parameter should be elevated from a constant in a config file to a key hyperparameter — something you deliberately tune and test, the same way you would tune a model's learning rate. This also means your data ingestion pipeline becomes more complex. Instead of a single, linear process, you might need to generate multiple sets of chunks and store them in different vector indexes (the specialized databases that hold your chunked, searchable content). This increases storage costs in your vector database and adds to the initial processing time.
The payoff is a system that can handle a wider variety of user needs. A sophisticated RAG application might first classify an incoming query — is it asking for a specific fact or a broad summary? — and then route the search to the index best suited for that type of question. This "strategy routing" adds a layer of logic but can dramatically improve the quality and precision of retrieved results, leading to better final answers from the AI.
For the people using the end product — the customer service agent looking for a policy detail, the doctor reviewing patient history, or the financial analyst researching a company — the difference is night and day. A system with naive, one-size-fits-all chunking feels brittle. It answers some questions well but fails inexplicably on others. It gives vague summaries when you need a specific number, and throws disconnected facts at you when what you actually needed was a coherent explanation.
By contrast, a system with chunking tuned to the kinds of questions being asked feels robust and reliable. When you ask for the Q2 revenue, it returns the exact table from the quarterly report. When you ask about market risks, it provides the full, coherent paragraph from the filing. This isn't about making the underlying AI model "smarter"; it's about giving the search step a fair chance to find the right information in the first place. That search step is the foundation everything else depends on — get it wrong, and no amount of clever writing by the AI can fix a bad set of retrieved facts.
Where this is heading
The industry is just beginning to grapple with the complexities of chunking, and several key questions remain unresolved. The work from Fraunhofer IAIS highlights the need for better "chunk quality measures" [1] — quick ways to score whether a chunk is any good. Right now, the only way to know if a chunking strategy is good is to run a full end-to-end retrieval test, which is expensive and slow. The field needs cheaper shortcuts to judge whether a chunk is semantically complete and useful before it's ever loaded into a vector database.
Looking forward, my analysis is that the idea of a single, fixed chunking strategy will disappear from production systems. We are heading towards more dynamic and adaptive approaches.
Hybrid Strategies: Expect a rise in hybrid techniques that start with fast, structural chunking (like splitting by paragraphs) and then use more expensive, meaning-aware methods to merge or further split those chunks intelligently — combining the speed of simple methods with the quality of complex ones.
Per-Document Optimization: Instead of one chunking strategy applied uniformly across every document, future systems might examine each document and apply a tailored approach. A dense legal contract might be chunked very differently from a casual conversation transcript.
Query-Time Chunking: The ultimate goal is to delay the chunking decision until the last possible moment. Some experimental approaches, like "late chunking," retrieve larger document segments first and then carve them into pieces on the fly, shaped by the specifics of the user's actual question. This is computationally demanding but offers the highest possible relevance.
The key thing to watch is the co-evolution of chunking strategies and embedding models — the systems that convert text into the numerical representations used for search. As these models become more powerful and context-aware, their sensitivity to where chunks are cut may change [1]. We may see models designed specifically to tolerate clumsy chunking, or models that feed information back into the chunking process itself. For now, the takeaway is clear: stop treating chunking as an afterthought. It is one of the most critical levers you can pull to improve your RAG system's performance. Start testing.