rag›chunking.py
chunking.py
Hints
Let's say you hire an assistant who is brilliant, but has never seen your company's documents.
You ask them a question. Instead of guessing, they walk over to a filing cabinet, pull out the folders that look related, and answer using only what's in those folders.
That's RAG, Retrieval-Augmented Generation. Retrieve first, then generate.
But before any of that filing happens, someone has to put pages into folders in the first place. Imagine a person standing in front of a 10-page report with a paper shredder, told: "cut this into pieces of about 200 words each."
They don't read the report. They don't look for sentence endings or section headers. Every 200 words, cut. Even if that slices straight through a table. Even if it separates a header, "all figures in thousands of USD," from the number three paragraphs later that depends on it.
chunks = split_into_pieces(document, chunk_size=200) # a chunk never knows what came right before or after it
The assistant only ever sees one folder at a time. If that folder says "$450," and the folder explaining "these are in thousands" got shredded into a completely different pile, the assistant will confidently tell you the number is $450. Not $450,000.
The pipeline above makes chunking sound like a formality: cut the document into pieces, embed them, done. It isn't. Where you cut determines what survives, and it's easy to cut through the one sentence that made everything after it make sense.
So the retriever hands the model whatever's in the winning chunk, and a chunk is just a slice a text splitter cut, with no understanding of the document's actual structure. A vector database only ever sees those raw, isolated pieces.
When you slice text naively by characters, document-wide context gets stripped. If a table spans multiple pages, or if a header on Page 1 states "All figures in thousands of USD", that critical contextual marker is clipped away. The LLM only receives a chunk containing "$450" and hallucinates a raw net income of four hundred and fifty dollars, rather than four hundred and fifty thousand.
If you attempt to fix this boundary loss by over-indexing overlap sizes, you risk embedding dilution. Huge, generic chunks will pull ahead in similarity rankings over highly specific sentences, diluting search accuracy and inflating token costs.
Visualize this data fragmentation in real time inside our sandbox.
Slide the window boundaries. Choose a query and see if the assistant retrieves enough context to answer accurately.
Experiment with low chunk sizes until the warning displays to unlock Stage 3.