You embed chunks, not whole documents. Split text so each chunk is self-contained and retrievable — the decision that quietly makes or breaks retrieval quality.
Why: a whole document embeds to one blurry vector that matches everything weakly, and it may exceed the model’s input limit — small, focused chunks embed to sharp vectors that match precisely. When: chunk at ingestion, before embedding. Where: chunk size trades off precision (small) against context (large); most text lands well at a few hundred tokens.
WHOLE DOC -> one vector, averages every topic -> weak, fuzzy matches
CHUNKS -> one vector each, one topic each -> sharp, precise matches
Too small: a chunk loses the context needed to make sense.
Too large: a chunk mixes topics and dilutes the match.
Overlap a little so a sentence split across the boundary isn't lost.Why: splitting on natural boundaries (paragraphs, then sentences) keeps ideas intact, and a small overlap prevents losing meaning at the seams. When: use fixed-size-with-overlap as a solid default before reaching for anything fancier. Where: measure size in tokens, not characters, so chunks fit the embedding model’s limit.
def chunk(text, size=500, overlap=50):
"""Split into ~size-word chunks with an overlap so boundary sentences
appear in both neighbors. Swap word counts for a real tokenizer in prod."""
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
chunks = chunk(open("handbook.txt").read())
print(len(chunks), "chunks")Why: a retrieved chunk is useless if you cannot say where it came from — metadata lets you cite sources, filter by document or date, and debug bad retrievals. When: attach source, position, and any filters (tenant, section) at chunk time. Where: the vector DB stores this beside the vector so it travels with every result.
records = []
for i, text in enumerate(chunks):
records.append({
"id": f"handbook-{i}",
"content": text,
"metadata": {"source": "handbook.txt", "chunk": i},
})
# Now embed record["content"] and store the vector + content + metadata together.
# At query time you return content (to answer) and metadata (to cite).