Text chunking length selection for embedding

This experiment begins with resume data that has already been converted from inconsistent text into structured JSON. The resulting fields are split into chunks of 25, 50, 100 and 200 units, embedded with a Snowflake Arctic embedding model, and indexed. A job description for an information-technology specialist is embedded and used to retrieve the nearest resume records.

For this dataset, smaller source chunks perform better: the 50-unit setting returns the strongest count of relevant IT resumes, while 100 and 200 decline. The video then changes a second variable by chunking the job-description query itself. Retrieval quality drops sharply. The observed lesson is specific but useful: keep a coherent short query intact while testing smaller chunks on long stored documents.

The result is not a universal recommendation that every system should use 25 or 50. It is a demonstration of how to find a working value: hold the model and query constant, create several indexes, measure relevance, then alter one assumption at a time. Overlap showed little benefit in this run, but that too should remain an empirical choice rather than a copied default.