mui-group@asdrp ©

ASDRP · Agentic RAG Track · Module 04

Chunking & the Corpus

Slice 8 real metals-market documents into overlapping chunks, embed them locally, and run your first meaningful retrieval query.

Retrieval quality = f(chunk size, chunk overlap)
sentence-transformers langchain-text-splitters qdrant-client
Built by mui-group @ ASDRP· Advanced Student Directed Research Program

What we'll cover

Contents

1

Why chunk?

  • Context-window limits
  • Precision vs. recall trade-off
  • What a "chunk" actually is
2

Chunking strategy

  • RecursiveCharacterTextSplitter
  • chunk_size & chunk_overlap
  • The 8-file metals corpus
3

Build & query

  • Embed chunks locally
  • Store in Qdrant :memory:
  • Retrieve top-k results

Track 1 · Why chunk?

Long documents break retrieval

Embedding a full document collapses thousands of words into one vector — specific facts drown in the average.
Context-window limits mean you can only stuff so many tokens into an LLM prompt anyway.
Chunking gives each small idea its own vector, so "gold spot price" and "silver industrial demand" can be retrieved independently.
Full Document ~2 000 words · 1 embedding split Chunk 1 words 0–500 Chunk 2 words 440–940 Chunk 3 words 880–1380 Chunk 4 words 1320+ ← overlap (60 chars) → embed each v₁ [384] v₂ [384] v₃ [384] v₄ [384] Qdrant :memory: cosine index · instant lookup chunk_size=500

Track 2 · Chunking strategy

RecursiveCharacterTextSplitter

LangChain's splitter tries to break on natural boundaries — paragraphs → sentences → words → characters — only moving to a finer boundary when the chunk is still too big.

chunk_size = 500
Max characters per chunk. Keeps each embedding focused on one idea.
chunk_overlap = 60
Characters shared between consecutive chunks. Prevents a sentence straddling two chunks from vanishing from both.
⚠ CAUTION  Size mismatch kills retrieval. chunk_size=50 embeds sentence fragments with no context. chunk_size=2000 embeds whole pages — specific facts disappear. 500 / 60 is the capstone's tuned default. 
from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=60,
)

# Split ONE document string → list[str]
pieces = splitter.split_text(long_text)
print(f"{len(pieces)} chunks from 1 doc")

Separator priority: \n\n → \n → → ""

Track 2 · The corpus

8-file metals-market knowledge base

#FileTopic
01spot_price_and_quotesBid/ask, LBMA fix
02gold_fundamentalsGold supply & demand
03silver_and_industrialSilver photovoltaics
04platinum_palladiumPGMs & autocatalysts
05investment_vehiclesETFs, futures, coins
06drivers_and_macroUSD, rates, geopolitics
07risk_and_portfolioVaR, diversification
08futures_and_derivativesCOMEX contracts
~2 000 words each
Realistic but synthetic — no live API keys needed. Introduces the domain every later module evaluates.
Load with Path glob: 
from pathlib import Path
raw = [
  {"source": p.name,
   "page_content": p.read_text()}
  for p in sorted(
    Path("corpus").glob("*.md"))
]

Track 3 · Build & query

From raw .md to searchable index

from pathlib import Path
from sentence_transformers import SentenceTransformer
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

# 1. Load corpus
raw_docs = [{"source": p.name, "page_content": p.read_text()}
            for p in sorted(Path("corpus").glob("*.md"))]

# 2. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=60)
chunks = [(d["source"], piece)
          for d in raw_docs
          for piece in splitter.split_text(d["page_content"])]

# 3. Embed (local model, no API key)
model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dim
vectors = model.encode([c[1] for c in chunks])

# 4. Index in Qdrant :memory:
client = QdrantClient(":memory:")
client.create_collection("metals_kb",
    vectors_config=VectorParams(size=384, distance=Distance.COSINE))
client.upsert("metals_kb", points=[
    PointStruct(id=i, vector=v.tolist(), payload={"text": t, "source": s})
    for i, ((s, t), v) in enumerate(zip(chunks, vectors))])

Track 3 · Retrieval

Querying the chunk index

def retrieve(query: str, k: int = 5) -> list[str]:
    """Return top-k chunk texts for a query."""
    q_vec = model.encode(query).tolist()
    hits = client.search(
        "metals_kb", query_vector=q_vec, limit=k
    )
    return [h.payload["text"] for h in hits]

# Example
results = retrieve(
    "How is the LBMA gold price set?", k=5
)
for i, r in enumerate(results, 1):
    print(f"[{i}] {r[:120]}…")
Each hit is a chunk — a focused passage, not a whole document. Retrieval precision improves dramatically over whole-doc search.
k controls breadth. k=5 is tight and fast. Later modules use k=10 with reranking to recover recall without sacrificing context quality.
⚠ CAUTION  Overlap ≠ duplication. Neighboring chunks share 60 chars, so two consecutive results may echo each other. Module 10 addresses this with Cohere reranking.

Track 2 · Parameters

Choosing chunk_size and chunk_overlap

chunk_size chunk_overlap Effect Risk
50–10010–20 Very fine-grained — great precision Loses sentence context; embeddings become noisy
50060 Capstone default — balanced Still may split a table or formula mid-way
1 000–2 000100–200 Fewer chunks, richer context per hit One vector encodes many ideas; precision suffers
0 (= whole doc)0 Simplest — no splitting Embeds 2 000-word doc as one point; retrieval is poor
⚠ CAUTION There is no universal best chunk size. The right value depends on your embedding model's ideal input length, your documents' structure, and your retrieval task. Always run an eval (Module 7) before tuning.

Summary

What you built in Module 04

1

Why chunk?

  • Whole-doc embeddings dilute specific facts
  • LLM context windows impose a hard limit
  • Smaller chunks → sharper cosine similarity
2

Chunking strategy

  • RecursiveCharacterTextSplitter with 500 / 60
  • Splits on paragraphs → sentences → words
  • 8-file metals corpus loaded via Path glob
3

Build & query

  • Local embedder (all-MiniLM-L6-v2, 384-dim)
  • Qdrant :memory: cosine index
  • Top-k chunk retrieval — no API key needed

Next: Module 05 migrates the local embedder to cloud Ollama via LiteLLM and builds the first prompt-stuffing RAG answer. · Run the notebook: 04_chunking_corpus.ipynb