ASDRP · Agentic RAG Track · Module 02

Embeddings & Meaning

Turn text into numbers a computer can compare — then prove that semantically similar sentences actually land close together.

sentence → model → [0.12, −0.83, …] (384 numbers)

sentence-transformers all-MiniLM-L6-v2 keyless · free

Built by mui-group @ ASDRP· Advanced Student Directed Research Program

What we'll cover

Text → Vectors

Why numbers instead of words?
The encoding pipeline
A 384-dim vector up close

Embedding Space

What "space" means geometrically
Cosine similarity by hand
Similar sentences cluster

Hands-on & Cautions

Encode metals sentences live
PCA scatter: see the clusters
What embeddings can't do

Track 1 · Text → Vectors

Why turn words into numbers?

Computers can't compare strings by meaning.

"gold" and "bullion" share zero characters — but mean the same thing. A search that matches characters will miss it.

Vectors support geometry.

Once text is a point in space we can measure how close two points are — and "close" becomes "semantically similar."

RAG depends on this.

Module 1 asked "retrieve the relevant passages." Embeddings are how relevance is measured.

Track 1 · Text → Vectors

The model: `all-MiniLM-L6-v2`

What it is

A small transformer fine-tuned on 1B+ sentence pairs
Distilled from a much larger model — fast & accurate
Output: one 384-dimensional unit-norm vector per input

Why we use it here

Downloads once (~90 MB), then fully offline
No API key, no cost — perfect for learning
Modules 1–4 all use it; Module 5 swaps in a cloud model

from sentence_transformers import SentenceTransformer

# Downloads the model the first time (~90 MB), then cached
model = SentenceTransformer("all-MiniLM-L6-v2")

# Encode a single sentence → numpy array shape (384,)
vec = model.encode("Gold is a safe-haven asset.")
print(vec.shape)    # (384,)
print(vec[:5])      # [ 0.12, -0.83,  0.41, ...]

The model is downloaded from HuggingFace Hub on first call, then cached in ~/.cache/huggingface/. Subsequent runs are instant.

Track 1 · Text → Vectors

Encoding a batch of sentences

sentences = [
    "Gold is a safe-haven asset.",          # metals → safety
    "Silver is used in solar panels.",       # metals → industry
    "Platinum is rarer than gold.",          # metals → comparison
    "The weather is nice today.",            # off-topic
    "Investors buy gold during recessions.", # semantically close to sentence 0
]

# Encode all at once — returns a (5, 384) numpy matrix
embeddings = model.encode(sentences)
print(embeddings.shape)   # (5, 384)

# Each row is one sentence's vector
for sent, vec in zip(sentences, embeddings):
    print(f"{sent[:35]!r:38s}  norm={np.linalg.norm(vec):.4f}")

⚠ NOTE — Order matters for batches. The model encodes each sentence independently; the batch is just a convenience API that runs them in parallel on the GPU/CPU. Row i in the output always corresponds to sentence i in the input list.

Track 2 · Embedding Space

What is "embedding space"?

Every sentence becomes a point in a 384-dimensional space. We can't draw 384 axes, but the geometry is the same as 2D or 3D:

Distance between points → how different two sentences are
Direction matters more than magnitude (vectors are normalised)
Sentences about the same topic cluster in the same region

Dimensionality is high (384) to capture many independent facets of meaning simultaneously — topic, sentiment, tense, entity type, etc.

Track 2 · Embedding Space

Measuring similarity: cosine

cos(θ) = a · b / (‖a‖ · ‖b‖)

The dot product divided by the product of magnitudes. Because MiniLM outputs unit-norm vectors, the denominator is always 1 — cosine similarity reduces to a plain dot product.

Value	Interpretation
≈ 1.0	Nearly identical meaning
≈ 0.7	Related topic
≈ 0.3	Loosely related
≈ 0.0	Unrelated
< 0	Opposite sentiment (rare)

import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    """Cosine similarity between two 1-D vectors."""
    a, b = np.asarray(a, float), np.asarray(b, float)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

# Pair-wise examples
gold_vec   = model.encode("Gold is a safe-haven asset.")
invest_vec = model.encode("Investors buy gold in recessions.")
weather_vec= model.encode("The weather is nice today.")

print(cosine(gold_vec, invest_vec))   # e.g. 0.74 (related)
print(cosine(gold_vec, weather_vec))  # e.g. 0.13 (unrelated)

Track 3 · Hands-on

Similarity table for metals sentences

sentences = [
    "Gold is a safe-haven asset.",
    "Investors buy gold during recessions.",
    "Silver is used in solar panels.",
    "Platinum group metals are catalysts.",
    "The weather is nice today.",
]
embeddings = model.encode(sentences)

# Build a 5×5 similarity matrix
n = len(sentences)
sim = np.zeros((n, n))
for i in range(n):
    for j in range(n):
        sim[i, j] = cosine(embeddings[i], embeddings[j])

# Pretty-print
import pandas as pd
labels = [s[:28] + "…" for s in sentences]
df = pd.DataFrame(sim, index=labels, columns=labels).round(2)
print(df.to_string())

⚠ CAUTION — Embeddings encode statistical co-occurrence, not logic. "Gold prices always rise in recessions" and "Gold prices never rise in recessions" may score high similarity because the same words appear — the model doesn't understand negation the way humans do.

Track 3 · Hands-on

Visualising 384 dimensions with PCA

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Compress 384 dims → 2 for plotting
pca = PCA(n_components=2)
pts = pca.fit_transform(embeddings)

fig, ax = plt.subplots(figsize=(5, 4))
colors = ["#2563eb","#2563eb","#7c3aed",
          "#7c3aed","#dc2626"]
for i, (x, y) in enumerate(pts):
    ax.scatter(x, y, color=colors[i], s=80, zorder=3)
    ax.annotate(labels[i], (x, y),
                fontsize=7, textcoords="offset points",
                xytext=(5, 3))
ax.set_title("PCA of sentence embeddings (2D)")
ax.set_xlabel("PC 1"); ax.set_ylabel("PC 2")
plt.tight_layout(); plt.show()

What to expect

The two "gold" sentences should appear close together. Metals sentences cluster away from the weather sentence. PCA loses most variance — the real clusters are sharper in 384D.

PCA is lossy

Two points that look far apart in 2D might actually be close in the full 384-dim space. Always measure cosine similarity in the original space, not on the plot.

Track 3 · Cautions

Where embeddings mislead

⚠ CORRELATION, NOT TRUTH — An embedding encodes statistical patterns from training data, not factual truth. A sentence full of financial jargon scores high similarity to other financial text even if the claim is false.

⚠ MODEL-DEPENDENT — The same pair of sentences scores differently with a different model. Cosine similarity values are not comparable across models. Use one model consistently throughout a project.

⚠ DOMAIN SHIFT — all-MiniLM-L6-v2 was fine-tuned on general-web text. It works well for metals market prose, but specialist jargon (e.g. derivatives notation) may embed poorly compared to a domain-fine-tuned model.

Module 02 recap

Summary

Text → Vectors

SentenceTransformer.encode() maps any string to 384 floats
The model downloads once; all further work is local & free

Embedding Space

Sentences are points; similar meanings cluster together
Cosine similarity = dot product on unit-norm vectors

Cautions

Similarity ≠ truth; correlation ≠ logic
Scores are model-dependent — stay consistent

Run the notebook: 02_embeddings.ipynb · Next: Module 03 — Similarity & Vector Search (top-k retrieval with Qdrant)

Embeddings & Meaning

Contents

Text → Vectors

Embedding Space

Hands-on & Cautions

Why turn words into numbers?

The model: all-MiniLM-L6-v2

Encoding a batch of sentences

What is "embedding space"?

Measuring similarity: cosine

Similarity table for metals sentences

Visualising 384 dimensions with PCA

Where embeddings mislead

Summary

Text → Vectors

Embedding Space

Cautions

The model: `all-MiniLM-L6-v2`