mui-group@asdrp ©

ASDRP · Agentic RAG Track · Module 08

More Retriever Metrics

Score whether retrieved passages capture key named entities — and whether irrelevant passages mislead the generator.

ContextEntityRecall + NoiseSensitivity = deeper retriever diagnosis
ragas OLLAMA_API_KEY retriever metrics inverted scale ⚠
Built by mui-group @ ASDRP· Advanced Student Directed Research Program

What we'll cover

Contents

1

Context Entity Recall

  • What named entities are
  • How the LLM judge extracts them
  • The recall formula
  • Run ContextEntityRecall
2

Noise Sensitivity

  • The inverted-scale trap ⚠
  • How irrelevant passages mislead
  • Run NoiseSensitivity
  • Single-sample debug pattern
3

Evaluate & Interpret

  • Full evaluate() run
  • Reading the results table
  • Connecting to Module 07
  • What to fix when scores are bad

Track 1 · Context Entity Recall

Did the retriever catch the key named entities?

Precision/Recall (Module 07) score whole passages. Entity Recall zooms in: does the retrieved context contain the specific named things the reference answer mentions?

  • LLM judge extracts named entities from the reference answer
  • Checks how many appear in the retrieved context
  • For metals: gold, silver, troy ounce, LBMA, contango, PGM…
EntityRecall = |entities in context| / |entities in reference|
Context Entity Recall diagram

Track 1 · Context Entity Recall

Running ContextEntityRecall

from ragas.metrics import ContextEntityRecall
from ragas import evaluate

# metrics list contains ONLY the two new metrics for this module
metrics = [ContextEntityRecall()]

results = evaluate(
    dataset=eval_dataset,   # EvaluationDataset with 8 golden samples
    metrics=metrics,
    llm=judge_llm,          # LLM judge extracts entities from reference
    embeddings=ragas_embeddings,
)
print(results.to_pandas()[["user_input", "context_entity_recall"]])
Single-sample debug — run one sample asynchronously to inspect exactly which entities were found: 
import asyncio
score = asyncio.run(
    ContextEntityRecall(llm=judge_llm).single_turn_ascore(samples[3])
)
print(f"Entity recall for Q4: {score:.3f}")

Track 2 · Noise Sensitivity

How often does irrelevant context mislead the generator?

Even a precise retriever can return a few irrelevant passages. NoiseSensitivity measures how often the generator makes incorrect claims that trace back to those noisy passages.

⚠ CAUTION — Inverted Scale
Lower NoiseSensitivity is better. A score of 0 means the generator was never misled. A score of 1 means every incorrect claim came from noisy context. Do NOT celebrate a high score here.
Noise Sensitivity diagram

Track 2 · Noise Sensitivity

What the metric actually measures

Step 1 — identify incorrect statements
The judge compares the generated answer to the reference and finds any claims that are wrong or unsupported.
Step 2 — trace each error to the context
For each incorrect claim, the judge checks: did this error come from an irrelevant (noisy) retrieved passage?
Step 3 — score
NoiseSensitivity = errors_from_noise / total_statements_in_answer
A retriever that returns only tight, relevant passages should keep this near 0.
Practical implication: if NoiseSensitivity is high, reduce k (fewer retrieved chunks) or add reranking (Module 10) to filter noise before the generator sees it.

Track 2 · Noise Sensitivity

Running NoiseSensitivity

from ragas.metrics import NoiseSensitivity

# NoiseSensitivity: LOWER is better — it measures how often the
# generator is misled by irrelevant retrieved passages.
metrics = [NoiseSensitivity()]

results = evaluate(
    dataset=eval_dataset,
    metrics=metrics,
    llm=judge_llm,
    embeddings=ragas_embeddings,
)
df = results.to_pandas()
print(df[["user_input", "noise_sensitivity_relevant"]])

# ⚠ A score of 0.18 is GOOD. A score of 0.80 is BAD.
⚠ REMINDER — The column is named noise_sensitivity_relevant. When you sort the table, sort ascending to put the best-performing questions at the top.

Track 3 · Evaluate & Interpret

Running both metrics together

from ragas.metrics import ContextEntityRecall, NoiseSensitivity
from ragas import evaluate

metrics = [
    ContextEntityRecall(),
    NoiseSensitivity(),      # LOWER is better
]

results = evaluate(
    dataset=eval_dataset,
    metrics=metrics,
    llm=judge_llm,
    embeddings=ragas_embeddings,
)
df = results.to_pandas()
print(df[["user_input",
          "context_entity_recall",
          "noise_sensitivity_relevant"]])

Track 3 · Evaluate & Interpret

How to read the results — the direction matters

Metric Good score Scale direction What to fix when bad
context_entity_recall High (→ 1.0) ↑ Higher is better Retriever missing entity-rich chunks; try larger k or better chunking
noise_sensitivity_relevant Low (→ 0) ↓ Lower is better ⚠ Retriever returning noisy passages; reduce k or add reranking (Module 10)
context_precision (M07) High (→ 1.0) ↑ Higher is better Too many irrelevant chunks returned
context_recall (M07) High (→ 1.0) ↑ Higher is better Key information missing from retrieval
NoiseSensitivity is the one metric in this track where DOWN is the direction you want. Keep a sticky note next to your dashboard until it's muscle memory.

Track 3 · Evaluate & Interpret

How these metrics complement Module 07

Module 07 — Precision & Recall
Did the retriever return the right passages? Were any key passages missed? Operates at the passage level.
Module 08 — Entity Recall & Noise
Did those passages contain the specific named entities the answer needs? Did irrelevant passages slip through and mislead the generator?

Together, all four metrics give a complete retriever health check:

  • Precision → are returned chunks relevant?
  • Recall → are needed chunks present?
  • Entity Recall → are the right facts present?
  • Noise Sensitivity → is the generator being misled?
Next step: Module 09 adds generator metrics — Faithfulness, ResponseRelevancy, FactualCorrectness — to complete the evaluation picture.

Wrap-up

Summary

1

Context Entity Recall

  • Measures named-entity coverage in retrieved context
  • LLM judge extracts entities from the reference answer
  • Higher → better entity coverage
  • Catches retrieval gaps invisible to passage-level metrics
2

Noise Sensitivity ⚠

  • LOWER is better — measures how often noise misleads the generator
  • High score = retriever returning too much irrelevant context
  • Fix: reduce k or add reranking (Module 10)
  • The one inverted-scale metric in the retriever suite
3

What's next

  • Module 09 adds generator metrics: Faithfulness, ResponseRelevancy, FactualCorrectness
  • Module 10 adds Cohere reranking to cut noise
  • Module 12 assembles all metrics into the full MDD loop

Run the notebook: 08_more_retriever_metrics.ipynb