ASDRP · Agentic RAG Track · Module 07

Retriever Metrics

Score your retriever with two complementary RAGAS metrics — context precision and context recall — and learn to read what each number is really telling you.

What we'll cover

Contents

Retriever evaluation

Why two metrics for one retriever?

Context Precision — quality & rank
Of all the chunks you returned, how many were actually useful — and were the useful ones near the top?

Context Recall — coverage
Did the retriever surface every chunk of evidence needed to answer this question fully?

⚠ KEY INSIGHT
These two metrics trade off against each other. Retrieving fewer chunks lifts precision but hurts recall. Retrieving many chunks improves recall but lets irrelevant noise in.

Intuition

The fishing analogy

LLMContextPrecisionWithReference

How RAGAS computes context precision

from ragas.metrics import LLMContextPrecisionWithReference

# RAGAS sends each (question, chunk, reference) triple to the judge LLM
# and asks: "Is this chunk useful for answering the question given this reference?"
# It weights the verdicts by position — top-ranked relevant chunks score higher.
metric = LLMContextPrecisionWithReference(llm=judge_llm)

LLMContextRecall

How RAGAS computes context recall

from ragas.metrics import LLMContextRecall

# RAGAS decomposes the reference answer into individual statements,
# then checks each statement: "Is there a retrieved chunk that supports this?"
# Recall = fraction of reference statements that are covered.
metric = LLMContextRecall(llm=judge_llm)

Comparison

Precision vs Recall — side by side

Step 0 · Setup

Environment setup: shared .env + RAGAS stub

import os
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())          # resolves to tutorials/.env
HAVE_KEYS = bool(os.environ.get("OLLAMA_API_KEY"))

# RAGAS 0.4.3 import stub — must come BEFORE importing ragas
import sys, types
_vx = types.ModuleType("langchain_community.chat_models.vertexai")
class ChatVertexAI: pass
_vx.ChatVertexAI = ChatVertexAI
sys.modules["langchain_community.chat_models.vertexai"] = _vx

import nest_asyncio; nest_asyncio.apply()   # RAGAS uses asyncio; notebooks already run a loop

Step 1 · RAGAS models

Wiring the judge LLM + embeddings

import litellm
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory

JUDGE_MODEL     = "ollama_chat/gemma4:31b-cloud"
EMBEDDING_MODEL = "ollama/qwen3-embedding:0.6b"

judge_llm       = llm_factory(JUDGE_MODEL, provider="litellm",
                               client=litellm.completion, temperature=0.0)
ragas_embeddings= embedding_factory("litellm", model=EMBEDDING_MODEL,
                                     api_base=os.environ["OLLAMA_API_BASE"])

Step 2 · Dataset

Building the EvaluationDataset from the golden set

import json
from ragas.dataset_schema import SingleTurnSample, EvaluationDataset

golden = json.load(open("golden_questions.json"))

samples = [
    SingleTurnSample(
        user_input=g["question"],
        retrieved_contexts=rag_result["retrieved_contexts"],
        response=rag_result["response"],
        reference=g["reference"],
    )
    for g in golden
    for rag_result in [rag_answer(g["question"])]
]

eval_dataset = EvaluationDataset(samples=samples)
print(f"Dataset: {len(samples)} samples")

Each SingleTurnSample bundles everything the metrics need: the question, what the retriever fetched, the generated answer, and the reference answer to grade against.

Step 3 · Evaluate

Running evaluate() over the full dataset

from ragas import evaluate
from ragas.metrics import LLMContextPrecisionWithReference, LLMContextRecall

retriever_metrics = [
    LLMContextPrecisionWithReference(),
    LLMContextRecall(),
]

results = evaluate(
    dataset=eval_dataset,
    metrics=retriever_metrics,
    llm=judge_llm,
    embeddings=ragas_embeddings,
)

df = results.to_pandas()
print(df[["user_input", "context_precision", "context_recall"]])

Step 4 · Single-sample debug

Debugging one sample with single_turn_ascore()

import asyncio

# Pick a sample that scored low on recall and inspect it
sample = samples[6]   # the multi-hop silver question

precision_score = asyncio.run(
    LLMContextPrecisionWithReference(llm=judge_llm).single_turn_ascore(sample)
)
recall_score = asyncio.run(
    LLMContextRecall(llm=judge_llm).single_turn_ascore(sample)
)

print(f"Question: {sample.user_input}")
print(f"Precision: {precision_score:.3f}   Recall: {recall_score:.3f}")
print(f"Retrieved {len(sample.retrieved_contexts)} chunks")

Watch out

Three things that make these numbers lie

Summary

What you built in Module 07

Run the notebook: 07_retriever_metrics.ipynb  ·  Next: Module 08 — More Retriever Metrics (ContextEntityRecall + NoiseSensitivity)