ASDRP · Agentic RAG Track · Module 09

Generator Metrics
+ LLM-as-Judge

Score the answer, not just the retrieval — and understand the judge behind every score.

What we'll cover

Contents

Motivation

Retriever metrics aren't enough

Modules 07–08 measured retrieval:
Did the right passages end up in the context window?

Module 09 measures generation:
Did the model turn those passages into a good answer?

A retriever can be perfect and the generator still hallucinate.
A generator can be fluent and still miss the point entirely.

⚠ Key insight
You need both sides. High retriever scores with low generator scores means your model is not using the context. High generator scores with low retriever scores is unlikely — garbage in, garbage out.

Generator Metric 1

Faithfulness — does every claim have a source?

score = grounded statements ÷ total statements

The judge LLM:

1. Decomposes the answer into atomic statements
2. Checks each statement against the retrieved contexts
3. Returns the fraction that are grounded

Score 1.0 — every claim supported by the context
Score 0.6 — 40% of claims are hallucinated

⚠ Caution
Faithfulness = 1.0 still allows wrong answers if the retrieved passages themselves contain errors. It only checks consistency with context, not truth.

Generator Metric 1 — Code

Running Faithfulness

from ragas import evaluate
from ragas.metrics import Faithfulness

results = evaluate(
    dataset=eval_dataset,
    metrics=[Faithfulness()],
    llm=judge_llm,          # temperature=0 for deterministic scores
    embeddings=ragas_embeddings,
)
df = results.to_pandas()
print(df[["user_input", "faithfulness"]])
# faithfulness: 0.0 – 1.0  (higher is better)

score = await Faithfulness(llm=judge_llm).single_turn_ascore(samples[0])
print(score)   # e.g. 0.88

nest_asyncio.apply() must be called first so await works inside a Jupyter event loop.

Generator Metric 2

Response Relevancy — is the answer on-topic?

score = mean cosine(back-questions, original question)

The judge generates n synthetic questions that the answer could be answering, then measures how similar they are to the original question using embeddings.

High score — answer stayed on-topic
Low score — answer drifted or addressed a different question

Note: a faithful answer can still score low here — it accurately cites the context but doesn't address what the user asked.

Generator Metric 3

Factual Correctness — does it match the reference?

F1 = 2 · precision · recall / (P + R)

Mode	Question asked
precision	Are the answer's claims correct?
recall	Were all reference points covered?
f1 (default)	Harmonic mean of both

⚠ Key difference
Faithfulness compares answer ↔ context.
Factual Correctness compares answer ↔ golden reference.
Only this metric catches errors the context already contained.

Generator Metric 3 — Code

Precision, Recall, and F1 modes

from ragas.metrics import FactualCorrectness

# Default: F1 (harmonic mean of precision and recall)
results_f1 = evaluate(
    dataset=eval_dataset,
    metrics=[FactualCorrectness()],
    llm=judge_llm, embeddings=ragas_embeddings,
)

# Precision only — are the answer's claims accurate?
fc_precision = FactualCorrectness(llm=judge_llm, mode="precision")

# Recall only — did the answer cover all reference points?
fc_recall = FactualCorrectness(llm=judge_llm, mode="recall")

scores_p = await fc_precision.single_turn_ascore(samples[0])
scores_r = await fc_recall.single_turn_ascore(samples[0])
print(f"Precision: {scores_p:.2f}  Recall: {scores_r:.2f}")

Track 3 — LLM-as-Judge

How judge scoring actually works

Every RAGAS generator metric sends a structured prompt to a judge LLM — a second model that reads the question, context, answer, and optional reference, then returns a score.

Why not rule-based matching?
NLP overlap metrics (ROUGE, BLEU) reward exact wording, not meaning. A judge LLM understands paraphrase, entailment, and implication.

Key setting: temperature=0
A deterministic judge gives reproducible scores across re-runs. Never use a high-temperature judge for evaluation.

Track 3 — LLM-as-Judge

Known biases — and how to guard against them

Putting it all together

Running all generator metrics at once

from ragas import evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy, FactualCorrectness

results = evaluate(
    dataset=eval_dataset,
    metrics=[
        Faithfulness(),
        ResponseRelevancy(),
        FactualCorrectness(),
    ],
    llm=judge_llm,
    embeddings=ragas_embeddings,
)

df = results.to_pandas()
print(df[["user_input","faithfulness","answer_relevancy","factual_correctness"]])

Interpretation

Reading your generator scores

Recap

Summary

Run the notebook: 09_generator_metrics.ipynb  ·  Next: Module 10 — Reranking (Cohere rerank-v3.5, before/after lift)