rag-evaluation
Test and benchmark RAG systems with proper metrics and evaluation frameworks. Use this skill when evaluating RAG quality, measuring retrieval performance, using RAGAS metrics, or building RAG test suites. Activate when: RAG evaluation, RAG testing, RAGAS, retrieval metrics, faithfulness, relevance, context precision, RAG benchmarking.
How do I install this agent skill?
npx skills add https://github.com/latestaiagents/agent-skills --skill rag-evaluationIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a framework for evaluating RAG systems and is generally safe. It identifies a standard surface for indirect prompt injection as it processes external content (questions and RAG outputs) through LLM judges without explicit sanitization or boundary markers.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
- Runlayerpass
1/1 file flagged
- ZeroLeakspass
Score: 93/100 · 2 sections analyzed
What does this agent skill do?
RAG Evaluation & Testing
Measure what matters: retrieval quality, answer faithfulness, and end-to-end performance. Evaluate retrieval and generation separately so you know which one to fix.
When to Use
- Setting up an evaluation pipeline for a new RAG system
- Comparing chunking, retrieval, or reranking strategies
- Building regression tests that run on every change
- Debugging poor answers (retrieval failure or generation failure?)
Key Metrics Overview
| Layer | Metric | What It Measures | Needs |
|---|---|---|---|
| Retrieval | Hit rate@k | Any relevant doc in the top k? | Relevant doc IDs |
| Retrieval | Recall@k | Share of relevant docs found in the top k | Relevant doc IDs |
| Retrieval | Precision@k | Share of the top k that is relevant | Relevant doc IDs |
| Retrieval | MRR | How high the first relevant doc ranks | Relevant doc IDs |
| Retrieval | NDCG@k | Ranking quality, rewarding relevant docs near the top | Relevant doc IDs |
| Retrieval | Context precision / recall (RAGAS) | Relevance and completeness of retrieved text, judged by an LLM | Reference answer |
| Generation | Faithfulness | Is the answer grounded in the retrieved context? | Context |
| Generation | Answer relevancy | Does the answer address the question? | Question |
| End-to-end | Answer correctness | Does the answer match the reference? | Reference answer |
| End-to-end | Latency, cost | Operational fitness | Traces |
There are no universal "good" scores. Record a baseline on your own dataset, then treat drops from that baseline as regressions.
Building the Test Set
- Golden dataset: human-validated questions with reference answers and the IDs of the documents that contain them. Start with a few dozen and grow it from real traffic.
- Edge cases: ambiguous queries, questions with no answer in the corpus (the right output is "I don't know"), out-of-domain questions, and queries with exact identifiers.
- Segments: tag each case by query type so you can compare metrics per segment.
- Keep it versioned: a metric change is only meaningful if the dataset did not change underneath it.
Retrieval Metrics (ID-Based)
When you know which document IDs are relevant for each query, compute retrieval metrics directly. They are cheap, deterministic, and need no LLM judge.
import numpy as np
def recall_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
scores = [len(set(res[:k]) & set(rel)) / len(rel) for res, rel in zip(results, relevant) if rel]
return float(np.mean(scores))
def precision_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
return float(np.mean([len(set(res[:k]) & set(rel)) / k for res, rel in zip(results, relevant)]))
def hit_rate(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
return float(np.mean([bool(set(res[:k]) & set(rel)) for res, rel in zip(results, relevant)]))
def mrr(results: list[list[str]], relevant: list[list[str]]) -> float:
scores = []
for res, rel in zip(results, relevant):
rel_set = set(rel)
scores.append(next((1 / rank for rank, d in enumerate(res, 1) if d in rel_set), 0.0))
return float(np.mean(scores))
def ndcg_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
"""Binary relevance. The ideal ranking puts all relevant docs first."""
def dcg(gains):
return sum(g / np.log2(i + 2) for i, g in enumerate(gains))
scores = []
for res, rel in zip(results, relevant):
rel_set = set(rel)
gains = [1 if d in rel_set else 0 for d in res[:k]]
ideal = dcg([1] * min(len(rel_set), k))
scores.append(dcg(gains) / ideal if ideal > 0 else 0.0)
return float(np.mean(scores))
def evaluate_retriever(retriever, dataset: list[dict], k_values=(1, 3, 5, 10)) -> dict:
"""dataset: [{"question": str, "relevant_ids": [str]}]; docs carry metadata["id"]."""
results = [[d.metadata["id"] for d in retriever.invoke(item["question"])] for item in dataset]
relevant = [item["relevant_ids"] for item in dataset]
metrics = {"mrr": mrr(results, relevant)}
for k in k_values:
metrics[f"recall@{k}"] = recall_at_k(results, relevant, k)
metrics[f"precision@{k}"] = precision_at_k(results, relevant, k)
metrics[f"ndcg@{k}"] = ndcg_at_k(results, relevant, k)
metrics[f"hit_rate@{k}"] = hit_rate(results, relevant, k)
return metrics
Make the retriever return at least max(k_values) results, or the larger-k metrics are capped.
If you only have reference text rather than IDs, a substring check is a rough substitute:
def text_hit_rate(test_cases: list[dict], retriever, k: int = 5) -> float:
"""test_cases: [{"question": str, "relevant_docs": [snippet, ...]}]"""
hits = 0
for tc in test_cases:
texts = [r.page_content for r in retriever.invoke(tc["question"])[:k]]
if any(snippet in text for snippet in tc["relevant_docs"] for text in texts):
hits += 1
return hits / len(test_cases)
RAGAS Framework
RAGAS (Retrieval Augmented Generation Assessment) provides LLM-judged metrics for RAG.
Model IDs:
CHAT_MODELin these examples is a placeholder. Set it to a current chat model ID from your provider's models page (for exampleCHAT_MODEL = os.environ["CHAT_MODEL"]); model IDs are retired over time.
RAGAS version note: the RAGAS API changes between releases. As documented at the time of writing (0.4.x), new projects should use the metric classes in
ragas.metrics.collectionswith an LLM fromllm_factory; the olderragas.metricsclasses and lowercase metric objects are legacy. Check https://docs.ragas.io for the version you install.
Scoring Samples
# pip install ragas openai
import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision, ContextRecall, Faithfulness
judge = llm_factory(CHAT_MODEL, client=AsyncOpenAI())
faithfulness = Faithfulness(llm=judge)
context_precision = ContextPrecision(llm=judge)
context_recall = ContextRecall(llm=judge)
async def score_case(case: dict) -> dict:
"""case: {"question", "response", "contexts", "reference"}"""
f = await faithfulness.ascore(
user_input=case["question"],
response=case["response"],
retrieved_contexts=case["contexts"],
)
cp = await context_precision.ascore(
user_input=case["question"],
reference=case["reference"],
retrieved_contexts=case["contexts"],
)
cr = await context_recall.ascore(
user_input=case["question"],
reference=case["reference"],
retrieved_contexts=case["contexts"],
)
return {"faithfulness": f.value, "context_precision": cp.value, "context_recall": cr.value}
async def score_dataset(cases: list[dict]) -> list[dict]:
return await asyncio.gather(*(score_case(c) for c in cases))
test_cases = [
{
"question": "What is the return policy?",
"response": "Items can be returned within 30 days with receipt.",
"contexts": [
"Our return policy allows returns within 30 days of purchase. A receipt is required.",
"Refunds are processed within 5-7 business days.",
],
"reference": "30-day return policy with receipt required",
},
# Add more test cases...
]
scores = asyncio.run(score_dataset(test_cases))
Keep per-case scores, not just the averages: the low-scoring cases are where you find bugs.
Understanding Each Metric
Faithfulness (Is the answer grounded?)
Extracts the claims in the answer and checks each one against the retrieved context. Score = supported claims / total claims.
Example: the answer "The product costs $99 and ships free" against the context "Product price: $99. Standard shipping: $5" scores 0.5: the price claim is supported, the shipping claim is not.
Context Precision (Are the retrieved docs relevant?)
Checks whether each retrieved chunk is useful for answering, weighting by rank. High precision means little noise in the context; low precision means irrelevant chunks crowd the prompt.
Context Recall (Did retrieval get everything?)
Requires a reference answer. Checks whether the claims in the reference can be found in the retrieved context. Low recall means relevant documents were missed, so look at chunking and retrieval rather than the prompt.
Answer Relevancy (Does it answer the question?)
Measures how directly the answer addresses the question; evasive or padded answers score lower. RAGAS provides this as a response relevancy metric; check the metrics list in its docs for the current class name.
Custom LLM-as-Judge
When you need criteria RAGAS does not cover, write a rubric and use structured output:
from typing import Literal
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
class Judgment(BaseModel):
score: Literal[1, 2, 3, 4, 5]
reasoning: str
FAITHFULNESS_RUBRIC = """Rate whether the answer uses only information from the context.
Context: {context}
Answer: {answer}
1: makes up most of its information
2: significant unsupported claims
3: some unsupported claims
4: minor extrapolations
5: every claim is supported by the context"""
judge = ChatOpenAI(model=CHAT_MODEL, temperature=0).with_structured_output(Judgment)
def judge_faithfulness(context: str, answer: str) -> Judgment:
return judge.invoke(FAITHFULNESS_RUBRIC.format(context=context, answer=answer))
LLM judges make mistakes and can disagree with each other. Before relying on one, score a sample by hand and check agreement; for important decisions, compare two judge models or prompts.
Latency Tracking
import time
from dataclasses import dataclass
@dataclass
class RAGMetrics:
retrieval_latency_ms: float
generation_latency_ms: float
total_latency_ms: float
num_docs_retrieved: int
num_tokens_generated: int
def measure_rag_performance(rag_chain, query: str) -> RAGMetrics:
"""Measure RAG pipeline performance."""
start = time.perf_counter()
docs = rag_chain.retriever.invoke(query)
retrieval_time = (time.perf_counter() - start) * 1000
start = time.perf_counter()
response = rag_chain.generate(query, docs)
generation_time = (time.perf_counter() - start) * 1000
return RAGMetrics(
retrieval_latency_ms=retrieval_time,
generation_latency_ms=generation_time,
total_latency_ms=retrieval_time + generation_time,
num_docs_retrieved=len(docs),
num_tokens_generated=len(response.split()) # rough; use the provider's usage field
)
Building a Test Suite
class RAGTestSuite:
"""Runs retrieval metrics, RAGAS scores, latency, and failure analysis."""
def __init__(self, rag_chain, test_cases: list[dict]):
"""test_cases: [{"question", "reference", "relevant_ids"}]"""
self.rag = rag_chain
self.test_cases = test_cases
def run_all(self) -> dict:
return {
"retrieval_metrics": evaluate_retriever(self.rag.retriever, self.test_cases),
"ragas_scores": self._run_ragas(),
"latency_metrics": self._run_latency_metrics(),
"failure_analysis": self._analyze_failures(),
}
def _run_ragas(self) -> list[dict]:
cases = []
for tc in self.test_cases:
contexts = self.rag.retriever.invoke(tc["question"])
cases.append({
"question": tc["question"],
"response": self.rag.generate(tc["question"], contexts),
"contexts": [c.page_content for c in contexts],
"reference": tc["reference"],
})
return asyncio.run(score_dataset(cases))
def _run_latency_metrics(self) -> dict:
totals = [measure_rag_performance(self.rag, tc["question"]).total_latency_ms
for tc in self.test_cases]
return {"p50_total_ms": float(np.percentile(totals, 50)),
"p95_total_ms": float(np.percentile(totals, 95))}
def _analyze_failures(self) -> list[dict]:
"""Separate retrieval failures from generation failures."""
failures = []
for tc in self.test_cases:
docs = self.rag.retriever.invoke(tc["question"])
retrieved_ids = {d.metadata["id"] for d in docs}
answer = self.rag.generate(tc["question"], docs)
if not answer_matches(answer, tc["reference"]): # LLM judge or exact check
failures.append({
"question": tc["question"],
"expected": tc["reference"],
"actual": answer,
# Right docs, wrong answer -> fix the prompt or model.
# Wrong docs -> fix chunking, retrieval, or the index.
"failure_type": "generation_failure"
if retrieved_ids & set(tc["relevant_ids"]) else "retrieval_failure",
})
return failures
Regression Gate in CI
Run the suite on every change to prompts, chunking, retrieval, or models, and fail the build when a metric drops below the stored baseline:
import json
def check_regression(current: dict, baseline_path: str, tolerance: float = 0.02) -> list[str]:
"""Return the metrics that dropped more than `tolerance` below the baseline."""
with open(baseline_path) as f:
baseline = json.load(f)
return [
f"{name}: {current[name]:.3f} < baseline {value:.3f}"
for name, value in baseline.items()
if name in current and current[name] < value - tolerance
]
regressions = check_regression(evaluate_retriever(retriever, dataset), "eval/baseline.json")
if regressions:
raise SystemExit("RAG regression:\n" + "\n".join(regressions))
The tolerance is a hypothetical starting value; set it from the run-to-run variance you see on your dataset (LLM-judged metrics vary between runs).
Continuous Evaluation in Production
import logging
import random
logger = logging.getLogger("rag_eval")
def monitored_rag_query(rag, query: str, sample_rate: float = 0.05) -> dict:
start = time.perf_counter()
docs = rag.retriever.invoke(query)
answer = rag.generate(query, docs)
logger.info({
"num_docs": len(docs),
"latency_ms": (time.perf_counter() - start) * 1000,
"answer_length": len(answer),
})
# Queue a sample of live traffic for offline faithfulness scoring
if random.random() < sample_rate:
enqueue_for_eval(query, answer, [d.page_content for d in docs])
return {"answer": answer, "sources": docs}
Production queries have no reference answers, so score them with reference-free metrics (faithfulness, answer relevancy) and with user feedback. Check your privacy policy before storing user queries.
Best Practices
- Build a golden dataset - Human-validated Q&A pairs with relevant document IDs; grow it from real traffic
- Separate retrieval from generation - Measure each so you know which to fix
- Test edge cases - Empty results, ambiguous queries, no-answer and out-of-domain questions
- Automate in CI/CD - Run the eval on every change and gate on regressions
- Validate your judges - Check LLM-judge scores against human labels; judges can disagree
- Track over time - Monitor metric trends, and sample production traffic
- Segment by query type - Averages hide failures in one segment
References
- RAGAS documentation: https://docs.ragas.io/en/stable/
- RAGAS available metrics: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
- RAGAS paper (Es et al., 2023): https://arxiv.org/abs/2309.15217
- NDCG: Järvelin and Kekäläinen, "Cumulated gain-based evaluation of IR techniques", ACM TOIS, 2002: https://doi.org/10.1145/582415.582418
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/rag-evaluation">View rag-evaluation on skillZs</a>