skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
latestaiagents/agent-skills71 installs

rag-evaluation

Test and benchmark RAG systems with proper metrics and evaluation frameworks. Use this skill when evaluating RAG quality, measuring retrieval performance, using RAGAS metrics, or building RAG test suites. Activate when: RAG evaluation, RAG testing, RAGAS, retrieval metrics, faithfulness, relevance, context precision, RAG benchmarking.

How do I install this agent skill?

npx skills add https://github.com/latestaiagents/agent-skills --skill rag-evaluation
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides a framework for evaluating RAG systems and is generally safe. It identifies a standard surface for indirect prompt injection as it processes external content (questions and RAG outputs) through LLM judges without explicit sanitization or boundary markers.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

  • Runlayerpass

    1/1 file flagged

  • ZeroLeakspass

    Score: 93/100 · 2 sections analyzed

What does this agent skill do?

RAG Evaluation & Testing

Measure what matters: retrieval quality, answer faithfulness, and end-to-end performance. Evaluate retrieval and generation separately so you know which one to fix.

When to Use

  • Setting up an evaluation pipeline for a new RAG system
  • Comparing chunking, retrieval, or reranking strategies
  • Building regression tests that run on every change
  • Debugging poor answers (retrieval failure or generation failure?)

Key Metrics Overview

LayerMetricWhat It MeasuresNeeds
RetrievalHit rate@kAny relevant doc in the top k?Relevant doc IDs
RetrievalRecall@kShare of relevant docs found in the top kRelevant doc IDs
RetrievalPrecision@kShare of the top k that is relevantRelevant doc IDs
RetrievalMRRHow high the first relevant doc ranksRelevant doc IDs
RetrievalNDCG@kRanking quality, rewarding relevant docs near the topRelevant doc IDs
RetrievalContext precision / recall (RAGAS)Relevance and completeness of retrieved text, judged by an LLMReference answer
GenerationFaithfulnessIs the answer grounded in the retrieved context?Context
GenerationAnswer relevancyDoes the answer address the question?Question
End-to-endAnswer correctnessDoes the answer match the reference?Reference answer
End-to-endLatency, costOperational fitnessTraces

There are no universal "good" scores. Record a baseline on your own dataset, then treat drops from that baseline as regressions.

Building the Test Set

  • Golden dataset: human-validated questions with reference answers and the IDs of the documents that contain them. Start with a few dozen and grow it from real traffic.
  • Edge cases: ambiguous queries, questions with no answer in the corpus (the right output is "I don't know"), out-of-domain questions, and queries with exact identifiers.
  • Segments: tag each case by query type so you can compare metrics per segment.
  • Keep it versioned: a metric change is only meaningful if the dataset did not change underneath it.

Retrieval Metrics (ID-Based)

When you know which document IDs are relevant for each query, compute retrieval metrics directly. They are cheap, deterministic, and need no LLM judge.

import numpy as np

def recall_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
    scores = [len(set(res[:k]) & set(rel)) / len(rel) for res, rel in zip(results, relevant) if rel]
    return float(np.mean(scores))

def precision_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
    return float(np.mean([len(set(res[:k]) & set(rel)) / k for res, rel in zip(results, relevant)]))

def hit_rate(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
    return float(np.mean([bool(set(res[:k]) & set(rel)) for res, rel in zip(results, relevant)]))

def mrr(results: list[list[str]], relevant: list[list[str]]) -> float:
    scores = []
    for res, rel in zip(results, relevant):
        rel_set = set(rel)
        scores.append(next((1 / rank for rank, d in enumerate(res, 1) if d in rel_set), 0.0))
    return float(np.mean(scores))

def ndcg_at_k(results: list[list[str]], relevant: list[list[str]], k: int) -> float:
    """Binary relevance. The ideal ranking puts all relevant docs first."""
    def dcg(gains):
        return sum(g / np.log2(i + 2) for i, g in enumerate(gains))
    scores = []
    for res, rel in zip(results, relevant):
        rel_set = set(rel)
        gains = [1 if d in rel_set else 0 for d in res[:k]]
        ideal = dcg([1] * min(len(rel_set), k))
        scores.append(dcg(gains) / ideal if ideal > 0 else 0.0)
    return float(np.mean(scores))

def evaluate_retriever(retriever, dataset: list[dict], k_values=(1, 3, 5, 10)) -> dict:
    """dataset: [{"question": str, "relevant_ids": [str]}]; docs carry metadata["id"]."""
    results = [[d.metadata["id"] for d in retriever.invoke(item["question"])] for item in dataset]
    relevant = [item["relevant_ids"] for item in dataset]
    metrics = {"mrr": mrr(results, relevant)}
    for k in k_values:
        metrics[f"recall@{k}"] = recall_at_k(results, relevant, k)
        metrics[f"precision@{k}"] = precision_at_k(results, relevant, k)
        metrics[f"ndcg@{k}"] = ndcg_at_k(results, relevant, k)
        metrics[f"hit_rate@{k}"] = hit_rate(results, relevant, k)
    return metrics

Make the retriever return at least max(k_values) results, or the larger-k metrics are capped.

If you only have reference text rather than IDs, a substring check is a rough substitute:

def text_hit_rate(test_cases: list[dict], retriever, k: int = 5) -> float:
    """test_cases: [{"question": str, "relevant_docs": [snippet, ...]}]"""
    hits = 0
    for tc in test_cases:
        texts = [r.page_content for r in retriever.invoke(tc["question"])[:k]]
        if any(snippet in text for snippet in tc["relevant_docs"] for text in texts):
            hits += 1
    return hits / len(test_cases)

RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) provides LLM-judged metrics for RAG.

Model IDs: CHAT_MODEL in these examples is a placeholder. Set it to a current chat model ID from your provider's models page (for example CHAT_MODEL = os.environ["CHAT_MODEL"]); model IDs are retired over time.

RAGAS version note: the RAGAS API changes between releases. As documented at the time of writing (0.4.x), new projects should use the metric classes in ragas.metrics.collections with an LLM from llm_factory; the older ragas.metrics classes and lowercase metric objects are legacy. Check https://docs.ragas.io for the version you install.

Scoring Samples

# pip install ragas openai
import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision, ContextRecall, Faithfulness

judge = llm_factory(CHAT_MODEL, client=AsyncOpenAI())

faithfulness = Faithfulness(llm=judge)
context_precision = ContextPrecision(llm=judge)
context_recall = ContextRecall(llm=judge)

async def score_case(case: dict) -> dict:
    """case: {"question", "response", "contexts", "reference"}"""
    f = await faithfulness.ascore(
        user_input=case["question"],
        response=case["response"],
        retrieved_contexts=case["contexts"],
    )
    cp = await context_precision.ascore(
        user_input=case["question"],
        reference=case["reference"],
        retrieved_contexts=case["contexts"],
    )
    cr = await context_recall.ascore(
        user_input=case["question"],
        reference=case["reference"],
        retrieved_contexts=case["contexts"],
    )
    return {"faithfulness": f.value, "context_precision": cp.value, "context_recall": cr.value}

async def score_dataset(cases: list[dict]) -> list[dict]:
    return await asyncio.gather(*(score_case(c) for c in cases))

test_cases = [
    {
        "question": "What is the return policy?",
        "response": "Items can be returned within 30 days with receipt.",
        "contexts": [
            "Our return policy allows returns within 30 days of purchase. A receipt is required.",
            "Refunds are processed within 5-7 business days.",
        ],
        "reference": "30-day return policy with receipt required",
    },
    # Add more test cases...
]

scores = asyncio.run(score_dataset(test_cases))

Keep per-case scores, not just the averages: the low-scoring cases are where you find bugs.

Understanding Each Metric

Faithfulness (Is the answer grounded?)

Extracts the claims in the answer and checks each one against the retrieved context. Score = supported claims / total claims.

Example: the answer "The product costs $99 and ships free" against the context "Product price: $99. Standard shipping: $5" scores 0.5: the price claim is supported, the shipping claim is not.

Context Precision (Are the retrieved docs relevant?)

Checks whether each retrieved chunk is useful for answering, weighting by rank. High precision means little noise in the context; low precision means irrelevant chunks crowd the prompt.

Context Recall (Did retrieval get everything?)

Requires a reference answer. Checks whether the claims in the reference can be found in the retrieved context. Low recall means relevant documents were missed, so look at chunking and retrieval rather than the prompt.

Answer Relevancy (Does it answer the question?)

Measures how directly the answer addresses the question; evasive or padded answers score lower. RAGAS provides this as a response relevancy metric; check the metrics list in its docs for the current class name.

Custom LLM-as-Judge

When you need criteria RAGAS does not cover, write a rubric and use structured output:

from typing import Literal
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

class Judgment(BaseModel):
    score: Literal[1, 2, 3, 4, 5]
    reasoning: str

FAITHFULNESS_RUBRIC = """Rate whether the answer uses only information from the context.

Context: {context}
Answer: {answer}

1: makes up most of its information
2: significant unsupported claims
3: some unsupported claims
4: minor extrapolations
5: every claim is supported by the context"""

judge = ChatOpenAI(model=CHAT_MODEL, temperature=0).with_structured_output(Judgment)

def judge_faithfulness(context: str, answer: str) -> Judgment:
    return judge.invoke(FAITHFULNESS_RUBRIC.format(context=context, answer=answer))

LLM judges make mistakes and can disagree with each other. Before relying on one, score a sample by hand and check agreement; for important decisions, compare two judge models or prompts.

Latency Tracking

import time
from dataclasses import dataclass

@dataclass
class RAGMetrics:
    retrieval_latency_ms: float
    generation_latency_ms: float
    total_latency_ms: float
    num_docs_retrieved: int
    num_tokens_generated: int

def measure_rag_performance(rag_chain, query: str) -> RAGMetrics:
    """Measure RAG pipeline performance."""
    start = time.perf_counter()
    docs = rag_chain.retriever.invoke(query)
    retrieval_time = (time.perf_counter() - start) * 1000

    start = time.perf_counter()
    response = rag_chain.generate(query, docs)
    generation_time = (time.perf_counter() - start) * 1000

    return RAGMetrics(
        retrieval_latency_ms=retrieval_time,
        generation_latency_ms=generation_time,
        total_latency_ms=retrieval_time + generation_time,
        num_docs_retrieved=len(docs),
        num_tokens_generated=len(response.split())  # rough; use the provider's usage field
    )

Building a Test Suite

class RAGTestSuite:
    """Runs retrieval metrics, RAGAS scores, latency, and failure analysis."""

    def __init__(self, rag_chain, test_cases: list[dict]):
        """test_cases: [{"question", "reference", "relevant_ids"}]"""
        self.rag = rag_chain
        self.test_cases = test_cases

    def run_all(self) -> dict:
        return {
            "retrieval_metrics": evaluate_retriever(self.rag.retriever, self.test_cases),
            "ragas_scores": self._run_ragas(),
            "latency_metrics": self._run_latency_metrics(),
            "failure_analysis": self._analyze_failures(),
        }

    def _run_ragas(self) -> list[dict]:
        cases = []
        for tc in self.test_cases:
            contexts = self.rag.retriever.invoke(tc["question"])
            cases.append({
                "question": tc["question"],
                "response": self.rag.generate(tc["question"], contexts),
                "contexts": [c.page_content for c in contexts],
                "reference": tc["reference"],
            })
        return asyncio.run(score_dataset(cases))

    def _run_latency_metrics(self) -> dict:
        totals = [measure_rag_performance(self.rag, tc["question"]).total_latency_ms
                  for tc in self.test_cases]
        return {"p50_total_ms": float(np.percentile(totals, 50)),
                "p95_total_ms": float(np.percentile(totals, 95))}

    def _analyze_failures(self) -> list[dict]:
        """Separate retrieval failures from generation failures."""
        failures = []
        for tc in self.test_cases:
            docs = self.rag.retriever.invoke(tc["question"])
            retrieved_ids = {d.metadata["id"] for d in docs}
            answer = self.rag.generate(tc["question"], docs)
            if not answer_matches(answer, tc["reference"]):    # LLM judge or exact check
                failures.append({
                    "question": tc["question"],
                    "expected": tc["reference"],
                    "actual": answer,
                    # Right docs, wrong answer -> fix the prompt or model.
                    # Wrong docs -> fix chunking, retrieval, or the index.
                    "failure_type": "generation_failure"
                    if retrieved_ids & set(tc["relevant_ids"]) else "retrieval_failure",
                })
        return failures

Regression Gate in CI

Run the suite on every change to prompts, chunking, retrieval, or models, and fail the build when a metric drops below the stored baseline:

import json

def check_regression(current: dict, baseline_path: str, tolerance: float = 0.02) -> list[str]:
    """Return the metrics that dropped more than `tolerance` below the baseline."""
    with open(baseline_path) as f:
        baseline = json.load(f)
    return [
        f"{name}: {current[name]:.3f} < baseline {value:.3f}"
        for name, value in baseline.items()
        if name in current and current[name] < value - tolerance
    ]

regressions = check_regression(evaluate_retriever(retriever, dataset), "eval/baseline.json")
if regressions:
    raise SystemExit("RAG regression:\n" + "\n".join(regressions))

The tolerance is a hypothetical starting value; set it from the run-to-run variance you see on your dataset (LLM-judged metrics vary between runs).

Continuous Evaluation in Production

import logging
import random

logger = logging.getLogger("rag_eval")

def monitored_rag_query(rag, query: str, sample_rate: float = 0.05) -> dict:
    start = time.perf_counter()
    docs = rag.retriever.invoke(query)
    answer = rag.generate(query, docs)

    logger.info({
        "num_docs": len(docs),
        "latency_ms": (time.perf_counter() - start) * 1000,
        "answer_length": len(answer),
    })

    # Queue a sample of live traffic for offline faithfulness scoring
    if random.random() < sample_rate:
        enqueue_for_eval(query, answer, [d.page_content for d in docs])

    return {"answer": answer, "sources": docs}

Production queries have no reference answers, so score them with reference-free metrics (faithfulness, answer relevancy) and with user feedback. Check your privacy policy before storing user queries.

Best Practices

  1. Build a golden dataset - Human-validated Q&A pairs with relevant document IDs; grow it from real traffic
  2. Separate retrieval from generation - Measure each so you know which to fix
  3. Test edge cases - Empty results, ambiguous queries, no-answer and out-of-domain questions
  4. Automate in CI/CD - Run the eval on every change and gate on regressions
  5. Validate your judges - Check LLM-judge scores against human labels; judges can disagree
  6. Track over time - Monitor metric trends, and sample production traffic
  7. Segment by query type - Averages hide failures in one segment

References

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/rag-evaluation">View rag-evaluation on skillZs</a>