skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
latestaiagents/agent-skills51 installs

corrective-rag

Implement Corrective RAG (CRAG) and Self-RAG for reliable, self-healing retrieval systems. Use this skill when building reliable RAG, preventing hallucinations, implementing retrieval evaluation, or adding self-correction to RAG pipelines. Activate when: corrective RAG, CRAG, self-RAG, hallucination prevention, retrieval evaluation, RAG reliability, self-healing RAG, document grading.

How do I install this agent skill?

npx skills add https://github.com/latestaiagents/agent-skills --skill corrective-rag
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides patterns for implementing Corrective RAG (CRAG) and Self-RAG using standard libraries like LangChain and LangGraph. It identifies a minor risk of indirect prompt injection inherent in systems that process external retrieved data, though it includes filtering steps to improve reliability.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

  • Runlayerwarn

    1/1 file flagged

What does this agent skill do?

Corrective RAG (CRAG) & Self-RAG

Build RAG systems that detect and correct retrieval failures automatically.

The Problem with Standard RAG

Standard RAG blindly trusts retrieved documents:

  • Irrelevant docs → hallucinated answers
  • Partially relevant docs → incomplete answers
  • Outdated docs → incorrect answers

Corrective RAG adds evaluation and correction loops.

CRAG Architecture

┌─────────────────────────────────────────────────────────────┐
│                      User Query                              │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Initial Retrieval                         │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│               Retrieval Evaluator (Grade Documents)          │
│                                                              │
│   For each document, classify as:                           │
│   • CORRECT - Relevant and supports answer                  │
│   • INCORRECT - Not relevant to query                       │
│   • AMBIGUOUS - Partially relevant, needs more context      │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
        ┌──────────┐   ┌──────────┐   ┌──────────┐
        │ CORRECT  │   │ AMBIGUOUS│   │ INCORRECT│
        │          │   │          │   │          │
        │ Use docs │   │ Docs +   │   │ Web      │
        │          │   │ Web      │   │ Search   │
        └──────────┘   └──────────┘   └──────────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                  Knowledge Refinement                        │
│         (Strip irrelevant parts, keep key info)             │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Generate Answer                           │
└─────────────────────────────────────────────────────────────┘

Pattern 1: Document Grading

Model IDs: CHAT_MODEL in these examples is a placeholder. Set it to a current chat model ID from your provider's models page (for example CHAT_MODEL = os.environ["CHAT_MODEL"]); model IDs are retired over time.

Use structured output so a grade can never be an unexpected string, and keep the grader's reasoning for logs and calibration.

import asyncio
from typing import Literal
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field

class RelevanceGrade(BaseModel):
    grade: Literal["correct", "incorrect", "ambiguous"]
    reasoning: str = Field(description="One-sentence justification")

GRADING_PROMPT = """You are a retrieval evaluator. Grade whether a document is relevant to a query.

Query: {query}

Document: {document}

- correct: the document directly answers or supports answering the query
- incorrect: the document is not relevant to the query
- ambiguous: the document is partially relevant or needs more context"""

grader = ChatPromptTemplate.from_template(GRADING_PROMPT) | ChatOpenAI(
    model=CHAT_MODEL, temperature=0
).with_structured_output(RelevanceGrade)

async def grade_documents(query: str, documents: list[str]) -> dict[str, list]:
    """Grade documents in parallel and bucket them."""
    grades = await asyncio.gather(
        *(grader.ainvoke({"query": query, "document": doc}) for doc in documents)
    )
    categorized = {"correct": [], "incorrect": [], "ambiguous": []}
    for doc, g in zip(documents, grades):
        categorized[g.grade].append(doc)
        log_grade(query, doc, g)  # keep for audits and grader calibration
    return categorized

Pattern 2: Full CRAG Pipeline

Follows the three actions in the CRAG paper: correct refines the retrieved documents, incorrect discards them for web search, and ambiguous combines both.

from langchain_tavily import TavilySearch  # pip install langchain-tavily

class CorrectiveRAG:
    def __init__(self, retriever, llm, web_search_tool=None, min_correct: int = 2):
        self.retriever = retriever
        self.llm = llm
        self.web_search = web_search_tool or TavilySearch(max_results=5)
        self.min_correct = min_correct  # tune on labeled queries

    async def query(self, question: str) -> dict:
        # Step 1: Initial retrieval
        docs = await self.retriever.ainvoke(question)
        documents = [d.page_content for d in docs]

        # Step 2: Grade documents
        graded = await grade_documents(question, documents)

        # Step 3: Decide action based on grades
        if len(graded["correct"]) >= self.min_correct:
            context_docs = graded["correct"]
            action = "correct"
        elif graded["correct"] or graded["ambiguous"]:
            # Partial evidence: keep it and add web results
            context_docs = graded["correct"] + graded["ambiguous"] + await self._web(question)
            action = "ambiguous"
        else:
            # Nothing usable: discard and search the web
            context_docs = await self._web(question)
            action = "incorrect"

        # Step 4: Refine knowledge (strip irrelevant parts)
        refined_context = await self._refine_knowledge(question, context_docs)

        # Step 5: Generate answer
        answer = await self._generate(question, refined_context)

        return {"answer": answer, "action": action, "docs_used": len(context_docs)}

    async def _web(self, question: str) -> list[str]:
        """Web fallback. TavilySearch returns a dict with a `results` list."""
        response = await self.web_search.ainvoke({"query": question})
        return [f"{r['content']}\n(Source: {r['url']})" for r in response.get("results", [])]

    async def _refine_knowledge(self, query: str, docs: list[str]) -> str:
        """Extract only the relevant parts of each document."""
        prompt = """Extract only the facts, numbers, and statements from this document
that help answer the question. If nothing is relevant, return NO_RELEVANT_INFO.

Question: {query}

Document:
{doc}"""
        parts = await asyncio.gather(
            *(self.llm.ainvoke(prompt.format(query=query, doc=d)) for d in docs)
        )
        return "\n\n".join(p.content for p in parts if "NO_RELEVANT_INFO" not in p.content)

    async def _generate(self, query: str, context: str) -> str:
        prompt = f"""
        Context: {context}

        Question: {query}

        Answer based only on the provided context and cite sources. If the context
        doesn't contain enough information, say so.
        """
        result = await self.llm.ainvoke(prompt)
        return result.content

Refining each document separately, rather than all at once, keeps one long document from crowding out the others and makes it easy to drop documents with nothing relevant.

Query Rewriting Instead of Web Search

If you cannot send queries to a web search provider (private data, regulated content, offline deployments), rewrite the query from the ambiguous documents and retrieve again from your own index:

async def refine_query(llm, query: str, ambiguous_docs: list[str]) -> str:
    prompt = f"""Original query: {query}

These documents were only partially relevant:
{ambiguous_docs[:2]}

Rewrite the query to be more specific. Return only the rewritten query."""
    return (await llm.ainvoke(prompt)).content

Web fallback sends the user's question to a third party; confirm that is allowed before enabling it.

Pattern 3: Self-RAG with Reflection Tokens

class SelfRAG:
    """
    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Uses special tokens to decide:
    - [Retrieve]: Should we retrieve? (yes/no)
    - [IsRel]: Is retrieved doc relevant? (yes/no)
    - [IsSup]: Does doc support the response? (fully/partially/no)
    - [IsUse]: Is the response useful? (1-5)
    """

    def __init__(self, retriever, llm):
        self.retriever = retriever
        self.llm = llm

    async def query(self, question: str) -> str:
        # Step 1: Decide if retrieval is needed
        needs_retrieval = await self._should_retrieve(question)

        if not needs_retrieval:
            # Generate without retrieval
            return await self._generate_direct(question)

        # Step 2: Retrieve and evaluate
        docs = await self.retriever.ainvoke(question)
        relevant_docs = []

        for doc in docs:
            # Check relevance
            is_relevant = await self._is_relevant(question, doc.page_content)
            if is_relevant:
                relevant_docs.append(doc)

        if not relevant_docs:
            # No relevant docs - generate without or try web
            return await self._generate_direct(question)

        # Step 3: Generate with relevant docs
        context = "\n".join([d.page_content for d in relevant_docs])
        response = await self._generate_with_context(question, context)

        # Step 4: Self-critique
        is_supported = await self._is_supported(response, context)
        usefulness = await self._rate_usefulness(question, response)

        if not is_supported or usefulness < 3:
            # Response not good enough - refine
            response = await self._refine_response(question, context, response)

        return response

    async def _should_retrieve(self, question: str) -> bool:
        """Decide if retrieval is needed for this question."""
        prompt = f"""
        Question: {question}

        Should external knowledge be retrieved to answer this question?
        Consider:
        - Is this factual or opinion-based?
        - Does it require specific/current information?
        - Can it be answered from general knowledge?

        Return: YES or NO
        """
        result = await self.llm.ainvoke(prompt)
        return "YES" in result.content.upper()

    async def _is_relevant(self, question: str, document: str) -> bool:
        """Check if document is relevant to question."""
        prompt = f"""
        Question: {question}
        Document: {document[:500]}

        Is this document relevant to answering the question?
        Return: YES or NO
        """
        result = await self.llm.ainvoke(prompt)
        return "YES" in result.content.upper()

    async def _is_supported(self, response: str, context: str) -> bool:
        """Check if response is supported by context."""
        prompt = f"""
        Context: {context}
        Response: {response}

        Is the response fully supported by the context?
        Return: FULLY, PARTIALLY, or NO
        """
        result = await self.llm.ainvoke(prompt)
        return "FULLY" in result.content.upper()

    async def _rate_usefulness(self, question: str, response: str) -> int:
        """Rate response usefulness 1-5."""
        prompt = f"""
        Question: {question}
        Response: {response}

        Rate how useful this response is (1-5):
        5 = Completely answers the question
        4 = Mostly answers with minor gaps
        3 = Partially answers
        2 = Barely relevant
        1 = Not useful

        Return only the number.
        """
        result = await self.llm.ainvoke(prompt)
        try:
            return int(result.content.strip())
        except ValueError:
            return 3  # unparseable: treat as "partially useful" and log it

Pattern 4: LangGraph CRAG Implementation

from langgraph.graph import StateGraph, END
from typing import TypedDict, Literal

class CRAGState(TypedDict):
    question: str
    documents: list
    grades: dict
    web_results: list
    generation: str

def retrieve(state: CRAGState) -> CRAGState:
    """Retrieve documents."""
    docs = retriever.invoke(state["question"])
    return {"documents": docs}

def grade_documents_node(state: CRAGState) -> CRAGState:
    """Grade retrieved documents."""
    grades = {"correct": [], "incorrect": [], "ambiguous": []}
    for doc in state["documents"]:
        grade = grader.invoke({"query": state["question"], "document": doc.page_content}).grade
        grades[grade].append(doc)
    return {"grades": grades}

def decide_action(state: CRAGState) -> Literal["generate", "web_search"]:
    """Route based on grades."""
    if len(state["grades"]["correct"]) >= 2:
        return "generate"
    return "web_search"

def web_search(state: CRAGState) -> CRAGState:
    """Fallback to web search (langchain_tavily.TavilySearch)."""
    response = tavily.invoke({"query": state["question"]})
    return {"web_results": response.get("results", [])}

def generate(state: CRAGState) -> CRAGState:
    """Generate answer from context."""
    if state.get("web_results"):
        context = "\n".join([r["content"] for r in state["web_results"]])
    else:
        context = "\n".join([d.page_content for d in state["grades"]["correct"]])

    response = llm.invoke(f"Context: {context}\n\nQuestion: {state['question']}")
    return {"generation": response.content}

# Build graph
workflow = StateGraph(CRAGState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade", grade_documents_node)
workflow.add_node("web_search", web_search)
workflow.add_node("generate", generate)

workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges("grade", decide_action)
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)

app = workflow.compile()

Evaluating CRAG

Track how often each action fires and whether answers stay grounded. Set targets from a baseline on your own labeled queries; there are no universal thresholds.

MetricWhat It Tells You
Share of queries graded correctHealth of the knowledge base and retriever
Web fallback rateCoverage gaps; a rising rate means missing or stale documents
Answer support rateGrounding: share of answers whose claims are supported by the context
Grader agreement with human labelsWhether the grader itself can be trusted
def evaluate_crag(test_cases: list[dict], crag: CorrectiveRAG) -> dict:
    """test_cases: [{"question": ..., "expected_answer": ...}]"""
    actions = {"correct": 0, "ambiguous": 0, "incorrect": 0}
    grounded = 0
    for case in test_cases:
        result = asyncio.run(crag.query(case["question"]))
        actions[result["action"]] += 1
        grounded += check_grounding(result["answer"], case)  # 1 if supported, else 0
    n = len(test_cases)
    return {
        **{f"{a}_rate": c / n for a, c in actions.items()},
        "fallback_rate": (actions["ambiguous"] + actions["incorrect"]) / n,
        "support_rate": grounded / n,
    }

check_grounding can be an LLM judge or a faithfulness metric; see rag-evaluation.

Best Practices

  1. Grade in parallel - Don't block on sequential grading
  2. Cache grades - Same doc + query = same grade
  3. Calibrate the grader - Compare its grades with human labels before trusting it; the grader can be wrong too
  4. Tune thresholds - Start strict, loosen if too many fallbacks
  5. Log everything - Track which documents fail, the grader's reasoning, and when fallback triggers
  6. Rate-limit and cache web search - Prevent cost overruns; identical queries shouldn't search twice
  7. A/B test - Compare CRAG vs standard RAG quality

When to Use

  • High-stakes applications - Medical, legal, financial
  • Low-quality knowledge bases - Outdated or noisy docs
  • User-facing products - Where hallucinations damage trust
  • Compliance requirements - Need audit trail of sources

References

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/corrective-rag">View corrective-rag on skillZs</a>