corrective-rag
Implement Corrective RAG (CRAG) and Self-RAG for reliable, self-healing retrieval systems. Use this skill when building reliable RAG, preventing hallucinations, implementing retrieval evaluation, or adding self-correction to RAG pipelines. Activate when: corrective RAG, CRAG, self-RAG, hallucination prevention, retrieval evaluation, RAG reliability, self-healing RAG, document grading.
How do I install this agent skill?
npx skills add https://github.com/latestaiagents/agent-skills --skill corrective-ragIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides patterns for implementing Corrective RAG (CRAG) and Self-RAG using standard libraries like LangChain and LangGraph. It identifies a minor risk of indirect prompt injection inherent in systems that process external retrieved data, though it includes filtering steps to improve reliability.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
- Runlayerwarn
1/1 file flagged
What does this agent skill do?
Corrective RAG (CRAG) & Self-RAG
Build RAG systems that detect and correct retrieval failures automatically.
The Problem with Standard RAG
Standard RAG blindly trusts retrieved documents:
- Irrelevant docs → hallucinated answers
- Partially relevant docs → incomplete answers
- Outdated docs → incorrect answers
Corrective RAG adds evaluation and correction loops.
CRAG Architecture
┌─────────────────────────────────────────────────────────────┐
│ User Query │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Initial Retrieval │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Retrieval Evaluator (Grade Documents) │
│ │
│ For each document, classify as: │
│ • CORRECT - Relevant and supports answer │
│ • INCORRECT - Not relevant to query │
│ • AMBIGUOUS - Partially relevant, needs more context │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ CORRECT │ │ AMBIGUOUS│ │ INCORRECT│
│ │ │ │ │ │
│ Use docs │ │ Docs + │ │ Web │
│ │ │ Web │ │ Search │
└──────────┘ └──────────┘ └──────────┘
│ │ │
└───────────────┼───────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ Knowledge Refinement │
│ (Strip irrelevant parts, keep key info) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Generate Answer │
└─────────────────────────────────────────────────────────────┘
Pattern 1: Document Grading
Model IDs:
CHAT_MODELin these examples is a placeholder. Set it to a current chat model ID from your provider's models page (for exampleCHAT_MODEL = os.environ["CHAT_MODEL"]); model IDs are retired over time.
Use structured output so a grade can never be an unexpected string, and keep the grader's reasoning for logs and calibration.
import asyncio
from typing import Literal
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field
class RelevanceGrade(BaseModel):
grade: Literal["correct", "incorrect", "ambiguous"]
reasoning: str = Field(description="One-sentence justification")
GRADING_PROMPT = """You are a retrieval evaluator. Grade whether a document is relevant to a query.
Query: {query}
Document: {document}
- correct: the document directly answers or supports answering the query
- incorrect: the document is not relevant to the query
- ambiguous: the document is partially relevant or needs more context"""
grader = ChatPromptTemplate.from_template(GRADING_PROMPT) | ChatOpenAI(
model=CHAT_MODEL, temperature=0
).with_structured_output(RelevanceGrade)
async def grade_documents(query: str, documents: list[str]) -> dict[str, list]:
"""Grade documents in parallel and bucket them."""
grades = await asyncio.gather(
*(grader.ainvoke({"query": query, "document": doc}) for doc in documents)
)
categorized = {"correct": [], "incorrect": [], "ambiguous": []}
for doc, g in zip(documents, grades):
categorized[g.grade].append(doc)
log_grade(query, doc, g) # keep for audits and grader calibration
return categorized
Pattern 2: Full CRAG Pipeline
Follows the three actions in the CRAG paper: correct refines the retrieved documents, incorrect discards them for web search, and ambiguous combines both.
from langchain_tavily import TavilySearch # pip install langchain-tavily
class CorrectiveRAG:
def __init__(self, retriever, llm, web_search_tool=None, min_correct: int = 2):
self.retriever = retriever
self.llm = llm
self.web_search = web_search_tool or TavilySearch(max_results=5)
self.min_correct = min_correct # tune on labeled queries
async def query(self, question: str) -> dict:
# Step 1: Initial retrieval
docs = await self.retriever.ainvoke(question)
documents = [d.page_content for d in docs]
# Step 2: Grade documents
graded = await grade_documents(question, documents)
# Step 3: Decide action based on grades
if len(graded["correct"]) >= self.min_correct:
context_docs = graded["correct"]
action = "correct"
elif graded["correct"] or graded["ambiguous"]:
# Partial evidence: keep it and add web results
context_docs = graded["correct"] + graded["ambiguous"] + await self._web(question)
action = "ambiguous"
else:
# Nothing usable: discard and search the web
context_docs = await self._web(question)
action = "incorrect"
# Step 4: Refine knowledge (strip irrelevant parts)
refined_context = await self._refine_knowledge(question, context_docs)
# Step 5: Generate answer
answer = await self._generate(question, refined_context)
return {"answer": answer, "action": action, "docs_used": len(context_docs)}
async def _web(self, question: str) -> list[str]:
"""Web fallback. TavilySearch returns a dict with a `results` list."""
response = await self.web_search.ainvoke({"query": question})
return [f"{r['content']}\n(Source: {r['url']})" for r in response.get("results", [])]
async def _refine_knowledge(self, query: str, docs: list[str]) -> str:
"""Extract only the relevant parts of each document."""
prompt = """Extract only the facts, numbers, and statements from this document
that help answer the question. If nothing is relevant, return NO_RELEVANT_INFO.
Question: {query}
Document:
{doc}"""
parts = await asyncio.gather(
*(self.llm.ainvoke(prompt.format(query=query, doc=d)) for d in docs)
)
return "\n\n".join(p.content for p in parts if "NO_RELEVANT_INFO" not in p.content)
async def _generate(self, query: str, context: str) -> str:
prompt = f"""
Context: {context}
Question: {query}
Answer based only on the provided context and cite sources. If the context
doesn't contain enough information, say so.
"""
result = await self.llm.ainvoke(prompt)
return result.content
Refining each document separately, rather than all at once, keeps one long document from crowding out the others and makes it easy to drop documents with nothing relevant.
Query Rewriting Instead of Web Search
If you cannot send queries to a web search provider (private data, regulated content, offline deployments), rewrite the query from the ambiguous documents and retrieve again from your own index:
async def refine_query(llm, query: str, ambiguous_docs: list[str]) -> str:
prompt = f"""Original query: {query}
These documents were only partially relevant:
{ambiguous_docs[:2]}
Rewrite the query to be more specific. Return only the rewritten query."""
return (await llm.ainvoke(prompt)).content
Web fallback sends the user's question to a third party; confirm that is allowed before enabling it.
Pattern 3: Self-RAG with Reflection Tokens
class SelfRAG:
"""
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Uses special tokens to decide:
- [Retrieve]: Should we retrieve? (yes/no)
- [IsRel]: Is retrieved doc relevant? (yes/no)
- [IsSup]: Does doc support the response? (fully/partially/no)
- [IsUse]: Is the response useful? (1-5)
"""
def __init__(self, retriever, llm):
self.retriever = retriever
self.llm = llm
async def query(self, question: str) -> str:
# Step 1: Decide if retrieval is needed
needs_retrieval = await self._should_retrieve(question)
if not needs_retrieval:
# Generate without retrieval
return await self._generate_direct(question)
# Step 2: Retrieve and evaluate
docs = await self.retriever.ainvoke(question)
relevant_docs = []
for doc in docs:
# Check relevance
is_relevant = await self._is_relevant(question, doc.page_content)
if is_relevant:
relevant_docs.append(doc)
if not relevant_docs:
# No relevant docs - generate without or try web
return await self._generate_direct(question)
# Step 3: Generate with relevant docs
context = "\n".join([d.page_content for d in relevant_docs])
response = await self._generate_with_context(question, context)
# Step 4: Self-critique
is_supported = await self._is_supported(response, context)
usefulness = await self._rate_usefulness(question, response)
if not is_supported or usefulness < 3:
# Response not good enough - refine
response = await self._refine_response(question, context, response)
return response
async def _should_retrieve(self, question: str) -> bool:
"""Decide if retrieval is needed for this question."""
prompt = f"""
Question: {question}
Should external knowledge be retrieved to answer this question?
Consider:
- Is this factual or opinion-based?
- Does it require specific/current information?
- Can it be answered from general knowledge?
Return: YES or NO
"""
result = await self.llm.ainvoke(prompt)
return "YES" in result.content.upper()
async def _is_relevant(self, question: str, document: str) -> bool:
"""Check if document is relevant to question."""
prompt = f"""
Question: {question}
Document: {document[:500]}
Is this document relevant to answering the question?
Return: YES or NO
"""
result = await self.llm.ainvoke(prompt)
return "YES" in result.content.upper()
async def _is_supported(self, response: str, context: str) -> bool:
"""Check if response is supported by context."""
prompt = f"""
Context: {context}
Response: {response}
Is the response fully supported by the context?
Return: FULLY, PARTIALLY, or NO
"""
result = await self.llm.ainvoke(prompt)
return "FULLY" in result.content.upper()
async def _rate_usefulness(self, question: str, response: str) -> int:
"""Rate response usefulness 1-5."""
prompt = f"""
Question: {question}
Response: {response}
Rate how useful this response is (1-5):
5 = Completely answers the question
4 = Mostly answers with minor gaps
3 = Partially answers
2 = Barely relevant
1 = Not useful
Return only the number.
"""
result = await self.llm.ainvoke(prompt)
try:
return int(result.content.strip())
except ValueError:
return 3 # unparseable: treat as "partially useful" and log it
Pattern 4: LangGraph CRAG Implementation
from langgraph.graph import StateGraph, END
from typing import TypedDict, Literal
class CRAGState(TypedDict):
question: str
documents: list
grades: dict
web_results: list
generation: str
def retrieve(state: CRAGState) -> CRAGState:
"""Retrieve documents."""
docs = retriever.invoke(state["question"])
return {"documents": docs}
def grade_documents_node(state: CRAGState) -> CRAGState:
"""Grade retrieved documents."""
grades = {"correct": [], "incorrect": [], "ambiguous": []}
for doc in state["documents"]:
grade = grader.invoke({"query": state["question"], "document": doc.page_content}).grade
grades[grade].append(doc)
return {"grades": grades}
def decide_action(state: CRAGState) -> Literal["generate", "web_search"]:
"""Route based on grades."""
if len(state["grades"]["correct"]) >= 2:
return "generate"
return "web_search"
def web_search(state: CRAGState) -> CRAGState:
"""Fallback to web search (langchain_tavily.TavilySearch)."""
response = tavily.invoke({"query": state["question"]})
return {"web_results": response.get("results", [])}
def generate(state: CRAGState) -> CRAGState:
"""Generate answer from context."""
if state.get("web_results"):
context = "\n".join([r["content"] for r in state["web_results"]])
else:
context = "\n".join([d.page_content for d in state["grades"]["correct"]])
response = llm.invoke(f"Context: {context}\n\nQuestion: {state['question']}")
return {"generation": response.content}
# Build graph
workflow = StateGraph(CRAGState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade", grade_documents_node)
workflow.add_node("web_search", web_search)
workflow.add_node("generate", generate)
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges("grade", decide_action)
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)
app = workflow.compile()
Evaluating CRAG
Track how often each action fires and whether answers stay grounded. Set targets from a baseline on your own labeled queries; there are no universal thresholds.
| Metric | What It Tells You |
|---|---|
Share of queries graded correct | Health of the knowledge base and retriever |
| Web fallback rate | Coverage gaps; a rising rate means missing or stale documents |
| Answer support rate | Grounding: share of answers whose claims are supported by the context |
| Grader agreement with human labels | Whether the grader itself can be trusted |
def evaluate_crag(test_cases: list[dict], crag: CorrectiveRAG) -> dict:
"""test_cases: [{"question": ..., "expected_answer": ...}]"""
actions = {"correct": 0, "ambiguous": 0, "incorrect": 0}
grounded = 0
for case in test_cases:
result = asyncio.run(crag.query(case["question"]))
actions[result["action"]] += 1
grounded += check_grounding(result["answer"], case) # 1 if supported, else 0
n = len(test_cases)
return {
**{f"{a}_rate": c / n for a, c in actions.items()},
"fallback_rate": (actions["ambiguous"] + actions["incorrect"]) / n,
"support_rate": grounded / n,
}
check_grounding can be an LLM judge or a faithfulness metric; see rag-evaluation.
Best Practices
- Grade in parallel - Don't block on sequential grading
- Cache grades - Same doc + query = same grade
- Calibrate the grader - Compare its grades with human labels before trusting it; the grader can be wrong too
- Tune thresholds - Start strict, loosen if too many fallbacks
- Log everything - Track which documents fail, the grader's reasoning, and when fallback triggers
- Rate-limit and cache web search - Prevent cost overruns; identical queries shouldn't search twice
- A/B test - Compare CRAG vs standard RAG quality
When to Use
- High-stakes applications - Medical, legal, financial
- Low-quality knowledge bases - Outdated or noisy docs
- User-facing products - Where hallucinations damage trust
- Compliance requirements - Need audit trail of sources
References
- Corrective Retrieval Augmented Generation (Yan et al., 2024): https://arxiv.org/abs/2401.15884
- Self-RAG (Asai et al., 2023): https://arxiv.org/abs/2310.11511
- Tavily integration for LangChain (
langchain-tavily): https://docs.langchain.com/oss/python/integrations/tools/tavily_search - LangGraph: https://docs.langchain.com/oss/python/langgraph/overview
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/corrective-rag">View corrective-rag on skillZs</a>