skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
latestaiagents/agent-skills60 installs

production-rag-checklist

Deploy RAG systems to production with proper architecture, monitoring, and operations. Use this skill when deploying RAG to production, scaling retrieval systems, implementing RAG monitoring, or building enterprise RAG infrastructure. Activate when: production RAG, RAG deployment, enterprise RAG, RAG architecture, RAG monitoring, RAG scaling, RAG infrastructure, RAG ops.

How do I install this agent skill?

npx skills add https://github.com/latestaiagents/agent-skills --skill production-rag-checklist
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill is a technical guide and checklist for deploying Production Retrieval-Augmented Generation (RAG) systems. It provides architecture patterns, example code, and security best practices. The skill promotes defensive measures like PII filtering and input validation, and its code snippets use only reputable, standard libraries.

  • Socketpass

    No alerts

  • Snykpass

    Risk: LOW · No issues

  • Runlayerwarn

    1/1 file flagged

What does this agent skill do?

Production RAG Checklist

Everything you need to deploy RAG systems that are reliable, scalable, and maintainable.

Pre-Production Checklist

Data & Indexing

  • Document pipeline defined

    • Ingestion source(s) identified
    • Parsing strategy per document type
    • Chunking strategy tested and benchmarked
    • Metadata extraction configured
  • Index refresh strategy

    • Real-time (streaming)
    • Scheduled (hourly/daily)
    • On-demand (manual trigger)
    • Change detection for modified documents (content hash per chunk)
    • Deletion handling: removed source documents are removed from the index
    • Stale document detection
    • Incremental updates (not full rebuild)
  • Data quality

    • Duplicate detection/removal
    • Empty/corrupt document handling
    • PII detection and handling
    • Source verification
  • Metadata on every chunk

    • Source (URL, file path, version) for citations
    • Timestamps (created, updated, indexed)
    • Document type
    • Access-control tags (tenant, group, or document ACL) when users may not see every document
  • Embedding pipeline robust

    • Rate limiting and retries for API-based embeddings
    • Failed embeddings retried or quarantined, not silently skipped

Retrieval

  • Retrieval strategy selected

    • Vector-only
    • Hybrid (vector + keyword)
    • GraphRAG
    • Reranking configured
  • Embeddings

    • Model selected and tested
    • Embedding dimension appropriate
    • Batch embedding for bulk operations
    • Embedding versioning strategy
  • Vector store

    • Production-grade store selected (Pinecone, Weaviate, Qdrant, etc.)
    • Index type and parameters chosen deliberately (for HNSW: m, ef_construction, query-time ef) and checked against recall and latency on your data
    • Distance metric matches what the embedding model was trained for (see its model card)
    • Estimated vector count and growth rate
    • Scaling strategy (sharding, replicas)
    • Failover tested; connection pooling enabled
    • Backup and recovery tested
  • Access control enforced at retrieval

    • Permission filters applied in the vector query, not after generation
    • Answers never quote documents the user cannot open

Generation

  • LLM configuration

    • Model selected (cost vs quality tradeoff)
    • Temperature and parameters tuned
    • Context window sized appropriately
    • Fallback model configured
  • Prompt engineering

    • System prompt tested across scenarios
    • Few-shot examples where they help
    • Output format specified
    • Guardrails in place (grounding checks, PII redaction where needed)
  • Fallback responses defined

    • "I don't know" when retrieval finds nothing relevant
    • User-facing error messages that don't leak internals
  • Evaluation baseline

    • Labeled query set, including edge cases and no-answer questions (see rag-evaluation)
    • Baseline retrieval and answer metrics recorded before launch

Architecture Patterns

Basic Production Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Load Balancer                         │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   API Server    │ │   API Server    │ │   API Server    │
│   (Stateless)   │ │   (Stateless)   │ │   (Stateless)   │
└─────────────────┘ └─────────────────┘ └─────────────────┘
              │               │               │
              └───────────────┼───────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   Redis Cache   │ │  Vector Store   │ │   LLM Gateway   │
│  (Query Cache)  │ │  (Retrieval)    │ │  (Rate Limit)   │
└─────────────────┘ └─────────────────┘ └─────────────────┘

API Layer With Timeouts

import asyncio
import time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class QueryRequest(BaseModel):
    query: str
    top_k: int = 5
    filters: dict | None = None

@app.post("/query")
async def query_rag(request: QueryRequest):
    start = time.perf_counter()
    try:
        result = await asyncio.wait_for(rag.query(request.query, request.top_k, request.filters), timeout=30.0)
    except asyncio.TimeoutError:
        raise HTTPException(status_code=504, detail="Query timed out")
    except Exception:
        logger.exception("rag_error")
        raise HTTPException(status_code=500, detail="Internal error")  # don't leak details
    return {
        "answer": result["answer"],
        "sources": [d.metadata["source"] for d in result["documents"]],
        "latency_ms": (time.perf_counter() - start) * 1000,
    }

Add per-user rate limiting at the gateway or with middleware (for example slowapi for FastAPI).

LLM-Agnostic Design

from abc import ABC, abstractmethod

class LLMProvider(ABC):
    @abstractmethod
    async def generate(self, prompt: str, **kwargs) -> str:
        pass

class OpenAIProvider(LLMProvider):
    async def generate(self, prompt: str, **kwargs) -> str:
        # OpenAI implementation
        pass

class AnthropicProvider(LLMProvider):
    async def generate(self, prompt: str, **kwargs) -> str:
        # Anthropic implementation
        pass

class RAGService:
    def __init__(self, llm_provider: LLMProvider, retriever):
        self.llm = llm_provider
        self.retriever = retriever

    async def query(self, question: str) -> str:
        docs = await self.retriever.ainvoke(question)
        context = self._format_context(docs)
        return await self.llm.generate(
            f"Context: {context}\n\nQuestion: {question}"
        )

Caching Strategies

Query Cache

The cache key must include everything that changes the answer: the query, filters, and the caller's permission scope. Otherwise one user's cached answer can leak documents to another.

import hashlib
import json
import redis

class RAGCache:
    def __init__(self, redis_client: redis.Redis, ttl: int = 3600):
        self.redis = redis_client
        self.ttl = ttl

    def _cache_key(self, query: str, filters: dict | None, acl_scope: str) -> str:
        raw = json.dumps({"q": query, "f": filters or {}, "acl": acl_scope}, sort_keys=True)
        return f"rag:query:{hashlib.sha256(raw.encode()).hexdigest()}"

    async def get_or_compute(self, query: str, filters: dict | None, acl_scope: str, compute_fn) -> dict:
        key = self._cache_key(query, filters, acl_scope)
        cached = self.redis.get(key)
        if cached:
            return json.loads(cached)

        result = await compute_fn(query, filters)
        self.redis.setex(key, self.ttl, json.dumps(result))
        # Remember which cache entries used which sources, for invalidation
        for source in result.get("sources", []):
            self.redis.sadd(f"rag:by_source:{source}", key)
        return result

    def invalidate_source(self, source: str):
        """Call when a source document changes or is deleted."""
        keys = self.redis.smembers(f"rag:by_source:{source}")
        if keys:
            self.redis.delete(*keys)
        self.redis.delete(f"rag:by_source:{source}")

Embedding Cache

class EmbeddingCache:
    """Cache embeddings to avoid recomputation."""

    def __init__(self, redis_client: redis.Redis):
        self.redis = redis_client

    def get_embedding(self, text: str) -> list[float] | None:
        key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
        cached = self.redis.get(key)
        return json.loads(cached) if cached else None

    def set_embedding(self, text: str, embedding: list[float]):
        key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
        self.redis.setex(key, 86400 * 7, json.dumps(embedding))  # 7 days

Error Handling & Fallbacks

class ResilientRAG:
    """RAG with comprehensive error handling."""

    def __init__(self, primary_llm, fallback_llm, retriever):
        self.primary_llm = primary_llm
        self.fallback_llm = fallback_llm
        self.retriever = retriever

    async def query(self, question: str) -> dict:
        try:
            # Retrieval with timeout
            docs = await asyncio.wait_for(
                self.retriever.ainvoke(question),
                timeout=5.0
            )
        except asyncio.TimeoutError:
            logger.warning("Retrieval timeout, using empty context")
            docs = []
        except Exception as e:
            logger.error(f"Retrieval error: {e}")
            docs = []

        if not docs:
            return {
                "answer": "I couldn't find relevant information to answer your question.",
                "sources": [],
                "status": "no_results"
            }

        context = self._format_context(docs)

        # Try primary LLM
        try:
            answer = await asyncio.wait_for(
                self.primary_llm.generate(question, context),
                timeout=30.0
            )
            return {"answer": answer, "sources": docs, "status": "success"}

        except Exception as e:
            logger.warning(f"Primary LLM failed: {e}, trying fallback")

            # Try fallback LLM
            try:
                answer = await self.fallback_llm.generate(question, context)
                return {"answer": answer, "sources": docs, "status": "fallback"}
            except Exception as e:
                logger.error(f"Fallback LLM failed: {e}")
                return {
                    "answer": "I'm having trouble generating a response. Please try again.",
                    "sources": docs,
                    "status": "error"
                }

Observability & Monitoring

Logging Structure

import structlog

logger = structlog.get_logger()

async def monitored_query(rag, query: str, user_id: str) -> dict:
    """Query with comprehensive logging."""
    request_id = str(uuid.uuid4())

    logger.info(
        "rag_query_started",
        request_id=request_id,
        user_id=user_id,
        query_length=len(query)
    )

    start = time.perf_counter()

    try:
        # Retrieval
        retrieval_start = time.perf_counter()
        docs = await rag.retriever.ainvoke(query)
        retrieval_ms = (time.perf_counter() - retrieval_start) * 1000

        logger.info(
            "rag_retrieval_complete",
            request_id=request_id,
            num_docs=len(docs),
            latency_ms=retrieval_ms
        )

        # Generation
        gen_start = time.perf_counter()
        answer = await rag.generate(query, docs)
        gen_ms = (time.perf_counter() - gen_start) * 1000

        total_ms = (time.perf_counter() - start) * 1000

        logger.info(
            "rag_query_complete",
            request_id=request_id,
            retrieval_ms=retrieval_ms,
            generation_ms=gen_ms,
            total_ms=total_ms,
            answer_length=len(answer)
        )

        return {"answer": answer, "request_id": request_id}

    except Exception as e:
        logger.error(
            "rag_query_failed",
            request_id=request_id,
            error=str(e),
            error_type=type(e).__name__
        )
        raise

Key Metrics to Track

# Prometheus metrics example
from prometheus_client import Counter, Histogram, Gauge

# Counters
rag_queries_total = Counter(
    'rag_queries_total',
    'Total RAG queries',
    ['status']  # success, no_results, error
)

# Histograms
rag_latency = Histogram(
    'rag_latency_seconds',
    'RAG query latency',
    ['stage'],  # retrieval, generation, total
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
)

# Gauges
rag_index_size = Gauge(
    'rag_index_document_count',
    'Number of documents in RAG index'
)

Dashboard Essentials

Example starting thresholds; set real ones from your baseline and SLOs:

MetricExample Alert Threshold
P95 latency>5 seconds
Error rate>5%
Cache hit rate<50%
Empty results rate>20%
LLM fallback rate>10%

Alerting Rules

Prometheus rules for the metrics above. histogram_quantile needs the per-bucket rate, not the raw histogram name:

groups:
  - name: rag_alerts
    rules:
      - alert: RAGHighLatency
        expr: histogram_quantile(0.95, sum by (le) (rate(rag_latency_seconds_bucket{stage="total"}[5m]))) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "RAG p95 latency above 5s"

      - alert: RAGHighErrorRate
        expr: sum(rate(rag_queries_total{status="error"}[5m])) / sum(rate(rag_queries_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "RAG error rate above 5%"

Track quality as well as uptime: the empty-results rate, the share of "I don't know" answers, and user feedback (thumbs up/down) often move before error rates do.

Security Checklist

  • Input validation

    • Query length limits
    • Prompt injection detection
    • Rate limiting per user
  • Data protection

    • PII filtering in responses
    • Document-level access control enforced in retrieval and in the cache key
    • Retrieved documents treated as untrusted input (indirect prompt injection)
    • Data retention policy for logged queries
    • Audit logging
  • API security

    • Authentication required
    • HTTPS only
    • API key rotation

Cost Optimization

class CostAwareRAG:
    """RAG with cost controls."""

    def __init__(self, rag, monthly_budget: float):
        self.rag = rag
        self.budget = monthly_budget
        self.month_spend = 0

    async def query(self, question: str) -> dict:
        # Check budget
        if self.month_spend >= self.budget:
            return {"error": "Monthly budget exceeded"}

        # Estimate cost before query
        estimated_cost = self._estimate_cost(question)

        if self.month_spend + estimated_cost > self.budget:
            # Use cheaper model
            return await self.rag.query_cheap(question)

        result = await self.rag.query(question)
        self.month_spend += self._actual_cost(result)
        return result

    def _estimate_cost(self, question: str) -> float:
        # HYPOTHETICAL arithmetic: the token heuristics and prices below are
        # placeholders. Use your provider's token counter and current pricing page.
        input_tokens = len(question.split()) * 1.3 + CONTEXT_TOKENS
        output_tokens = 500
        return input_tokens * PRICE_PER_INPUT_TOKEN + output_tokens * PRICE_PER_OUTPUT_TOKEN

Set spend alerts with your LLM provider as well; in-process budgets reset when the process restarts.

Deployment Checklist

Pre-Deploy

  • Load testing completed (target QPS reached)
  • Failover tested
  • Evaluation metrics meet the targets you set from your baseline
  • Rollback plan documented
  • Runbook written (reindex, cache flush, provider outage)
  • On-call rotation set up
  • Feedback collection in place

Deploy

  • Canary deployment (10% traffic)
  • Monitor error rates
  • Monitor latency
  • Gradual rollout

Post-Deploy

  • Verify metrics stable
  • Check cache hit rates
  • Monitor user feedback
  • Schedule first review (1 week)

Best Practices

  1. Version everything - Embedding model, chunking settings, prompts, and index builds, so any answer can be traced and reproduced
  2. Plan for reindexing - Changing the embedding model or chunking means rebuilding; build the index alongside the old one and switch over
  3. Monitor quality, not just uptime - Empty results, "I don't know" rate, and user feedback
  4. Cache carefully - Caching cuts cost and latency, but keys must include filters and permissions, and entries must be invalidated when sources change
  5. Set cost alerts - Agentic and long-context queries can spike spend

References

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/production-rag-checklist">View production-rag-checklist on skillZs</a>