production-rag-checklist
Deploy RAG systems to production with proper architecture, monitoring, and operations. Use this skill when deploying RAG to production, scaling retrieval systems, implementing RAG monitoring, or building enterprise RAG infrastructure. Activate when: production RAG, RAG deployment, enterprise RAG, RAG architecture, RAG monitoring, RAG scaling, RAG infrastructure, RAG ops.
How do I install this agent skill?
npx skills add https://github.com/latestaiagents/agent-skills --skill production-rag-checklistIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill is a technical guide and checklist for deploying Production Retrieval-Augmented Generation (RAG) systems. It provides architecture patterns, example code, and security best practices. The skill promotes defensive measures like PII filtering and input validation, and its code snippets use only reputable, standard libraries.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
- Runlayerwarn
1/1 file flagged
What does this agent skill do?
Production RAG Checklist
Everything you need to deploy RAG systems that are reliable, scalable, and maintainable.
Pre-Production Checklist
Data & Indexing
-
Document pipeline defined
- Ingestion source(s) identified
- Parsing strategy per document type
- Chunking strategy tested and benchmarked
- Metadata extraction configured
-
Index refresh strategy
- Real-time (streaming)
- Scheduled (hourly/daily)
- On-demand (manual trigger)
- Change detection for modified documents (content hash per chunk)
- Deletion handling: removed source documents are removed from the index
- Stale document detection
- Incremental updates (not full rebuild)
-
Data quality
- Duplicate detection/removal
- Empty/corrupt document handling
- PII detection and handling
- Source verification
-
Metadata on every chunk
- Source (URL, file path, version) for citations
- Timestamps (created, updated, indexed)
- Document type
- Access-control tags (tenant, group, or document ACL) when users may not see every document
-
Embedding pipeline robust
- Rate limiting and retries for API-based embeddings
- Failed embeddings retried or quarantined, not silently skipped
Retrieval
-
Retrieval strategy selected
- Vector-only
- Hybrid (vector + keyword)
- GraphRAG
- Reranking configured
-
Embeddings
- Model selected and tested
- Embedding dimension appropriate
- Batch embedding for bulk operations
- Embedding versioning strategy
-
Vector store
- Production-grade store selected (Pinecone, Weaviate, Qdrant, etc.)
- Index type and parameters chosen deliberately (for HNSW:
m,ef_construction, query-timeef) and checked against recall and latency on your data - Distance metric matches what the embedding model was trained for (see its model card)
- Estimated vector count and growth rate
- Scaling strategy (sharding, replicas)
- Failover tested; connection pooling enabled
- Backup and recovery tested
-
Access control enforced at retrieval
- Permission filters applied in the vector query, not after generation
- Answers never quote documents the user cannot open
Generation
-
LLM configuration
- Model selected (cost vs quality tradeoff)
- Temperature and parameters tuned
- Context window sized appropriately
- Fallback model configured
-
Prompt engineering
- System prompt tested across scenarios
- Few-shot examples where they help
- Output format specified
- Guardrails in place (grounding checks, PII redaction where needed)
-
Fallback responses defined
- "I don't know" when retrieval finds nothing relevant
- User-facing error messages that don't leak internals
-
Evaluation baseline
- Labeled query set, including edge cases and no-answer questions (see
rag-evaluation) - Baseline retrieval and answer metrics recorded before launch
- Labeled query set, including edge cases and no-answer questions (see
Architecture Patterns
Basic Production Architecture
┌─────────────────────────────────────────────────────────────┐
│ Load Balancer │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ API Server │ │ API Server │ │ API Server │
│ (Stateless) │ │ (Stateless) │ │ (Stateless) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────┼───────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Redis Cache │ │ Vector Store │ │ LLM Gateway │
│ (Query Cache) │ │ (Retrieval) │ │ (Rate Limit) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
API Layer With Timeouts
import asyncio
import time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
class QueryRequest(BaseModel):
query: str
top_k: int = 5
filters: dict | None = None
@app.post("/query")
async def query_rag(request: QueryRequest):
start = time.perf_counter()
try:
result = await asyncio.wait_for(rag.query(request.query, request.top_k, request.filters), timeout=30.0)
except asyncio.TimeoutError:
raise HTTPException(status_code=504, detail="Query timed out")
except Exception:
logger.exception("rag_error")
raise HTTPException(status_code=500, detail="Internal error") # don't leak details
return {
"answer": result["answer"],
"sources": [d.metadata["source"] for d in result["documents"]],
"latency_ms": (time.perf_counter() - start) * 1000,
}
Add per-user rate limiting at the gateway or with middleware (for example slowapi for FastAPI).
LLM-Agnostic Design
from abc import ABC, abstractmethod
class LLMProvider(ABC):
@abstractmethod
async def generate(self, prompt: str, **kwargs) -> str:
pass
class OpenAIProvider(LLMProvider):
async def generate(self, prompt: str, **kwargs) -> str:
# OpenAI implementation
pass
class AnthropicProvider(LLMProvider):
async def generate(self, prompt: str, **kwargs) -> str:
# Anthropic implementation
pass
class RAGService:
def __init__(self, llm_provider: LLMProvider, retriever):
self.llm = llm_provider
self.retriever = retriever
async def query(self, question: str) -> str:
docs = await self.retriever.ainvoke(question)
context = self._format_context(docs)
return await self.llm.generate(
f"Context: {context}\n\nQuestion: {question}"
)
Caching Strategies
Query Cache
The cache key must include everything that changes the answer: the query, filters, and the caller's permission scope. Otherwise one user's cached answer can leak documents to another.
import hashlib
import json
import redis
class RAGCache:
def __init__(self, redis_client: redis.Redis, ttl: int = 3600):
self.redis = redis_client
self.ttl = ttl
def _cache_key(self, query: str, filters: dict | None, acl_scope: str) -> str:
raw = json.dumps({"q": query, "f": filters or {}, "acl": acl_scope}, sort_keys=True)
return f"rag:query:{hashlib.sha256(raw.encode()).hexdigest()}"
async def get_or_compute(self, query: str, filters: dict | None, acl_scope: str, compute_fn) -> dict:
key = self._cache_key(query, filters, acl_scope)
cached = self.redis.get(key)
if cached:
return json.loads(cached)
result = await compute_fn(query, filters)
self.redis.setex(key, self.ttl, json.dumps(result))
# Remember which cache entries used which sources, for invalidation
for source in result.get("sources", []):
self.redis.sadd(f"rag:by_source:{source}", key)
return result
def invalidate_source(self, source: str):
"""Call when a source document changes or is deleted."""
keys = self.redis.smembers(f"rag:by_source:{source}")
if keys:
self.redis.delete(*keys)
self.redis.delete(f"rag:by_source:{source}")
Embedding Cache
class EmbeddingCache:
"""Cache embeddings to avoid recomputation."""
def __init__(self, redis_client: redis.Redis):
self.redis = redis_client
def get_embedding(self, text: str) -> list[float] | None:
key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
cached = self.redis.get(key)
return json.loads(cached) if cached else None
def set_embedding(self, text: str, embedding: list[float]):
key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
self.redis.setex(key, 86400 * 7, json.dumps(embedding)) # 7 days
Error Handling & Fallbacks
class ResilientRAG:
"""RAG with comprehensive error handling."""
def __init__(self, primary_llm, fallback_llm, retriever):
self.primary_llm = primary_llm
self.fallback_llm = fallback_llm
self.retriever = retriever
async def query(self, question: str) -> dict:
try:
# Retrieval with timeout
docs = await asyncio.wait_for(
self.retriever.ainvoke(question),
timeout=5.0
)
except asyncio.TimeoutError:
logger.warning("Retrieval timeout, using empty context")
docs = []
except Exception as e:
logger.error(f"Retrieval error: {e}")
docs = []
if not docs:
return {
"answer": "I couldn't find relevant information to answer your question.",
"sources": [],
"status": "no_results"
}
context = self._format_context(docs)
# Try primary LLM
try:
answer = await asyncio.wait_for(
self.primary_llm.generate(question, context),
timeout=30.0
)
return {"answer": answer, "sources": docs, "status": "success"}
except Exception as e:
logger.warning(f"Primary LLM failed: {e}, trying fallback")
# Try fallback LLM
try:
answer = await self.fallback_llm.generate(question, context)
return {"answer": answer, "sources": docs, "status": "fallback"}
except Exception as e:
logger.error(f"Fallback LLM failed: {e}")
return {
"answer": "I'm having trouble generating a response. Please try again.",
"sources": docs,
"status": "error"
}
Observability & Monitoring
Logging Structure
import structlog
logger = structlog.get_logger()
async def monitored_query(rag, query: str, user_id: str) -> dict:
"""Query with comprehensive logging."""
request_id = str(uuid.uuid4())
logger.info(
"rag_query_started",
request_id=request_id,
user_id=user_id,
query_length=len(query)
)
start = time.perf_counter()
try:
# Retrieval
retrieval_start = time.perf_counter()
docs = await rag.retriever.ainvoke(query)
retrieval_ms = (time.perf_counter() - retrieval_start) * 1000
logger.info(
"rag_retrieval_complete",
request_id=request_id,
num_docs=len(docs),
latency_ms=retrieval_ms
)
# Generation
gen_start = time.perf_counter()
answer = await rag.generate(query, docs)
gen_ms = (time.perf_counter() - gen_start) * 1000
total_ms = (time.perf_counter() - start) * 1000
logger.info(
"rag_query_complete",
request_id=request_id,
retrieval_ms=retrieval_ms,
generation_ms=gen_ms,
total_ms=total_ms,
answer_length=len(answer)
)
return {"answer": answer, "request_id": request_id}
except Exception as e:
logger.error(
"rag_query_failed",
request_id=request_id,
error=str(e),
error_type=type(e).__name__
)
raise
Key Metrics to Track
# Prometheus metrics example
from prometheus_client import Counter, Histogram, Gauge
# Counters
rag_queries_total = Counter(
'rag_queries_total',
'Total RAG queries',
['status'] # success, no_results, error
)
# Histograms
rag_latency = Histogram(
'rag_latency_seconds',
'RAG query latency',
['stage'], # retrieval, generation, total
buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
)
# Gauges
rag_index_size = Gauge(
'rag_index_document_count',
'Number of documents in RAG index'
)
Dashboard Essentials
Example starting thresholds; set real ones from your baseline and SLOs:
| Metric | Example Alert Threshold |
|---|---|
| P95 latency | >5 seconds |
| Error rate | >5% |
| Cache hit rate | <50% |
| Empty results rate | >20% |
| LLM fallback rate | >10% |
Alerting Rules
Prometheus rules for the metrics above. histogram_quantile needs the per-bucket rate, not the raw histogram name:
groups:
- name: rag_alerts
rules:
- alert: RAGHighLatency
expr: histogram_quantile(0.95, sum by (le) (rate(rag_latency_seconds_bucket{stage="total"}[5m]))) > 5
for: 5m
labels:
severity: warning
annotations:
summary: "RAG p95 latency above 5s"
- alert: RAGHighErrorRate
expr: sum(rate(rag_queries_total{status="error"}[5m])) / sum(rate(rag_queries_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "RAG error rate above 5%"
Track quality as well as uptime: the empty-results rate, the share of "I don't know" answers, and user feedback (thumbs up/down) often move before error rates do.
Security Checklist
-
Input validation
- Query length limits
- Prompt injection detection
- Rate limiting per user
-
Data protection
- PII filtering in responses
- Document-level access control enforced in retrieval and in the cache key
- Retrieved documents treated as untrusted input (indirect prompt injection)
- Data retention policy for logged queries
- Audit logging
-
API security
- Authentication required
- HTTPS only
- API key rotation
Cost Optimization
class CostAwareRAG:
"""RAG with cost controls."""
def __init__(self, rag, monthly_budget: float):
self.rag = rag
self.budget = monthly_budget
self.month_spend = 0
async def query(self, question: str) -> dict:
# Check budget
if self.month_spend >= self.budget:
return {"error": "Monthly budget exceeded"}
# Estimate cost before query
estimated_cost = self._estimate_cost(question)
if self.month_spend + estimated_cost > self.budget:
# Use cheaper model
return await self.rag.query_cheap(question)
result = await self.rag.query(question)
self.month_spend += self._actual_cost(result)
return result
def _estimate_cost(self, question: str) -> float:
# HYPOTHETICAL arithmetic: the token heuristics and prices below are
# placeholders. Use your provider's token counter and current pricing page.
input_tokens = len(question.split()) * 1.3 + CONTEXT_TOKENS
output_tokens = 500
return input_tokens * PRICE_PER_INPUT_TOKEN + output_tokens * PRICE_PER_OUTPUT_TOKEN
Set spend alerts with your LLM provider as well; in-process budgets reset when the process restarts.
Deployment Checklist
Pre-Deploy
- Load testing completed (target QPS reached)
- Failover tested
- Evaluation metrics meet the targets you set from your baseline
- Rollback plan documented
- Runbook written (reindex, cache flush, provider outage)
- On-call rotation set up
- Feedback collection in place
Deploy
- Canary deployment (10% traffic)
- Monitor error rates
- Monitor latency
- Gradual rollout
Post-Deploy
- Verify metrics stable
- Check cache hit rates
- Monitor user feedback
- Schedule first review (1 week)
Best Practices
- Version everything - Embedding model, chunking settings, prompts, and index builds, so any answer can be traced and reproduced
- Plan for reindexing - Changing the embedding model or chunking means rebuilding; build the index alongside the old one and switch over
- Monitor quality, not just uptime - Empty results, "I don't know" rate, and user feedback
- Cache carefully - Caching cuts cost and latency, but keys must include filters and permissions, and entries must be invalidated when sources change
- Set cost alerts - Agentic and long-context queries can spike spend
References
- Prometheus histograms and
histogram_quantile: https://prometheus.io/docs/practices/histograms/ - OWASP Top 10 for LLM Applications (prompt injection, sensitive information disclosure): https://genai.owasp.org/llm-top-10/
- FastAPI: https://fastapi.tiangolo.com/
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/production-rag-checklist">View production-rag-checklist on skillZs</a>