Rag Evaluation

Skill Profile

(Select at least one profile to enable specific modules)

Comprehensive guide to evaluating Retrieval-Augmented Generation systems, including retrieval metrics, generation quality, and end-to-end evaluation.

Inputs:
- Query dataset (questions with relevant documents and ground truth answers)
- RAG system components (retriever, generator, optional reranker)
- Evaluation configuration (K values, metrics to compute)
Entry Conditions:
- RAG system is deployed and accessible
- Vector database is populated with documents
- Query dataset with ground truth is available
- LLM client for evaluation is configured
Outputs:
- Retrieval metrics (precision@K, recall@K, MRR, NDCG, MAP)
- Generation metrics (faithfulness, relevance, coherence, factuality)
- End-to-end metrics (answer correctness, completeness)
- Evaluation report with visualizations
Artifacts Required (Deliverables):
- Evaluation results JSON
- Metrics dashboard
- Comparison report (if comparing systems)
- Benchmark dataset
Acceptance Evidence:
- All metrics computed successfully
- Results saved to file/database
- Dashboard displays metrics correctly
- Report generated with analysis
Success Criteria:
- Retrieval precision@5 > 80%
- Generation faithfulness > 0.9
- End-to-end accuracy > 85%
- Evaluation completes within expected time

Depends on:
- 52-ai-evaluation/ground-truth-management
- 52-ai-evaluation/llm-judge-patterns
Compatible with:
- 52-ai-evaluation/offline-vs-online-eval
- 52-ai-evaluation/regression-benchmarks
Conflicts with: None
Related Skills:

# Example implementation following best practices
def example_function():
    # Your implementation here
    pass

Assumptions:
- Development environment is properly configured
- Required dependencies are available
- Team has basic understanding of domain
Constraints:
- Must follow existing codebase conventions
- Time and resource limitations
- Compatibility requirements
Non-goals:
- This skill does not cover edge cases outside scope
- Not a replacement for formal training

Supported Versions:
- Python 3.8+
- Node.js 16+
- Modern browsers (Chrome, Firefox, Safari, Edge)
Required AI Tools:
- Code editor (VS Code recommended)
- Testing framework appropriate for language
- Version control (Git)
Dependencies:
- Language-specific package manager
- Build tools
- Testing libraries
Environment Setup:
- .env.example keys: API_KEY, DATABASE_URL (no values)

[ ] Data Handling: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
[ ] Secrets Management: No hardcoded API keys. Use Env Vars/Secrets Manager
[ ] Authorization: Validate user permissions before state changes

[ ] Logging Standards: Structured JSON, include trace IDs request_id
[ ] Metrics: Track error_rate, latency, queue_depth
[ ] Error Handling: Standardized error codes, no bare except
[ ] Observability Artifacts:
- Log Fields: timestamp, level, message, request_id
- Metrics: request_count, error_count, response_time
- Dashboards/Alerts: High Error Rate > 5%

(ข้อกำหนดสำหรับ AI Agent ในการคิดและแก้ปัญหาเมื่อเกิดข้อผิดพลาด)

Thinking Process: Analyze root cause before fixing. Do not brute-force.
Fallback Strategy: Stop after 3 failed test attempts. Output root cause and ask for human intervention/clarification.
Self-Review: Check against Guardrails & Anti-patterns before finalizing.
Output Constraints: Output ONLY the modified code block. Do not explain unless asked.