Setup & Installation
Or with the ClawHub CLI, for registry-managed skill folders outside a full OpenClaw workspace:
Version History
Fixed versioning regression and added simplified installation instructions.
What This Skill Does
Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.
Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.
When to use it
- Checking if a chatbot's answers are supported by retrieved documents
- Detecting hallucinations in RAG-generated responses before shipping
- Running batch quality checks on a Q&A dataset ahead of deployment
- Identifying irrelevant context chunks reducing retrieval precision
- Tracking RAG output quality over time with daily scored logs
Example Workflow
Here's how your AI assistant might use this skill in practice.
User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']
- 1Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json
- 2Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json
- 3Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score
- 4Delete the temp file
- 5Present a human-readable summary with PASS/REVIEW/FAIL verdict
Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS
Requirements
Accounts, API keys, or tools you or your AI assistant may need to set up while using this skill.