Assignment: RAGAS Evaluation Metrics

Assignment Metadata

Field	Description
Assignment Name	RAGAS Evaluation Metrics for RAG Systems
Course	LLMOps and Evaluation
Project Name	ragas-evaluation-lab
Estimated Time	90 minutes
Framework	Python 3.10+, RAGAS, LangChain, OpenAI API

---

Learning Objectives

By completing this assignment, you will be able to:

• Implement RAGAS evaluation metrics for RAG systems
• Calculate Faithfulness scores by decomposing answers into verifiable statements
• Measure Answer Relevancy using reverse-engineered questions and embedding similarity
• Evaluate Context Precision and Context Recall for retrieval quality assessment
• Analyze the relationship between different metrics and overall RAG performance

---

Problem Description

You are tasked with building an evaluation pipeline for a Q&A RAG system. The system retrieves documents and generates answers, but you need to measure its quality across multiple dimensions:

1. Faithfulness: Are generated answers grounded in the retrieved context?
2. Answer Relevancy: Do answers actually address the user's questions?
3. Context Precision: Are relevant documents ranked higher in retrieval?
4. Context Recall: Does retrieval capture all necessary information?

---

Technical Requirements

Environment Setup

• Python 3.10 or higher
• Required packages:
  • ragas >= 0.1.0
  • langchain >= 0.1.0
  • openai >= 1.0.0
  • datasets (HuggingFace)

Dataset

Create or use a Q&A dataset with:

• At least 20 question-answer pairs
• Each item containing: question, ground_truth answer, retrieved contexts, generated answer

---

Tasks

Task 1: Faithfulness Evaluation (25 points)

1. Implement Faithfulness scoring that:

   • Decomposes generated answers into individual claims/statements
   • Verifies each claim against the retrieved context
   • Calculates the ratio of supported claims

2. Create test cases demonstrating:

   • High faithfulness (score > 0.9): All claims supported by context
   • Medium faithfulness (0.5-0.9): Partial support
   • Low faithfulness (< 0.5): Hallucinated content

3. Document at least 3 examples with detailed analysis of claim decomposition

Task 2: Answer Relevancy Evaluation (25 points)

1. Implement Answer Relevancy scoring that:

   • Generates N hypothetical questions from the answer
   • Computes embedding similarity with the original question
   • Returns average cosine similarity score

2. Test with examples showing:

   • Complete answers (high relevancy)
   • Partial answers (medium relevancy)
   • Off-topic answers (low relevancy)

3. Analyze how answer completeness affects the relevancy score

Task 3: Context Precision & Recall (25 points)

1. Implement Context Precision that:

   • Evaluates relevance of each retrieved chunk
   • Calculates Precision@k at each position
   • Computes weighted average for final score

2. Implement Context Recall that:

   • Decomposes reference answer into claims
   • Checks attribution to retrieved contexts
   • Calculates coverage ratio

3. Create a retrieval analysis with:

   • At least 5 queries with varying retrieval quality
   • Precision/Recall scores for each
   • Recommendations for improvement

Task 4: End-to-End Evaluation Pipeline (25 points)

1. Build a complete evaluation pipeline using RAGAS:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

# Your evaluation code here

2. Evaluate your RAG system on the full dataset

3. Generate a report with:

   • Summary statistics (mean, std, min, max for each metric)
   • Correlation analysis between metrics
   • Identified failure cases and root causes

---

Submission Requirements

Required Deliverables

• Source code (Jupyter notebook or Python scripts)
• README.md with setup and usage instructions
• Evaluation results table with all four metrics
• Analysis report with examples and insights
• Screenshots of RAGAS evaluation outputs

Submission Checklist

• All code runs without errors
• All four RAGAS metrics are implemented correctly
• Test cases cover edge cases and failure modes
• Analysis includes actionable recommendations
• Documentation is complete and clear

---

Evaluation Criteria

Criteria	Points
Faithfulness implementation & analysis	25
Answer Relevancy implementation	25
Context Precision & Recall	25
End-to-end pipeline & reporting	15
Code quality and documentation	10
Total	100

---

Hints

tip

• Use the companion notebook 10_RAG_Evaluation_with_Ragas.ipynb as a reference
• Start with small examples to understand each metric before scaling up
• For Faithfulness, consider using GPT-4 for more accurate claim verification
• When testing Answer Relevancy, vary the completeness of answers systematically
• Compare your manual calculations with RAGAS automated scores to validate understanding