Syllabus: LLMOps and Evaluation
1. Header Information
| Field | Value |
|---|---|
| Technical Group | ? |
| Topic Name | LLMOps and Evaluation |
| Topic Code | ? |
| Version | 1.0 |
| Training Audience | Freshers |
2. Course Objectives
[!NOTE] This topic introduces LLMOps and Evaluation knowledge; adapting trainees with skills, lessons, and practices specific to evaluating RAG systems, implementing observability tools like LangFuse and LangSmith, and designing deep architectural experiments.
Output Standards table
| Name | Code | Description |
|---|---|---|
| Ragas Evaluation Metrics | ? | Evaluate RAG quality using Faithfulness, Answer Relevancy, Context Precision, and Context Recall. |
| Observability Tools | ? | Master open source and native observability tools (LangFuse & LangSmith) for tracing and debugging. |
| Experiment Comparison | ? | Design experimental frameworks to rigorously compare Naive, Advanced, GraphRAG, and Hybrid systems. |
Detailed Learning Outcomes (LOs)
?LO1- Able to comprehend and apply Ragas framework metrics for automated LLM evaluation.?LO2- Able to calculate Faithfulness and Answer Relevancy for generation quality.?LO3- Able to assess Context Precision and Context Recall for retrieval performance.?LO4- Able to set up and integrate LangFuse for Open Source tracing and prompt management.?LO5- Able to configure LangSmith for native LangChain execution tracing and playground debugging.?LO6- Able to implement best practices for production LLMOps (Sampling, PII, Alerts).?LO7- Able to design an experimental infrastructure for comparing different RAG architectures.?LO8- Able to analyze trade-offs (quality, cost, latency) across Naive, Advanced, Graph, and Hybrid setups.
3. Topic Outline
- Unit 01: Ragas Evaluation Metrics
- Unit 02: Observability: LangFuse & LangSmith
- Unit 03: Experiment Comparison: Naive, Graph, Hybrid
4. Time Allocation
| Activity Type | Percentage | Description |
|---|---|---|
| Concept/Lecture | 40% | Concepts, theory (3 Sessions) |
| Assignment/Lab | 40% | Assignment, Lab |
| Guides/Review | 10% | Self review/Cross review |
| Exam | 10% | Pre-Test, Final Topic Test |
5. Training Materials & Environments
Textbooks
- Advanced AI Module: LLMOps and Evaluation Guidelines
References
Technical Requirements
- Python 3.10+
- Ragas, LangChain, LangSmith, LangFuse
- Neo4j, Qdrant/ChromaDB
- IDE (VSCode / PyCharm)
6. Assessment Scheme
| Component | Quantity | Weight (%) | Note |
|---|---|---|---|
| Quiz | 5 | 20% | In-class labs |
| Assignments | 2 | 30% | In-class labs |
| Final Practice Test | 1 | 50% | Comprehensive project |
Pass Criteria
- Total topic GPA >= 7.0/10
- Completed 100% Online video, Quiz, Assignments, Labs, Final Test
- Successful demo of the Final Project (RAG system with evaluation and tracing)
7. Training Delivery Principles
| Role | Responsibility/Criteria |
|---|---|
| Trainees | Must be familiar with basic RAG. |
| Trainer | Certified AI Engineer or higher. |
| Training | Daily offline training and labs. |
| Re-Test | Allowed 1 time for Failed status. |
| Marking | Direct grading of Practice tests. |
| Waiver Criteria | Equivalent industry certificates. |
| Others | Need personal laptop for labs. |