LLMOps and Evaluation Question Bank

No.	Training Unit	Lecture	Training content	Question	Level	Mark	Answer	Answer Option A	Answer Option B	Answer Option C	Answer Option D	Explanation
1	Unit 1: LLMOps	Lec2	RAGAS Metrics	What does the Faithfulness metric measure in RAGAS?	Easy	1	A	The truthfulness of the generated answer compared to the retrieved context	The relevance of the answer to the original question	The accuracy of the ranking of contexts	The coverage of the retrieval process	Faithfulness checks if all statements in the answer can be supported by the retrieved context, avoiding hallucinations.
2	Unit 1: LLMOps	Lec2	RAGAS Metrics	Which LLM framework is RAGAS designed to evaluate?	Easy	1	B	Agents	RAG systems	Fine-tuned models	Traditional Search Engines	Ragas is an automated evaluation framework designed specifically for RAG systems.
3	Unit 1: LLMOps	Lec2	RAGAS Metrics	What do you need to annotate data manually when using RAGAS?	Easy	1	C	Large scale human annotations	Only expert domain knowledge	Nothing, it uses LLMs like GPT-4 to automate evaluation	Both standard Q&A pairs and ranking queries	Unlike traditional methods, Ragas uses LLMs to automate the evaluation process without needing heavy human annotations.
4	Unit 1: LLMOps	Lec2	RAGAS Metrics	Which dimension is measured by Context Precision?	Easy	1	C	Quality of generation	Semantic similarity to the user query	Accuracy of the retrieval process	Coverage of expected facts	Context Precision measures the accuracy of the retrieval process by assessing the ranking of contexts.
5	Unit 1: LLMOps	Lec2	RAGAS Metrics	What is the main purpose of Answer Relevancy?	Easy	1	D	Fact-checking the answer	Verifying truthfulness	Guaranteeing context coverage	Measuring relevance between answer and original question	It evaluates the relevance between the answer and question to confirm it addresses the problem asked.
6	Unit 1: LLMOps	Lec2	RAGAS Metrics	What value range do Ragas metrics return?	Easy	1	B	0 to 100	0 to 1	-1 to 1	1 to 5	Each metric gives a value from 0 to 1, with higher values indicating better quality.
7	Unit 1: LLMOps	Lec2	RAGAS Metrics	Which metric evaluates if relevant chunks are ranked high in retrieved contexts?	Easy	1	C	Faithfulness	Context Recall	Context Precision	Answer Relevancy	Context Precision checks if relevant chunks are ranked high in the list of retrieved contexts.
8	Unit 1: LLMOps	Lec2	RAGAS Metrics	How many main metrics are covered in the RAGAS documentation?	Easy	1	A	4	5	3	6	The four main metrics are faithfulness, answer relevancy, context precision, and context recall.
9	Unit 1: LLMOps	Lec2	RAGAS Metrics	If Context Recall is 0, what does that indicate?	Easy	1	A	Retriever failed to find necessary context	Rank 1 is an irrelevant context	LLM generated hallucination	The answer is irrelevant to the query	It indicates the retriever failed to find context containing necessary information to answer the question.
10	Unit 1: LLMOps	Lec2	RAGAS Metrics	Which two metrics evaluate the "retrieval" performance?	Easy	1	B	Faithfulness & Answer Relevancy	Context Precision & Context Recall	Answer Relevancy & Context Recall	Context Precision & Faithfulness	Context precision and context recall evaluate retrieval performance.
11	Unit 1: LLMOps	Lec2	RAGAS Metrics	Describe the calculation process for Faithfulness in Ragas.	Medium	2	A	Decompose answer to statements, verify against context, calculate ratio	Generate questions, embed them, calculate cosine similarity	Determine context relevance, calculate Precision@k, aggregate	Decompose reference answer, verify if inferences exist in retrieved context	The process is: Decomposition (claims), Verification (checked against context), and Scoring (ratio).
12	Unit 1: LLMOps	Lec2	RAGAS Metrics	How does Answer Relevancy determine its score technically?	Medium	2	C	By classifying the answer using a trained classifier	By matching keywords between answer and question	By reverse-engineering questions from answer and calculating embedding cosine similarity	By comparing the character count of answer vs question	LLM generates N questions from the given answer, converts them to embeddings, and compares cosine similarity with the original question.
13	Unit 1: LLMOps	Lec2	RAGAS Metrics	A low Context Recall score means what in terms of information availability?	Medium	2	D	The information is hallucinated	The answer has redundant information	The retrieved information is scattered	The necessary facts from the reference answer are missing in the retrieved contexts	It means the necessary information from the reference answer was not found in the retrieved contexts.
14	Unit 1: LLMOps	Lec2	RAGAS Metrics	In Context Precision calculation, what is $v_k$?	Medium	2	C	Velocity of retrieval	Volume of chunks	Relevance indicator at position k	Value of cosine similarity	$v_k \in \{0, 1\}$ is the relevance indicator at position k.
15	Unit 1: LLMOps	Lec2	RAGAS Metrics	Why might an answer score high in Faithfulness but low in Answer Relevancy?	Medium	2	B	The answer is hallucinated but relevant	The answer is entirely true based on context but fails to address the user's specific question	The retriever brought back poor context	The context precision is very low	It can be completely faithful to retrieved context, but that context (and answer) might not be what the user asked for.
16	Unit 1: LLMOps	Lec2	RAGAS Metrics	Why is Faithfulness strictly compared to retrieved context and not world knowledge?	Medium	2	A	To prevent LLM hallucinations from being counted as correct if the retriever failed	Ragas has no access to world knowledge	The LLM doesn't know facts	World knowledge costs more tokens	RAG's core value is grounding generation on specific private/provided context, so it measures adherence to that context only to prevent unaccounted hallucinations.
17	Unit 1: LLMOps	Lec2	RAGAS Metrics	If LLM splits an answer into 3 statements, and only 2 are verified in context, Faithfulness is?	Medium	2	B	0.5	0.67	0.33	1.0	Faithfulness relies on the ratio of correct statements: 2 out of 3 makes it ~0.67.
18	Unit 1: LLMOps	Lec2	RAGAS Metrics	Given a scenario where a user asks about Einstein's death, but the context only contains his birth, and the LLM answers "Einstein died in 1955" using its internal knowledge. What are the RAGAS metric implications?	Hard	3	B	High Faithfulness, Low Answer Relevancy	Low Faithfulness, High Answer Relevancy	Low Faithfulness, Low Context Recall	High Context Precision, High Context Recall	It answers the user (High Relevancy), but the claim isn't in context, making Faithfulness low.
19	Unit 1: LLMOps	Lec2	RAGAS Metrics	To improve Context Precision in a RAG pipeline, what architecture modification would you introduce?	Hard	3	C	Increase LLM temperature	Swap FAISS for ChromaDB	Add a Cross-encoder reranking step	Generate multiple answers and average them	Reranking specifically improves the order/ranking of retrieved chunks, heavily impacting Context Precision metrics.
20	Unit 1: LLMOps	Lec2	RAGAS Metrics	Detail the mathematical rationale behind using N reverse-engineered questions for calculating Answer Relevancy.	Hard	3	A	Averages out the stochastic nature of LLMs generating questions to provide a stable semantic similarity	It is required to satisfy vector dimensions	One question uses up too few tokens	N acts as a padding token for embeddings	Generating N questions and averaging their cosine similarities mitigates the variance inherent in LLM generation, ensuring a robust relevancy score.
21	Unit 2: Observability	Lec6	Observability Concepts	What is Observability in the context of LLM applications?	Easy	1	A	The ability to track flows, errors and costs of LLM apps acting as black boxes	A library for generating UI code	A vector database	The algorithm used for chunking texts	It tracks probabilistic components acting as black boxes, aiding in tracing, tracking costs, and debugging.
22	Unit 2: Observability	Lec6	LangFuse Basics	Which of these tools is known for being Open Source?	Easy	1	B	LangChain	LangFuse	LangSmith	OpenAI	LangFuse is a popular open-source tool focusing on engineering observability.
23	Unit 2: Observability	Lec6	Observability Challenges	What makes LLM applications harder to debug than traditional software?	Easy	1	C	They use more memory	They require internet connections	They involve probabilistic, non-deterministic components	They use Python	You give input, get output. LLMs act as probabilisitic black boxes.
24	Unit 2: Observability	Lec6	LangSmith Basics	Who built LangSmith?	Easy	1	B	Google	The LangChain Team	OpenAI	Meta	LangSmith is built by the LangChain team for native integration.
25	Unit 2: Observability	Lec6	LangFuse Integration	In LangFuse, what is used to automatically instrument LangChain chains code?	Easy	1	C	System.out.println	VectorEmbeddings	CallbackHandler	FAISS	LangFuse provides a CallbackHandler that automatically instruments chains.
26	Unit 2: Observability	Lec6	Prompt Management	Why should you manage prompts in a tool like LangFuse instead of hardcoding in Git?	Easy	1	A	To allow non-engineers to tweak them	Because Git is too slow	Because Git charges per token	To hide prompts from developers	It acts as a CMS for prompts so non-engineers can comfortably inspect and tweak them.
27	Unit 2: Observability	Lec6	Setup	How can you enable LangSmith auto-tracing in a LangChain project usually?	Easy	1	D	Rewrite all code to use LangSmith classes	Contact support to enable it	Import enable_smith module	Just set environment variables	LangSmith is magic; you often don't need code changes, just environment variables.
28	Unit 2: Observability	Lec6	Production Best Practices	What is the recommended tracing sampling rate for Production environments?	Easy	1	C	100%	50%	1-5% of traffic	None	In production, tracing every request is noisy and expensive, so 1-5% or high importance traces are recommended.
29	Unit 2: Observability	Lec6	Privacy	How handle PII Data Privacy before logging to a cloud observability tool?	Easy	1	B	Do nothing	Run PII Masking/Redaction functions	Encrypt with simple base64	Delete all logs	Never log sensitive data; run PII Masking or use enterprise redacting features.
30	Unit 2: Observability	Lec6	Alerts	What is an example of a good alert to set up in observability?	Easy	1	A	Error Rate Spike > 10% in 5 min	"Hello World" printed	CPU temperature	Single user logged out	You should alert on things like Error Rate > 10%, Latency Spikes, or Cost Anomalies.
31	Unit 2: Observability	Lec6	LangFuse vs LangSmith	If self-hosting data privacy is an absolute requirement and budget is zero, which tool is recommended?	Medium	2	C	Weights & Biases	LangSmith	LangFuse	CloudWatch	LangFuse is Open Source (MIT) and offers easy self-hosting (Docker Compose) for free.
32	Unit 2: Observability	Lec6	LangSmith Playground	What is the "Playground: Edit and Re-run" feature in LangSmith useful for?	Medium	2	A	You can take a failed production trace, change the prompt, and test a fix immediately	Training new models	Deploying code to AWS	Chatting with other developers	It allows you to take failed real-world traces and edit prompts/parameters to instantly see if the issue resolves.
33	Unit 2: Observability	Lec6	Latency Debugging	If a RAG request takes 10 seconds, how does tracing help?	Medium	2	B	It makes the query faster	It breaks down the latency per component (e.g., Vector DB vs API completion)	It charges the user for the wait time	It cancels requests longer than 5 seconds	Tracing visualizes the execution flow, pinpointing exactly which step (Vector Search vs Generate) is the bottleneck.
34	Unit 2: Observability	Lec6	Cost Tracking	Why is Cost Tracking a critical feature in LLM Observability compared to traditional app monitoring?	Medium	2	D	Because AWS charges are cheap	Because you don't need servers	Because LLMs don't cost real money	Because LLM API calls are charged per-token and single runaway loops can cost hundreds of dollars quickly	API calls are expensive, requiring real-time tracking to prevent unmanaged financial overruns.
35	Unit 2: Observability	Lec6	Langchain Integration	What environment variable activates LangSmith tracing?	Medium	2	B	LANGCHAIN_DEBUG=1	LANGCHAIN_TRACING_V2=true	LANGCHAIN_LOG=all	LANGSMITH_ACTIVE=1	export LANGCHAIN_TRACING_V2=true activates LangSmith native tracing.
36	Unit 2: Observability	Lec6	Prompt CMS	How do you fetch a production prompt dynamically using LangFuse SDK?	Medium	2	A	Using langfuse.get_prompt(name, version)	Reading from a local .json file	Executing a GraphQL query to Github	Using prompt = os.getenv('PROMPT')	Langfuse acts as a CMS and lets you retrieve prompts using get_prompt("name", version="production").
37	Unit 2: Observability	Lec6	Alerts & Best Practices	Why shouldn't you just "stare at dashboards" for production LLM apps?	Medium	2	A	You need automated alerts (error spikes, costs) to respond fast to anomalies	Dashboards are always broken	It slows down the computer	Observability doesn't provide dashboards	Dashboards are passive. Automated alerts are needed to actively manage sudden cost, latency, or error anomalies.
38	Unit 2: Observability	Lec6	Advanced LangChain Integration	You have a complex application utilizing standard Python code, LangChain agent loops, and custom API calls. Should you prefer LangSmith or LangFuse, and why?	Hard	3	B	LangSmith, because it supports Python natively better	LangFuse, because it is platform-agnostic and instruments cleanly across non-LangChain code too.	LangSmith, because LangChain is mandatory.	LangFuse, because it has an "Edit and Re-run" playground.	LangFuse is platform-agnostic for non-LangChain code, making it better for mixed-stack integrations, while LangSmith is highly specific and native to LangChain execution loops.
39	Unit 2: Observability	Lec6	Debugging Scenarios	In production, users report the chatbot occasionally ignores their negative feedback instructions. How would you leverage LangSmith to resolve this?	Hard	3	C	By deleting the user history and trying again	Check the VectorDB logs	Locate the failed traces in LangSmith, transition them to the Playground, adjust the system prompt, and replay to verify compliance	Re-index the FAISS database	LangSmith's Playground allows you to take directly failed traces, manipulate the prompt, and replay the exact trace environment to find the fix.
40	Unit 2: Observability	Lec6	Data Security Architecture	Explain a robust architectural design for handling HIPAA/PII compliance while using a SaaS LLM Observability platform like LangSmith Enterprise.	Hard	3	A	Run an edge/middleware service that performs localized PII Entity masking/redaction before transmitting traces to the LangSmith API	Avoid observability tools completely	Share passwords directly via the agent	Mask PII inside the LangSmith GUI	PII must not leave the secure perimeter; redaction must happen at the application layer or middleware before data is shipped via logs/traces.
41	Unit 3: Architecture	Lec10	RAG Architectures	What is the main characteristic of Naive RAG?	Easy	1	A	Relies purely on semantic vector similarity search	Combines Graph and Vector	Uses query transformation	Extracts relationships	Naive RAG relies purely on semantic similarity and simple Top-K lookup.
42	Unit 3: Architecture	Lec10	RAG Architectures	Which element separates Advanced RAG from Naive RAG?	Easy	1	B	Connecting to APIs	Hybrid search (Vector + BM25) and Reranking	Using GPT-4	Splitting text	Advanced uses hybrid search, query transformations like HyDE, and reranking.
43	Unit 3: Architecture	Lec10	RAG Architectures	What structure does GraphRAG utilize?	Easy	1	C	Relational SQL tables	Simple vector arrays	Knowledge graph + vectors with nodes and edges	Document JSON trees	GraphRAG utilizes knowledge graphs, treating entities as nodes and relationships as edges.
44	Unit 3: Architecture	Lec10	RAG Architectures	What is a major disadvantage of Naive RAG?	Easy	1	D	High cost	Extremely slow	Too complex	No relationships mapping, treats chunks as islands	Simple RAG ignores relationships, treating every text chunk as an isolated island.
45	Unit 3: Architecture	Lec10	RAG Architectures	What is semantic chunking?	Easy	1	B	Cutting text by character limit	Splitting text based on meaning breaks	Chunking by number of words	Randomly breaking paragraphs	Semantic chunking splits text based on meaning boundaries rather than arbitrary character counts.
46	Unit 3: Architecture	Lec10	RAG Architectures	In Advanced RAG, what does the "Cross-encoder reranking" do?	Easy	1	A	It re-scores retrieved results to maximize true relevance	It generates the final answer	It translates the query	It graphs the entities	Reranking is a heavy, accurate scoring model that filters noise from initial retrieval results.
47	Unit 3: Architecture	Lec10	RAG Architectures	What is a "multi-hop" query?	Easy	1	B	A query traversing multiple servers	A question that requires connecting facts across different documents	A fast query	A query generating 5 hops	It requires the system to traverse A -> B -> C across different chunks/nodes to answer.
48	Unit 3: Architecture	Lec10	RAG Architectures	What graph database is typically highlighted for GraphRAG use cases?	Easy	1	C	PostgreSQL	MongoDB	Neo4j	Redis	Neo4j is a popular graph database for exploring nodes and edges.
49	Unit 3: Architecture	Lec10	RAG Architectures	Which system combines Vector + Graph + Routing?	Easy	1	D	Naive	Advanced	GraphRAG	Hybrid System	The Hybrid System attempts to synthesize both graph and advanced vector search via routing.
50	Unit 3: Architecture	Lec10	RAG Architectures	To execute an experimental comparison, what data split was used?	Easy	1	A	20 Train, 180 Test	100 Train, 100 Test	50 Train, 150 Test	No split	The dataset was strictly split into 20 questions for Train (prompt tuning) and 180 for Test.
51	Unit 3: Architecture	Lec10	RAG Architectures	Explain the concept of HyDE in Advanced RAG.	Medium	2	B	Hybrid Data Execution	Hypothetical Document Embeddings: Generating a hypothetical answer to search against	High Yield Document Extraction	Hierarchical Dynamic Extraction	Query transformation rewrites ambiguous queries or generates hypothetical answers to improve semantic matching.
52	Unit 3: Architecture	Lec10	RAG Architectures	Why does GraphRAG perform exceptionally well on Relational and Multi-Hop Questions compared to Naive RAG?	Medium	2	C	It uses a larger Embedding Model	It stores more documents	It has explicit defined relationships separating entities, letting it traverse A->B->C seamlessly	It generates shorter answers	GraphRAG explicitly builds "knows how things are connected" structures, preventing loss of context.
53	Unit 3: Architecture	Lec10	RAG Architectures	Name the primary trade-off encountered when building a Hybrid RAG system.	Medium	2	D	Lower overall quality	Reduced contextual awareness	Slower vector lookups	Highest complexity, highest cost, and immense maintenance burden	While offering peak recall, it combines all pipelines, meaning severe engineering complexity and top cost.
54	Unit 3: Architecture	Lec10	RAG Architectures	In evaluation design, why is "Number of API calls" considered an important metric?	Medium	2	A	It is a direct measure of system complexity, cost, and risk of failure for the architecture	It measures token length	It tells us if OpenAI is down	It determines the embedding algorithm	A complex agent/RAG design with many API calls risks cascading failures and multiplies costs quickly.
55	Unit 3: Architecture	Lec10	RAG Architectures	When should you purposefully choose Naive RAG over Advanced RAG?	Medium	2	C	When queries are highly relational	When budget is unlimited	When dealing with simple factual questions, low-latency requirements, and very small document datasets (< 50 docs)	When needing hybrid search	Naive RAG excels at being fast, cheap, and simple for small boundaries of facts where deep ranking isn't needed.
56	Unit 3: Architecture	Lec10	RAG Architectures	How does Community Detection (e.g. Leiden algorithm) assist GraphRAG?	Medium	2	B	It deletes old nodes	It groups related nodes into subgraphs, allowing the retrieval of entire themes or communities as context	It ranks vectors by similarity	It manages API rate limits	Community detection algorithms find clusters of related entities, helping abstract higher level context.
57	Unit 3: Architecture	Lec10	RAG Architectures	In a Hybrid system, how does "Adaptive Routing" theoretically work to improve efficiency?	Hard	3	A	By using a classifier LLM/model to inspect the incoming query to decide whether Graph traversal or Vector search is necessary before executing	By sending all queries to all databases	By routing requests to cheaper cloud regions	By storing data in multiple places	An intelligent routing step analyzes the query intent (e.g., Relational vs Factual) and directs traffic to the most efficient specific sub-pipeline to balance cost vs quality.
58	Unit 3: Architecture	Lec10	RAG Architectures	Critically analyze the phrase "Entity extraction quality: Garbage in, garbage out" inside the context of GraphRAG setup.	Hard	3	D	Entity extraction is cheap so it doesn't matter	Vector DBs handle bad entities naturally	It means Neo4j databases corrupt easily	If the LLM extracting (Subject, Predicate, Object) fails to identify correct ontological boundaries initially, the resulting Graph structure will be noisy and useless for retrieval	The entire Graph relies on the accuracy of the foundational knowledge extraction step. Bad extraction leads to bad graphs.
59	Unit 3: Architecture	Lec10	RAG Architectures	Assume a system integrates BM25 and Dense Vectors. Describe a mathematically robust way to merge these distinct ranked result lists.	Hard	3	A	Reciprocal Rank Fusion (RRF)	Just concatenate them	Only use the Dense Vector results	Average the raw similarity scores directly	RRF is critical because BM25 scores (sparse) and Cosine Similarity (dense) exist on different scales. RRF merges based on positional ranks.
60	Unit 3: Architecture	Lec10	RAG Architectures	Synthesize a final recommendation: You're launching an enterprise startup for legal documents covering 1M+ PDF pages. You have a mid-range budget but need flawless Relational analysis. What's the pragmatic technical roadmap?	Hard	3	C	Start Naive RAG -> Scale to Naive	Build a massive Hybrid System on day one	Start with Advanced RAG as default for stability and search quality, then phase in GraphRAG specifically for mapping explicitly defined legal contract relationships	Use GraphRAG for everything	The text suggests Advanced RAG as the definitive pragmatic baseline, while GraphRAG is specialized for highly relational data. Building hybrid early is too risky and costly.