Deploying Retrieval-Augmented Generation (RAG) systems without proper evaluation will lead to unpredictable outcomes and missed opportunities for improvement.

RAG systems gained popularity due to their ease of construction and intuitive design. However, this accessibility has also led to instances where engineers without a data science background oversee these systems. Unlike traditional software, where testing is ingrained in the development process, evaluating AI models requires specialized knowledge that not all engineers possess. This gap can result in models that behave unpredictably, making it difficult to identify both their strengths and areas for enhancement.

Evaluating RAG systems is particularly challenging due to the complexity of text-based inputs and generated outputs. Unlike straightforward tasks like classification, where evaluation is typically more straightforward, RAG systems demand a more nuanced approach.

Here's how we approached the evaluation of RAG systems in a recent project:

1. Harmful and Irrelevant Questions

We created a dataset of harmful (e.g., jailbreaking) and irrelevant questions. By prompting the language model with these questions and comparing its responses to expected default answers, we could compute classification metrics, ensuring the model handled such queries appropriately.

2. Retrieval Path Analysis

The retrieval component of RAG systems functions similarly to search engines, a well-established field of evaluation. We generated a dataset of questions with an expected context list and used Information Retrieval (IR) metrics like NDCG and precision@n to assess the effectiveness of the retrieval process.

3. End-to-End Testing

Evaluating the complex text output of RAG systems isn't straightforward with deterministic metrics like Regex or Levenshtein distance, as correct answers can be phrased differently. To address this, we employed model-graded metrics, also known as LLM-as-a-judge. By prompting a second language model with the context, expected answer, and generated answer, we could assess the output using metrics like context-recall, context-relevance, context-faithfulness, factuality, and question-relevance.

What’s your experience with RAG based system evaluation?

#RAGSystems #LLMEvaluation #AIDeployment