Evaluation with DeepEval

This guide demonstrates how to use DeepEval to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon Milvus.

The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.

DeepEval is a framework that helps you evaluate your RAG pipelines. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where DeepEval comes in.

Prerequisites

Before running this notebook, make sure you have the following dependencies installed:

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

We will use OpenAI as the LLM in this example. You should prepare the api key OPENAI_API_KEY as an environment variable.

Define the RAG pipeline

We will define the RAG class that use Milvus as the vector store, and OpenAI as the LLM. The class contains the load method, which loads the text data into Milvus, the retrieve method, which retrieves the most similar text data to the given question, and the answer method, which answers the given question with the retrieved knowledge.

Let's initialize the RAG class with OpenAI and Milvus clients.

As for the argument of MilvusClient:

• Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
• If you have large scale of data, you can set up a more performant Milvus server on docker or kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.
• If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.

Run the RAG pipeline and get results

We use the Milvus development guide to be as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.

Download it and load it into the rag pipeline.

Let's define a query question about the content of the development guide documentation. And then use the answer method to get the answer and the retrieved context texts.

Now let's prepare some questions with its corresponding ground truth answers. We get answers and contexts from our RAG pipeline.

Evaluating Retriever

When evaluating a retriever in large language model (LLM) systems, it's crucial to assess the following:

1. Ranking Relevance: How effectively the retriever prioritizes relevant information over irrelevant data.

2. Contextual Retrieval: The ability to capture and retrieve contextually relevant information based on the input.

3. Balance: How well the retriever manages text chunk size and retrieval scope to minimize irrelevancies.

Together, these factors provide a comprehensive understanding of how the retriever prioritizes, captures, and presents the most useful information.

Evaluating Generation

To assess the quality of generated outputs in large language models (LLMs), it's important to focus on two key aspects:

1. Relevance: Evaluate whether the prompt effectively guides the LLM to generate helpful and contextually appropriate responses.

2. Faithfulness: Measure the accuracy of the output, ensuring the model produces information that is factually correct and free from hallucinations or contradictions. The generated content should align with the factual information provided in the retrieval context.

These factors together ensure that the outputs are both relevant and reliable.