Notebooks
M
Milvus Evaluation With Ragas

Evaluation With Ragas

image-searchvector-databasesemantic-searchIntegrationmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonragNLP
alph-notebooks/milvus-bootcamp / evaluation_with_ragas.ipynb

Evaluation with Ragas

This guide demonstrates how to use Ragas to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon Milvus.

The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.

Ragas is a framework that helps you evaluate your RAG pipelines. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.

Prerequisites

Before running this notebook, make sure you have the following dependencies installed:

[ ]

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

We will use OpenAI as the LLM in this example. You should prepare the api key OPENAI_API_KEY as an environment variable.

[ ]

Define the RAG pipeline

We will define the RAG class that use Milvus as the vector store, and OpenAI as the LLM. The class contains the load method, which loads the text data into Milvus, the retrieve method, which retrieves the most similar text data to the given question, and the answer method, which answers the given question with the retrieved knowledge.

[1]

Let's initialize the RAG class with OpenAI and Milvus clients.

[2]

As for the argument of MilvusClient:

  • Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
  • If you have large scale of data, you can set up a more performant Milvus server on docker or kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.
  • If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.

Run the RAG pipeline and get results

We use the Milvus development guide to be as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.

Download it and load it into the rag pipeline.

[3]
Creating embeddings: 100%|██████████| 27/27 [00:20<00:00,  1.34it/s]

Let's define a query question about the content of the development guide documentation. And then use the answer method to get the answer and the retrieved context texts.

[4]
('The hardware requirements specification for building Milvus and running it from source code is as follows:\n\n- 8GB of RAM\n- 50GB of free disk space',
, ['Hardware Requirements\n\nThe following specification (either physical or virtual machine resources) is recommended for Milvus to build and run from source code.\n\n```yaml\n- 8GB of RAM\n- 50GB of free disk space\n```\n\n##',
,  'Building Milvus on a local OS/shell environment\n\nThe details below outline the hardware and software requirements for building on Linux and MacOS.\n\n##',
,  "Software Requirements\n\nAll Linux distributions are available for Milvus development. However a majority of our contributor worked with Ubuntu or CentOS systems, with a small portion of Mac (both x86_64 and Apple Silicon) contributors. If you would like Milvus to build and run on other distributions, you are more than welcome to file an issue and contribute!\n\nHere's a list of verified OS types where Milvus can successfully build and run:\n\n- Debian/Ubuntu\n- Amazon Linux\n- MacOS (x86_64)\n- MacOS (Apple Silicon)\n\n##"])

Now let's prepare some questions with its corresponding ground truth answers. We get answers and contexts from our RAG pipeline.

[8]
Answering questions: 100%|██████████| 3/3 [00:04<00:00,  1.37s/it]

Evaluation with Ragas

We use Ragas to evaluate the performance of our RAG pipeline results.

Ragas provides a set of metrics that is easy to use. We take Answer relevancy, Faithfulness, Context recall, and Context precision as the metrics to evaluate our RAG pipeline. For more information about the metrics, please refer to the Ragas Metrics.

[7]
Evaluating: 100%|██████████| 12/12 [00:10<00:00,  1.11it/s]

{'answer_relevancy': 0.9894, 'faithfulness': 1.0000, 'context_recall': 1.0000, 'context_precision': 1.0000}