Build a RAG with Milvus and Unstructured

Unstructured provides a platform and tools to ingest and process unstructured documents for Retrieval Augmented Generation (RAG) and model fine-tuning. It offers both a no-code UI platform and serverless API services, allowing users to process data on Unstructured-hosted compute resources.

In this tutorial, we will use Unstructured to ingest PDF documents and then use Milvus to build a RAG pipeline.

Preparation

Dependencies and Environment

Installation Options:

• For processing all document formats: pip install "unstructured[all-docs]"
• For specific formats (e.g., PDF): pip install "unstructured[pdf]"
• For more installation options, see the Unstructured documentation

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

We will use OpenAI as the LLM in this example. You should prepare the api key OPENAI_API_KEY as an environment variable.

Prepare Milvus and OpenAI clients

You can use the Milvus client to create a Milvus collection and insert data into it.

As for the argument of MilvusClient:

• Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
• If you have large scale of data, say more than a million vectors, you can set up a more performant Milvus server on Docker or Kubernetes. In this setup, please use the server address and port as your uri, e.g.http://localhost:19530. If you enable the authentication feature on Milvus, use "<your_username>:<your_password>" as the token, otherwise don't set the token.
• If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.

Check if the collection already exists and drop it if it does.

Prepare a OpenAI client to generate embeddings and generate responses.

Generate a test embedding and print its dimension and first few elements.

Create Milvus Collection

We will create a collection with the following schema:

• id: the primary key, which is a unique identifier for each document.
• vector: the embedding of the document.
• text: the text content of the document.
• metadata: the metadata of the document.

Then we build an AUTOINDEX index on the vector field. And then create the collection.

Load data from Unstructured

Unstructured provides a flexible and powerful ingestion pipeline to process various file types, including PDF, HTML, and more. We will partition and chunk a local PDF file. And then load the data into Milvus.

Let's examine the partitioned elements from the PDF file. Each element represents a chunk of content extracted by Unstructured's partitioning process.

Insert data into Milvus.

Retrieve and Generate Response

Define a function to retrieve relevant documents from Milvus.

Define a function to generate a response using the retrieved documents in the RAG pipeline.

Let's test the RAG pipeline with a sample question.