ReadtheDocs Retrieval Augmented Generation (RAG) using Zilliz Free Tier

In this notebook, we are going to use Milvus documentation pages to create a chatbot about our product. The chatbot is going to follow RAG steps to retrieve chunks of data using Semantic Vector Search, then the Question + Context will be fed as a Prompt to a LLM to generate an answer.

Many RAG demos use OpenAI for the Embedding Model and ChatGPT for the Generative AI model. In this notebook, we will demo a fully open source RAG stack.

Using open-source Q&A with retrieval saves money since we make free calls to our own data almost all the time - retrieval, evaluation, and development iterations. We only make a paid call to OpenAI once for the final chat generation step.

Let's get started!

Download Milvus documentation.

The data we’ll use is our own product documentation web pages. ReadTheDocs is an open-source free software documentation hosting platform, where documentation is written with the Sphinx document generator.

The code block below downloads the web pages into a local directory called rtdocs.

I've already uploaded the rtdocs data folder to github, so you should see it if you cloned my repo.

Start up Milvus running in local Docker (or Zilliz free tier)

⛔️ Make sure you pip install the correct version of pymilvus and server yml file. Versions (major and minor) should all match.

1. Install Docker
2. Start your Docker Desktop
3. Download the latest docker-compose.yml (or run the wget command, replacing version to what you are using)

wget https://github.com/milvus-io/milvus/releases/download/v2.4.0-rc.1/milvus-standalone-docker-compose.yml -O docker-compose.yml

4. From your terminal:
   • cd into directory where you saved the .yml file (usualy same dir as this notebook)
   • docker compose up -d
   • verify (either in terminal or on Docker Desktop) the containers are running
5. From your code (see notebook code below):
   • Import milvus
   • Connect to the local milvus server

Optionally, use Zilliz free tier cluster

To use fully-managed Milvus on Ziliz Cloud free trial.

1. Choose the default "Starter" option when you provision > Create collection > Give it a name > Create cluster and collection.
2. On the Cluster main page, copy your API Key and store it locally in a .env variable. See note below how to do that.
3. Also on the Cluster main page, copy the Public Endpoint URI.

💡 Note: To keep your tokens private, best practice is to use an env variable. See how to save api key in env variable.

👉🏼 In Jupyter, you need a .env file (in same dir as notebooks) containing lines like this:

• ZILLIZ_API_KEY=f370c...
• OPENAI_API_KEY=sk-H...
• VARIABLE_NAME=value...

Load the Embedding Model checkpoint and use it to create vector embeddings

What are Embeddings?

Check out this blog for an introduction to embeddings.

An excellent place to start is by selecting an embedding model from the HuggingFace MTEB Leaderboard, sorted descending by the "Retrieval Average'' column since this task is most relevant to RAG. Then, choose the smallest, highest-ranking embedding model. But, Beware!! some models listed are overfit to the training data, so they won't perform on your data as promised.

Milvus (and Zilliz) only supports tested embedding models that are not overfit.

In this notebook, we will use the open-source BGE-M3 which supports:

• over 100 languages
• context lengths of up to 8192
• multiple embedding inferences such as dense (semantic), sparse (lexical), and multi-vector Colbert reranking.

BGE-M3 holds the distinction of being the first embedding model to offer support for all three retrieval methods, achieving state-of-the-art performance on multi-lingual (MIRACL) and cross-lingual (MKQA) benchmark tests. Paper, HuggingFace

Milvus, the world's first Open Source Vector Database, plays a vital role in semantic search with scaleable, efficient storage and search for GenerativeAI workflows. Its advanced functionalities include metadata filtering and hybrid search. Since version 2.4, Milvus has built-in support for BGE M3.

Create a Milvus collection

You can think of a collection in Milvus like a "table" in SQL databases. The collection will contain the

• Schema (or no-schema Milvus client).
  💡 You'll need the vector EMBEDDING_DIM parameter from your embedding model. Typical values are:
  • 1024 for sbert embedding models
  • 1536 for ada-002 OpenAI embedding models
• Vector index for efficient vector search
• Vector distance metric for measuring nearest neighbor vectors
• Consistency level In Milvus, transactional consistency is possible; however, according to the CAP theorem, some latency must be sacrificed. 💡 Searching movie reviews is not mission-critical, so eventually consistent is fine here.

Add a Vector Index

The vector index determines the vector search algorithm used to find the closest vectors in your data to the query a user submits.

Most vector indexes use different sets of parameters depending on whether the database is:

• inserting vectors (creation mode) - vs -
• searching vectors (search mode)

Scroll down the docs page to see a table listing different vector indexes available on Milvus. For example:

• FLAT - deterministic exhaustive search
• IVF_FLAT or IVF_SQ8 - Hash index (stochastic approximate search)
• HNSW - Graph index (stochastic approximate search)
• AUTOINDEX - OSS or Zilliz cloud automatic index based on type of GPU, size of data.

Besides a search algorithm, we also need to specify a distance metric, that is, a definition of what is considered "close" in vector space. In the cell below, the HNSW search index is chosen. Its possible distance metrics are one of:

• L2 - L2-norm
• IP - Dot-product
• COSINE - Angular distance

💡 Most use cases work better with normalized embeddings, in which case L2 is useless (every vector has length=1) and IP and COSINE are the same. Only choose L2 if you plan to keep your embeddings unnormalized.

Chunking

Before embedding, it is necessary to decide your chunk strategy, chunk size, and chunk overlap. In this demo, I will use:

• Strategy = Use markdown header hierarchies. Keep markdown sections together unless they are too long.
• Chunk size = Use the embedding model's parameter MAX_SEQ_LENGTH
• Overlap = Rule-of-thumb 10-15%
• Function =
  • Langchain's HTMLHeaderTextSplitter to split markdown sections.
  • Langchain's RecursiveCharacterTextSplitter to split up long reviews recursively.

Notice below, each chunk is grounded with the document source page.
In addition, header titles are kept together with the chunk of markdown text.

Use the built-in Milvus BGE M3 embedding functions. The output will be 2 vectors:

• embeddings['dense'][i] is a list of numpy arrays, one per chunk. Milvus supports more than 1 dense embedding vector if desired, so i is the ith dense embedding vector.
• embeddings['sparse'][:, [i]] is a scipy sparse matrix where each column represents a chunk.

Insert data into Milvus

For each original text chunk, we'll write the sextuplet (chunk, h1, h2, source, dense_vector, sparse_vector) into the database.

The Milvus Client wrapper can only handle loading data from a list of dictionaries.

Otherwise, in general, Milvus supports loading data from:

• pandas dataframes
• list of dictionaries

Aside - example Milvus collection API calls

https://milvus.io/docs/manage-collections.md#View-Collections

Below are some common API calls for checking a collection.

• .num_entities, flushes data and executes row count.
• .describe_collection(), gives details about the schema, index, collection.
• .query(), gives back selected data from the collection.

Ask a question about your data

So far in this demo notebook:

1. Your custom data has been mapped into a vector embedding space
2. Those vector embeddings have been saved into a vector database

Next, you can ask a question about your custom data!

💡 In LLM vocabulary:

Query is the generic term for user questions.
A query is a list of multiple individual questions, up to maybe 1000 different questions!

Question usually refers to a single user question.
In our example below, the user question is "What is AUTOINDEX in Milvus Client?"

Semantic Search = very fast search of the entire knowledge base to find the TOP_K documentation chunks with the closest embeddings to the user's query.

💡 The same model should always be used for consistency for all the embeddings data and the query.

Execute a vector search

Search Milvus using PyMilvus API.

💡 By their nature, vector searches are "semantic" searches. For example, if you were to search for "leaky faucet":

Traditional Key-word Search - either or both words "leaky", "faucet" would have to match some text in order to return a web page or link text to the document.

Semantic search - results containing words "drippy" "taps" would be returned as well because these words mean the same thing even though they are different words.

Assemble and inspect the search result

The search result is in the variable results[0] consisting of top_k-count of objects of type 'pymilvus.client.abstract.Hits'

Use an LLM to Generate a chat response to the user's question using the Retrieved Context.

Many different generative LLMs exist these days. Check out the lmsys leaderboard.

In this notebook, we'll try these LLMs:

• The newly released open-source Llama 3 from Meta.
• The cheapest, paid model from Anthropic Claude3 Haiku.
• The standard in its price cateogory, gpt-3.5-turbo, from Openai.

Try Llama3 with Ollama to generate a human-like chat response to the user's question

Follow the instructions to install ollama and pull a model.
https://github.com/ollama/ollama

View details about which models are supported by ollama.
https://ollama.com/library/llama3

That page says ollama run llama3 will by default pull the latest "instruct" model, which is fine-tuned for chat/dialogue use cases.

The other kind of llama3 models are "pre-trained" base model.
Example: ollama run llama3:text ollama run llama3:70b-text

Format gguf means the model runs on CPU. gg = "Georgi Gerganov", creator of the C library model format ggml, which was recently changed to gguf.

Quantization (think of it like vector compaction) can lead to higher throughput at the expense of lower accuracy. For the curious, quantization meanings can be found on:
https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/tree/main.

Below just listing the main quantization types.

• q4_0: Original quant method, 4-bit.
• q4_k_m: Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K
• q5_0: Higher accuracy, higher resource usage and slower inference.
• q5_k_m: Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K
• q 6_k: Uses Q8_K for all tensors
• q8_0: Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.

Use Anthropic to generate a human-like chat response to the user's question

We've practiced retrieval for free on our own data using open-source LLMs.

Now let's make a call to the paid Claude3. List of models

• Opus - most expensive
• Sonnet
• Haiku - least expensive!

Prompt engineering tutorials

• Interactive
• Static

Try OpenAI to generate a human-like chat response to the user's question

We've practiced retrieval for free on our own data using open-source LLMs.

Now let's make a call to the paid OpenAI GPT.

💡 Note: For use cases that need to always be factually grounded, use very low temperature values while more creative tasks can benefit from higher temperatures.