Notebooks
M
Milvus 1. RAG Basic

1. RAG Basic

image-searchvector-databasesemantic-searchmilvusWorkshopsembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonbootcampdbta_may_2024ragNLP
alph-notebooks/milvus-bootcamp / 1. RAG_basic.ipynb

ReadtheDocs Retrieval Augmented Generation (RAG)

In this notebook, we are going to use Milvus documentation pages to create a chatbot about our product. The chatbot is going to follow RAG steps to retrieve chunks of data using Semantic Vector Search, then the Question + Context will be fed as a Prompt to a LLM to generate an answer.

Many RAG demos use OpenAI for the Embedding Model and ChatGPT for the Generative AI model. In this notebook, we will demo a fully open source RAG stack.

Using open-source Q&A with retrieval saves money since we make free calls to our own data almost all the time - retrieval, evaluation, and development iterations.

Let's get started!

[1]
[2]
[3]

Download Data

The data used in this notebook is Milvus documentation web pages.

The code block below downloads all the web pages into a local directory called rtdocs.

I've already uploaded the rtdocs data folder to github, so you should see it if you cloned my repo.

[4]
[5]
loaded 22 documents
Why Milvus Docs Tutorials Tools Blog Community Stars0 Try Managed Milvus FREE Search Home v2.4.x About Milvus Get StartedPrerequisitesInstall MilvusInstall SDKsQuickstart Concepts User Guide Models Administration Guide Tools Integrations Example Applications FAQs API reference Quickstart This guide explains how to connect to your Milvus cluster and performs CRUD operations in minutes Before you start You have installed Milvus standalone or Milvus cluster. You have installed preferred SDKs. You c
{'source': 'https://milvus.io/docs/quickstart.md'}

Load the Embedding Model checkpoint and use it to create vector embeddings

What are Embeddings?

Check out this blog for an introduction to embeddings.

An excellent place to start is by selecting an embedding model from the HuggingFace MTEB Leaderboard, sorted descending by the "Retrieval Average'' column since this task is most relevant to RAG. Then, choose the smallest, highest-ranking embedding model. But, Beware!! some models listed are overfit to the training data, so they won't perform on your data as promised.

Milvus (and Zilliz) only supports tested embedding models that are not overfit!

Use open source Embedding Model from HuggingFace

[6]
/opt/miniconda3/envs/py311-unum/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(

model_name: BAAI/bge-large-en-v1.5
EMBEDDING_DIM: 1024
MAX_SEQ_LENGTH: 512

Create Milvus Collection and Index

[7]
[8]
Successfully dropped collection: `MilvusDocs`
Successfully created collection: `MilvusDocs`

Fixed-length Chunking using LangChain

Before embedding, it is necessary to decide your chunk strategy, chunk size, and chunk overlap. This section uses:

  • Strategy = Use markdown header hierarchies. Keep markdown sections together unless they are too long.
  • Chunk size = Use the embedding model's parameter MAX_SEQ_LENGTH
  • Overlap = Rule-of-thumb 10-15%
  • Function =
    • Langchain's RecursiveCharacterTextSplitter to split up long reviews recursively.

Notice below, each chunk is grounded with the document source page.

[9]
chunk_size: 512, chunk_overlap: 51.0
22 docs split into 355 child documents.
Output

Transform Chunks into Vectors

[10]
Embedding time for 355 chunks: 34.82 seconds
[11]
type embeddings: <class 'list'> of <class 'numpy.ndarray'>
of numbers: <class 'numpy.float32'>

Insert data into Milvus

For each original text chunk, we'll write the sextuplet (chunk, h1, h2, source, dense_vector, sparse_vector) into the database.

The Milvus Client wrapper can only handle loading data from a list of dictionaries.

Otherwise, in general, Milvus supports loading data from:

  • pandas dataframes
  • list of dictionaries

Below, we use the embedding model provided by HuggingFace, download its checkpoint, and run it locally as the encoder.

[12]
Start inserting entities
Milvus insert time for 355 vectors: 0.21 seconds

Define 4 questions

Search Milvus using PyMilvus API.

💡 By their nature, vector searches are "semantic" searches. For example, if you were to search for "leaky faucet":

Traditional Key-word Search - either or both words "leaky", "faucet" would have to match some text in order to return a web page or link text to the document.

Semantic search - results containing words "drippy" "taps" would be returned as well because these words mean the same thing even though they are different words.

[13]
example query length: 75
[14]

Execute a vector search

Search Milvus using PyMilvus API.

💡 By their nature, vector searches are "semantic" searches. For example, if you were to search for "leaky faucet":

Traditional Key-word Search - either or both words "leaky", "faucet" would have to match some text in order to return a web page or link text to the document.

Semantic search - results containing words "drippy" "taps" would be returned as well because these words mean the same thing even though they are different words.

[15]
output fields: ['chunk', 'source']
[16]
filter: 
Milvus Client search time for 355 vectors: 0.17427706718444824 seconds
type: <class 'list'>, count: 2
[17]
Retrieved result #1
distance = 0.7001987099647522
('Chunk text: layer, finds the node closest to the target in this layer, and '
 'then enters the next layer to begin another search. After multiple '
 'iterations, it can quickly approach the target position. In order to improve '
 'performance, HNSW limits the maximum degree of nodes on each layer of the '
 'graph to M. In addition, you can use efConstruction (when building index) or '
 'ef (when searching targets) to specify a search range. Index building '
 'parameters Parameter Description Range M M defines tha maximum number of '
 'outgoing')
source: https://milvus.io/docs/index.md

Retrieved result #2
distance = 0.6953287124633789
('Chunk text: this value can improve recall rate at the cost of increased '
 'search time. [1, 65535] 2 HNSW HNSW (Hierarchical Navigable Small World '
 'Graph) is a graph-based indexing algorithm. It builds a multi-layer '
 'navigation structure for an image according to certain rules. In this '
 'structure, the upper layers are more sparse and the distances between nodes '
 'are farther; the lower layers are denser and the distances between nodes are '
 'closer. The search starts from the uppermost layer, finds the node closest '
 'to the target')
source: https://milvus.io/docs/index.md

Get ready for G-Generation part of RAG

[18]
[19]
Length long text to summarize: 1021
sources: https://milvus.io/docs/index.md

Use an LLM to Generate a chat response to the user's question using the Retrieved Context.

Many different generative LLMs exist these days. Check out the lmsys leaderboard.

In this notebook, we'll try these LLMs:

  • The newly released open-source Llama 3 from Meta.
  • The cheapest, paid model from Anthropic Claude3 Haiku.
  • The standard in its price cateogory, gpt-3.5-turbo, from Openai.
[20]
Length prompt: 1432

Try Meta Llama 3 with Ollama to generate a human-like chat response to the user's question

Follow the instructions to install ollama and pull a model.
https://github.com/ollama/ollama

View details about which models are supported by ollama.
https://ollama.com/library/llama3

That page says ollama run llama3 will by default pull the latest "instruct" model, which is fine-tuned for chat/dialogue use cases.

The other kind of llama3 models are "pre-trained" base model.
Example: ollama run llama3:text ollama run llama3:70b-text

Format gguf means the model runs on CPU. gg = "Georgi Gerganov", creator of the C library model format ggml, which was recently changed to gguf.

Quantization (think of it like vector compaction) can lead to higher throughput at the expense of lower accuracy. For the curious, quantization meanings can be found on:
https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/tree/main.

Below just listing the main quantization types.

  • q4_0: Original quant method, 4-bit.
  • q4_k_m: Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K
  • q5_0: Higher accuracy, higher resource usage and slower inference.
  • q5_k_m: Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K
  • q 6_k: Uses Q8_K for all tensors
  • q8_0: Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.
[21]
MODEL:llama3.1:latest, FORMAT:gguf, PARAMETER_SIZE:8.0B, QUANTIZATION_LEVEL:Q4_0,
[22]
('The parameter "M" in HNSW limits the maximum degree of nodes on each layer, '
 'controlling recall rate and search time. A higher value can improve recall '
 'at a cost of increased search time.')
ollama_llama3_time: 10.65 seconds
[23]
('The parameters for HNSW (Hierarchical Navigable Small World Graph) are:  * '
 'M: The maximum degree of nodes on each layer of the graph, which can improve '
 'recall rate at the cost of increased search time. * efConstruction and ef: '
 'These parameters specify a search range when building or searching an index.')
llama3_octai_endpoints_time: 1.72 seconds
[24]
('The parameters for HNSW (Hierarchical Navigable Small World Graph) are:  * '
 'M: Maximum degree of nodes on each layer of the graph, which can be adjusted '
 'to balance recall rate and search time. * efConstruction (or ef): Search '
 'range when building the index, which can be used to control the trade-off '
 'between recall rate and search time.')
llama3_groq_endpoints_time: 0.54 seconds

Also try OpenAI

💡 Note: For use cases that need to always be factually grounded, use very low temperature values while more creative tasks can benefit from higher temperatures.

[25]
Length prompt: 1483
[26]
Question: What do the parameters for HNSW mean?
('Answer: The parameters for HNSW (Hierarchical Navigable Small World Graph) '
 'include M, which defines the maximum number of outgoing connections from '
 'each node in the graph, and efConstruction or ef, which specify the search '
 'range during index building and searching respectively. These parameters '
 'help control the trade-off between recall rate and search time.')


chatgpt_3.5_turbo_time: 1.57389
[27]
[28]
Author: Christy Bergman

Python implementation: CPython
Python version       : 3.11.8
IPython version      : 8.22.2

unstructured: 0.14.4
torch       : 2.3.0
pymilvus    : 2.4.4
langchain   : 0.2.6
ollama      : 0.1.8
groq        : 0.8.0
octoai      : 1.0.2
openai      : 1.35.0

conda environment: py311-unum