Notebooks
M
Milvus Hyde With Langchain

Hyde With Langchain

image-searchvector-databasesemantic-searchmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonbootcampragadvanced_ragNLP
alph-notebooks/milvus-bootcamp / hyde_with_langchain.ipynb

Open In Colab

HyDE

Google Colab preparation[optional]

This is an optional step, if you want to run this notebook on Google Colab.

[ ]
[ ]
[ ]

Please prepare you OPENAI_API_KEY in your environment variables.

If you are running this notebook on Google Colab, you have to restart this session by Cmd/Ctrl + M, then press . to make the environment take effect.

[ ]

Get started

Prepare the data

We use the Langchain WebBaseLoader to load documents from blog sources and split them into chunks using the RecursiveCharacterTextSplitter.

[1]

Build the chain

We load the docs into milvus vectorstore, and build a milvus retriever.

[2]

Build the vanilla RAG chain.

[3]

Build a hyde chain.

[4]

Test the chain

[5]

[vanilla_result]:
The approximate nearest neighbors (ANN) algorithm is commonly used for vector approximate searching in a vector store. Some specific implementations of ANN algorithms include FAISS (Facebook AI Similarity Search), ScaNN (Scalable Nearest Neighbors), LSH (Locality-Sensitive Hashing), and ANNOY (Approximate Nearest Neighbors Oh Yeah).

[hyde_result]:
The vector approximate searching algorithms that work in a vector store include Locality-Sensitive Hashing (LSH), ANNOY (Approximate Nearest Neighbors Oh Yeah), FAISS (Facebook AI Similarity Search), ScaNN (Scalable Nearest Neighbors), and HNSW (Hierarchical Navigable Small World). These algorithms are commonly used for fast Maximum Inner Product Search (MIPS) in a vector store database.

In the [hyde_result], it retrieved with ground truth "HNSW", which does not appear in the [vanilla_result].

Let's dive deep into the retrieved results to find the reason.

[8]

[9]

The hyde_retriever retrieved the document with HNSW, while the vanilla_retriever did not. This is because HyDE generates fake documents that may contain various vector approximate searching algorithms like HNSW, which makes the retriever's results more accurate.