Hybrid Search with Milvus and LangChain

Hybrid search combines the strengths of different search paradigms to enhance retrieval accuracy and robustness. It leverages the capabilities of both dense vector search and sparse vector search, as well as combinations of multiple dense vector search strategies, ensuring comprehensive and precise retrieval for diverse queries.

This diagram illustrates the most common hybrid search scenario, which is the dense + sparse hybrid search. In this case, candidates are retrieved using both semantic vector similarity and precise keyword matching. Results from these methods are merged, reranked, and passed to an LLM to generate the final answer. This approach balances precision and semantic understanding, making it highly effective for diverse query scenarios.

In addition to dense + sparse hybrid search, hybrid strategies can also combine multiple dense vector models. For instance, one dense vector model might specialize in capturing semantic nuances, while another focuses on contextual embeddings or domain-specific representations. By merging results from these models and reranking them, this type of hybrid search ensures a more nuanced and context-aware retrieval process.

LangChain Milvus integration provides a flexible way to implement hybrid search, it supports any number of vector fields, and any custom dense or sparse embedding models, which allows LangChain Milvus to flexibly adapt to various hybrid search usage scenarios, and at the same time compatible with other capabilities of LangChain.

In this tutorial, we will start with the most common dense + sparse case, and then introduce any number of general hybrid search usage approachs.

Note: The MilvusCollectionHybridSearchRetriever, which is another implementation of hybrid search with Milvus and LangChain, is about to be deprecated. Please use the approach in this document to implement hybrid search because it is more flexible and compatible with LangChain.

Prerequisites

Before running this notebook, make sure you have the following dependencies installed:

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

We will use the models from OpenAI. You should prepare the environment variables OPENAI_API_KEY from OpenAI.

Specify your Milvus server URI (and optionally the TOKEN). For how to install and start the Milvus server following this guide.

Prepare some example documents, which are fictional story summaries categorized by theme or genre.

Dense embedding + Sparse embedding

Option 1(Recommended): dense embedding + Milvus BM25 built-in function

Use dense embedding + Milvus BM25 built-in function to assemble the hybrid retrieval vector store instance.

When you use BM25BuiltInFunction, please note that full-text search is currently available in Milvus Standalone, Milvus Distributed, and Zilliz Cloud, though not yet supported in Milvus Lite (which has this feature planned for future implementation). Reach out support@zilliz.com for more information.

In the code above, we define an instance of BM25BuiltInFunction and pass it to the Milvus object. BM25BuiltInFunction is a lightweight wrapper class for Function in Milvus. We can use it with OpenAIEmbeddings to initialize a dense + sparse hybrid search Milvus vector store instance.

BM25BuiltInFunction does not require the client to pass corpus or training, all are automatically processed at the Milvus server's end, so users do not need to care about any vocabulary and corpus. In addition, users can also customize the analyzer to implement the custom text processing in the BM25.

For more information about BM25BuiltInFunction, please refer to the Full-Text-Search and Using Full-Text Search with LangChain and Milvus.

Option 2: Use dense and customized LangChain sparse embedding

You can inherit the class BaseSparseEmbedding from langchain_milvus.utils.sparse, and implement the embed_query and embed_documents methods to customize the sparse embedding process. This allows you to customize any sparse embedding method both based on term frequency statistics(e.g. BM25) or neural networks(e.g. SPADE).

Here is an example:

We have a demo class BM25SparseEmbedding inherited from BaseSparseEmbedding in langchain_milvus.utils.sparse. You can pass it into the initialization embedding list of the Milvus vector store instance just like other langchain dense embedding classes.

Although this is a way to use BM25, it requires users to manage the corpus for term frequency statistics. We recommend using the BM25 built-in function(Option 1) instead, as it handles everything on the Milvus server side. This eliminates the need for users to concern about managing the corpus or training a vocabulary. For more information, please refer to the Using Full-Text Search with LangChain and Milvus.

Define multiple arbitrary vector fields

When initializing the Milvus vector store, you can pass in the list of embeddings (and will also list of build-in functions in the future) to implement multi-ways retrival, and then rerank these candidates. Here is an example:

In this example, we have three vector fields. Among them, sparse is used as the output field for BM25BuiltInFunction, while the other two, dense1 and dense2, are automatically assigned as the output fields for the two OpenAIEmbeddings models (based on the order).

Specify the index params for multi-vector fields

By default, the index types of each vector field will be automatically determined by the type of embedding or built-in function. However, you can also specify the index type for each vector field to optimize the search performance.

Please keep the order of list of index params consistent with the order of vectorstore.vector_fields to avoid confusion.

Rerank the candidates

After the first stage of retrieval, we need to rerank the candidates to get a better result. You can choose WeightedRanker or RRFRanker depending on your requirements. You can refer to the Reranking for more information.

Here is an example for weighted reranking:

Here is an example of RRF reranking:

If you don't pass any parameters about rerank, the average weighted rerank strategy is used by default.

Using Hybrid Search and Reranking in RAG

In the scenario of RAG, the most prevalent approach for hybrid search is dense + sparse retrieval, followed by reranking. The subsequent example demonstrates a straightforward end-to-end code.

Prepare the data

We use the Langchain WebBaseLoader to load documents from web sources and split them into chunks using the RecursiveCharacterTextSplitter.

Load the document into Milvus vector store

As the introduction above, we initialize and load the prepared documents into Milvus vector store, which contains two vector fields: dense is for the OpenAI embedding and sparse is for the BM25 function.

Build RAG chain

We prepare the LLM instance and prompt, then conbine them into a RAG pipeline using the LangChain Expression Language.

Use the LCEL(LangChain Expression Language) to build a RAG chain.

Invoke the RAG chain with a specific question and retrieve the response

Congratulations! You have built a hybrid(dense vector + sparse bm25 function) search RAG chain powered by Milvus and LangChain.