Hybrid Search using RRF

In this example we'll use the reciprocal rank fusion algorithm to combine the results of BM25 and kNN semantic search. We'll use the same dataset we used in our quickstart guide.

You can use RRF for hybrid search out of the box, without any additional configuration. This example demonstrates how RRF ranking works at a basic level.

Install packages and initialize the Elasticsearch Python client

To get started, we'll need to connect to our Elastic deployment using the Python client. Because we're using an Elastic Cloud deployment, we'll use the Cloud ID to identify our deployment.

First we need to pip install the packages we need for this example.

Next we need to import the elasticsearch module and the getpass module. getpass is part of the Python standard library and is used to securely prompt for credentials.

Now we can instantiate the Python Elasticsearch client. First we prompt the user for their password and Cloud ID.

🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory.

Then we create a client object that instantiates an instance of the Elasticsearch class.

Enable Telemetry

Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See telemetry.py for details. Thank you!

Test the Client

Before you continue, confirm that the client has connected with this test.

Refer to https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect to a self-managed deployment.

Read https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect using API keys.

Pretty printing Elasticsearch responses

Let's add a helper function to print Elasticsearch responses in a readable format. This function is similar to the one that was used in the quickstart guide.

Querying Documents with Hybrid Search

🔐 NOTE: Before you can run the query in this section, you need the book_index dataset from our quick start. If you haven't worked through the quick start, please follow the steps described there to create an Elasticsearch deployment with the dataset in it, and then come back to run the query here.

Now we need to perform a query using two different search strategies:

• Semantic search using the "all-MiniLM-L6-v2" embedding model
• Keyword search using the "title" field

We then use Reciprocal Rank Fusion (RRF) to balance the scores to provide a final list of documents, ranked in order of relevance. RRF is a ranking algorithm for combining results from different information retrieval strategies.

Note: With the retriever API, _score contains the document’s relevance score, and the rank is simply the position in the results (first result is rank 1, etc.).