Ensuring Semantic Precision With Minimum Score

Learn how to use min_score with semantic text and hybrid search to improve precision. Please see blog post for further context.

Requirements

For this example, you will need:

• An Elastic deployment:

  • We'll be using Elastic Cloud for this example (available with a free trial)

• Elasticsearch 9.1 or above, or Elasticsearch serverless

Create Elastic Cloud deployment

If you don't have an Elastic Cloud deployment, sign up here for a free trial.

Install packages and connect with Elasticsearch Client

To get started, we'll need to connect to our Elastic deployment using the Python client (version 8.15.0 or above). Because we're using an Elastic Cloud deployment, we'll use the Cloud ID to identify our deployment.

First we need to pip install the following packages:

• elasticsearch

Next, we need to import the modules we need.

Now we can instantiate the Python Elasticsearch client.

First we prompt the user for their password and Cloud ID. Then we create a client object that instantiates an instance of the Elasticsearch class. 🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory.

Test the Client

Before you continue, confirm that the client has connected with this test.

Refer to the documentation to learn how to connect to a self-managed deployment.

Read this page to learn how to connect using API keys.

Create Inference Service

To allow for fast inference we will use the inference service.

Create the Index

Now we need to create the movie index. Note that this code deletes any existing index with the same name.

Notice how we configured the mappings. We defined overview_vector as a semantic_text field. The inference_id parameter defines the inference endpoint that is used to generate the embeddings for the field. Then we configured the overview field to copy its value to the overview_vector field.

While copy_to is not required to use semantic_text, it enables use cases like hybrid search where semantic and lexical techniques are used together. We will cover a hybrid search example later in this notebook.

Populate the Index

Let's populate the index with our example dataset

Colab - Download Data

If running the notebook in Colab, as opposed to checking out or downloading the elasticsearch-labs repository, we need to download the data to the colab environment

Run data import script

This will take a little while, probably around 15 minutes...

Semantic Search

Now that our index is populated, we can query it using semantic search.

Aside: Pretty printing Elasticsearch search results

Your search API calls will return hard-to-read nested JSON. We'll create a little function called pretty_search_response to return nice, human-readable outputs from our examples.

Semantic Search with the semantic Query

We can use the semantic query to quickly & easily query the semantic_text field in our index. Under the hood, an embedding is automatically generated for our query text using the semantic_text field's inference endpoint. Notice at the bottom of search results, some movies are not really super hero movies, such as Beethoven's 3rd

We will now move on to increase the precision by introducing minimum score.

Normalising and Enforcing Minimum Score

The below example adds the minimum score parameter to reduce irrelevant results. It utilises the minmax normalizer to get a nice score distribution of 0-1, which also helps in setting an appropriate score threshold. Feel free to experiment with different thresholds

These results demonstate how to cut-off the long tail of irrelavant results. Lets build on the semantic search retriever with a couple of hybrid search examples

Hybrid Search Using Linear Retriever

The below hybrid search request uses linear retriever with weights to control the importance of the semantic vs lexical search. Note that weights add up to 1, which means the total score will remain within the 0-1 interval. This will again make it easier to set and understand the minimum threshold.

Note that in this case the minimum score is applied to the total score, not just the semantic score.

Hybrid Search Using RRF

The below search uses RRF and applies the minimum score only to the semantic score. This allows us to control the semantic and lexical precision individually. To control lexical precision, we often use the minimum_should_match parameter, as in this example.

Conclusion

We have shown how the minimum score parameter can be used to improve precision in search results, especially for semantic search. Feel free to experiment with different queries and min_score settings.