Semantic Search with Semantic Text

Learn how to use the semantic_text field type to quickly get started with semantic search.

Requirements

For this example, you will need:

• An Elastic deployment:

  • We'll be using Elastic Cloud for this example (available with a free trial)

• Elasticsearch 9.1 or above, or Elasticsearch serverless

Create Elastic Cloud deployment

If you don't have an Elastic Cloud deployment, sign up here for a free trial.

Install packages and connect with Elasticsearch Client

To get started, we'll need to connect to our Elastic deployment using the Python client (version 8.15.0 or above). Because we're using an Elastic Cloud deployment, we'll use the Cloud ID to identify our deployment.

First we need to pip install the following packages:

• elasticsearch

Next, we need to import the modules we need.

🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory.

Now we can instantiate the Python Elasticsearch client.

First we prompt the user for their password and Cloud ID. Then we create a client object that instantiates an instance of the Elasticsearch class.

Enable Telemetry

Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See telemetry.py for details. Thank you!

Test the Client

Before you continue, confirm that the client has connected with this test.

Refer to the documentation to learn how to connect to a self-managed deployment.

Read this page to learn how to connect using API keys.

Create the Index

Now we need to create an index with a semantic_text field. Let's create one that enables us to perform semantic search on movie plots. We use the preconfigured .elser-2-elastic inference endpoint which uses the ELSER model via the Elastic Inference Service.

Notice how we configured the mappings. We defined plot_semantic as a semantic_text field. The inference_id parameter defines the inference endpoint that is used to generate the embeddings for the field. Then we configured the plot field to copy its value to the plot_semantic field.

While copy_to is not required to use semantic_text, it enables use cases like hybrid search where semantic and lexical techniques are used together. We will cover a hybrid search example later in this notebook.

Populate the Index

Let's populate the index with our example dataset of 12 movies.

Semantic Search

Now that our index is populated, we can query it using semantic search.

Aside: Pretty printing Elasticsearch search results

Your search API calls will return hard-to-read nested JSON. We'll create a little function called pretty_search_response to return nice, human-readable outputs from our examples.

Semantic Search with the semantic Query

We can use the semantic query to quickly & easily query the semantic_text field in our index. Under the hood, an embedding is automatically generated for our query text using the semantic_text field's inference endpoint.

These results demonstrate the power of semantic search. Our top results are all movies involving organized crime, even if the exact term "organized crime" doesn't appear in the plot description. This works because the ELSER model understands the semantic similarity between terms like "organized crime" and "mob".

However, these results also show the weaknesses of semantic search. Because semantic search is based on vector similarity, there is a long tail of results that are weakly related to our query vector. That's why movies like The Matrix are returned towards the tail end of our search results.

Hybrid Search with the semantic Query

We can address some of the issues with pure semantic search by combining it with lexical search techniques. Here, we use a boolean query to require that all matches contain at least term from the query text, in either the plot or genre fields.

These results demonstrate that the application of lexical search techniques can help focus the results, while retaining many of the advantages of semantic search. In this example, the top search results are all still movies involving organized crime, but the multi_match query keeps the long tail shorter and focused on movies in the crime genre.

The copy_to parameter we defined in the mapping enables this query pattern. It ensures that the content provided for the plot field is indexed both lexically and semantically.

Note the boost parameters applied to the multi_match and semantic queries. Combining lexical and semantic search techniques in a boolean query like this is called "linear combination" and when doing this, it is important to normalize the scores of the component queries. This involves consideration of a few factors, including:

• The range of scores generated by the query
• The relative importance and accuracy of the query in the context of the dataset

In this example, the multi_match query is mostly used as a filter to constrain the search results' long tail, so we assign it a lower boost than the semantic query.

Conclusion

The semantic_text field type is a powerful tool that can help you quickly and easily integrate semantic search. It can greatly improve the relevancy of your search results, particularly when combined with lexical search techniques.