How to train and deploy Learning To Rank

In this notebook, we'll:

• Connect to an Elasticsearch deployment using the official Python client.
• Import and index a movie dataset into Elasticsearch.
• Extract features from our dataset using Elasticsearch's Query DSL, including custom script_score queries.
• Build a training dataset by combining extracted features with a human curated judgment list.
• Train a Learning To Rank model using XGBoost.
• Deploy the trained model to Elasticsearch using Eland.
• Use the model as a rescorer for second stage re-ranking.
• Evaluate the impact of the LTR model on search relevance, by comparing search results before and after applying the model.

NOTE:

• Learning To Rank is generally available for Elastic Stack versions 8.15.0 and newer and requires an Enterprise subscription or higher.

Install required packages

First we must install the packages we need for this notebook.

Configure your Elasticsearch deployment

For this example, we will be using an Elastic Cloud deployment (available with a free trial).

Enable Telemetry

Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See telemetry.py for details. Thank you!

Test the Client

Before you continue, confirm that the client has connected with this test.

Configure the dataset

We'll use a dataset derived from the MSRD (Movie Search Ranking Dataset).

The dataset is available here and contains the following files:

• movies_corpus.jsonl.gz: Movie dataset to be indexed.
• movies_judgements.tsv.gz: Judgment list of relevance judgments for a set of queries.
• movies_index_settings.json: Settings to be applied to the documents and index.

Import the document corpus

This step will import the documents of the corpus into the movies index .

Documents contains the following fields:

Loading the judgment list

The judgment list contains human evaluations that we'll use to train our Learning To Rank model.

Each row represents a query-document pair with an associated relevance grade and contains the following columns:

Note:

In this example the relevance grade is a binary value (relevant or not relavant). You could also use a number that represents the degree of relevance (e.g. from 0 to 4).

Configure feature extraction

Features are the inputs to our model. They represent information about the query alone, a result document alone, or a result document in the context of a query, such as BM25 scores.

Features are defined using standard templated queries and the Query DSL.

To streamline the process of defining and refining feature extraction during training, we have incorporated a number of primitives directly in eland.

Building the training dataset

Now that we have our basic datasets loaded, and feature extraction configured, we'll use our judgment list to come up with the final dataset for training. The dataset will consist of rows containing <query, document> pairs, as well as all of the features we need to train the model. To generate this dataset, we'll run each query from the judgment list and add the extracted features as columns for each of the labelled result documents.

For example, if we have a query q1 with two labelled documents d3 and d9, the training dataset will end up with two rows — one for each of the pairs <q1, d3> and <q1, d9>.

Note that because this executes queries on your Elasticsearch cluster, the time to run this operation will vary depending on where the cluster is hosted and where this notebook runs. For example, if you run the notebook on the same server or host as the Elasticsearch cluster, this operation tends to run very quickly on the sample dataset (< 2 mins).

Create and train the model

The LTR rescorer supports XGBRanker trained models.

Learn more in the XGBoost documentation.

Import the model into Elasticsearch

Once the model is trained we can use Eland to load it into Elasticsearch.

Please note that the MLModel.import_ltr_model method contains the LTRModelConfig object which defines how features should be extracted for the model being imported.

Using the rescorer

Once the model is uploaded to Elasticsearch, you will be able to use it as a rescorer in the _search API, as shown in this example:

	GET /movies/_search
{
   "query" : {
      "multi_match" : {
         "query": "star wars",
         "fields": ["title", "overview", "actors", "director", "tags", "characters"]
      }
   },
   "rescore" : {
      "window_size" : 50,
      "learning_to_rank" : {
         "model_id": "ltr-model-xgboost",
         "params": { 
            "query": "star wars"
         }
      }
   }
}

As also shown in the feature importance graph above, we can see in this results list that the title_bm25 and popularity features are weighted highly in our trained model. Now all results include the query terms in the title, showing the importance of the title_bm25 feature. Similarly, more popular movies now rank higher, for example Rogue One: A Star Wars Story is now in sixth position.