Contextual Bandits with Parametric Actions -- Experimentation Mode

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

We demonstrate how you can use varying number of actions with contextual bandits algorithms in SageMaker. This notebook builds on the Contextual Bandits example notebook example notebook which used fixed number of actions. Please refer to that notebook for basics on contextual bandits.

In the contextual bandit setting, an agent recommends an action given a state. This notebook introduces three features to bandit algorithms that make them applicable to a broader set of real-world problems. We use the movie recommendation problem as an example.

1. The number of actions available to the agent can change over time. For example, the movies in the catalog changes over time.
2. Each action may have features associated with it. For the movie recommendation problem, each movie can have features such as genre, cast, etc.
3. The agent can produce a ranked list of actions/items. When recommending movies, it is natural that multiple movies are recommended at a time step.

The contextual bandit agent will trade-off between exploitation and exploration to quickly learn user preferences and minimize poor recommendations. The bandit algorithms are appropriate to use in recommendation problems when there are many cold items (items which have no or little interaction data) in the catalog or if user preferences change over time.

What is Experimentation Mode?

Contextual bandits are often used to train models by interacting with the real world. In movie recommendation, the bandit learns user preferences based on their feedback from past interactions. To test if bandit algorithms are applicable for your use case, you may want to test different algorithms and understand the impact of different features, hyper-parameters. Experimenting with real users can lead to poor experience due to unanticipated issues or poor performance. Experimenting in production comes with the complexity of working with infrastructure components (e.g. web services, data engines, databases) designed for scale. With Experimentation Mode, you can get started with a small dataset or a simulator and identify the algorithm, features and hyper-parameters that are best applicable for your use case. The experimentation is much faster, does not impact real users and easy to work with. Once you are satisfied with the algorithm performance, you can switch to Deployment Mode, where we provide infrastructure support that scales to production requirements.

Pre-requisites

Imports

To get started, we'll import the Python libraries we need, set up the environment with a few prerequisites for permissions and configurations.

Setup S3 bucket

Set up the linkage and authentication to the S3 bucket that you want to use for data and model outputs.

Configure where training happens

You can run this notebook on a SageMaker notebook instance or on your own machine. In both of these scenarios, you can do the training/inference in either the local or the SageMaker mode. The local mode uses the SageMaker Python SDK to run your code in a docker container locally. This can speed up iterative testing and debugging while using the same familiar Python SDK interface. You just need to set local_mode = True.

If local mode is False, then training/inference runs on a SageMaker machine.

Create an IAM role

Either get the execution role when running from a SageMaker notebook instance role = sagemaker.get_execution_role() or, when running from local notebook instance, use utils method role = get_execution_role() to create an execution role.

Simulation environment (from MovieLens data)

The simulation environment src/env.py simulates a live environment that can interact with the SageMaker bandits training to produce a recommender agent (or policy). The logic of reward generation resides in simulator itself. We simulate the online learning loop with feedback using this environment inside the training job itself. The simulator uses MovieLens 100k dataset.

The training workflow is as follows:

• User sampling and candidate list generation: The simulator picks a user u and a list of 100 items (defined by item_pool_size) at random, which is sent to the SageMaker agent for retrieving recommendations. This list consists of the movies that the user u has rated in the past, as we know the true user preferences (ratings) for these movies. In this simulator, we use user_id to identify the user and represent each movie using the genre features.
• Bandit Slate recommendation: SageMaker bandit agent returns a recommendation - a list of top-k items.
• Feedback generation by simulating user behaviour: The reward is given to the agent based on user ratings in the dataset. We assume a Cascade Click model, where the user scans the list top-down, and clicks on the item that she likes. We give a reward of 0 to all the items above the clicked item and a reward to 1 to the item that was clicked. No feedback is generated for the items below the clicked item.
• Feedback ingestion: The corresponding rewards and the actions are reported to the agent for learning.

The training job uses the files in the src folder. The descriptions of the important files are as follows:

• src/train.py - This is the entrypoint for the training job. This file contains the main logic for training:
  • It initializes a bandit agent.
  • Starts an interaction loop in which the agent interacts with the envrionment, recommends some actions, ingests the feedback and improves over time.
  • The agent is saved on S3 after training finishes and can be used later for inference.
• src/env.py - This file implements the simulation environment using MovieLens 100K dataset. It also contains the logic for reward generation using the Cascade Click model.
• src/vw_agent.py - This implements a bandit agent interface in python that communicates with a VW C++ process at the backend.

MovieLens 100K usage license

Please be aware of the following requirements regarding acknowledgment, copyright and availability, cited from the data set description page.

The data set may be used for any research purposes under the following conditions:

• The user may not state or imply any endorsement from the University of Minnesota or the GroupLens Research Group.
• The user must acknowledge the use of the data set in publications resulting from the use of the data set (see below for citation information).
• The user may not redistribute the data without separate permission.
• The user may not use this information for any commercial or revenue-bearing purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota.

If you have any further questions or comments, please contact GroupLens (grouplens-info@cs.umn.edu).

Download MovieLens 100K and upload to S3

Train the Bandit model using the Python SDK Script mode

If you are using local mode, the training will run on the notebook instance/your local machine. When using SageMaker for training, you can select a CPU instance. The RLEstimator is used for training the bandit agent.

1. Specify the hyperparameters for the bandit algorithm and the environment configuration.
2. Specify the source directory where the environment, training code and dependencies are present - src folder
3. Specify the training entrypoint - train.py
4. Specify the container image
5. Define the training parameters such as the instance count, job name, S3 path for output and job name.
6. Specify the input dataset - movielens_data_s3_path in the .fit call

Define the hyperparameters and the training job name prefix

Download the outputs to plot performance

Once the training is complete, we can download the regrets data to plot the performance of the bandit agent.

SageMaker saves the model in model.tar.gz and other job output in output.tar.gz

Regret at a time step is defined as the difference between the optimal reward that an agent can get and the actual reward that the agent got.

As can be seen from the above plot, the bandit agent leads to a much lesser cumulative regret compared to choosing the actions at random. If we run the training for a larger no. of interactions (total_interactions), we will observe that the cumulative regret curve flattens out, which means that the agent has learned the user preferences successfully and training has converged.

Now, let's see how we can use the trained model to perform inference on new data.

Create a SageMaker model for inference

Inference on SageMaker can be performed using the following two modes:

• Batch Transform: Useful for scenarios that do not require a persistent serving endpoint with sub-second latency. As the name suggests, a batch transformation job processes a batch of data and is useful for achieving high throughput inference on large volumes of input data.
• Real-time inference: This mode spins up a SageMaker HTTP web server end-point, that can serve predictions in real-time.

We demonstrate both the modes in the cells below.

1. Batch Transform

Generating test dataset for inference

Here we generate some test data instances using the MovieLens simulator.

Let's upload this data to S3. Note that the format of the file should be jsonlines, which means each line of the file is a JSON dictionary.

Download batch transform results

Let's inspect the head of the results.

2. Real-time inference

If we want to do real-time inference, we can deploy the model behind a SageMaker endpoint and make requests as shown below:

Clean Up endpoint

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.