Amazon SageMaker Batch Transform: Associate prediction results with their corresponding input records

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.


Use SageMaker's XGBoost to train a binary classification model and for a list of tumors in batch file, predict if each is malignant

It also shows how to use the input output joining / filter feature in Batch transform in details

Background

This purpose of this notebook is to train a model using SageMaker's XGBoost and UCI's breast cancer diagnostic data set to illustrate at how to run batch inferences and how to use the Batch Transform I/O join feature. UCI's breast cancer diagnostic data set is available at https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29. The data set is also available on Kaggle at https://www.kaggle.com/uciml/breast-cancer-wisconsin-data. The purpose here is to use this data set to build a predictve model of whether a breast mass image indicates benign or malignant tumor.

Setup

Let's start by specifying:

• The SageMaker role arn used to give training and batch transform access to your data. The snippet below will use the same role used by your SageMaker notebook instance. Otherwise, specify the full ARN of a role with the SageMakerFullAccess policy attached.
• The S3 bucket that you want to use for training and storing model objects.

Data sources

Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

Breast Cancer Wisconsin (Diagnostic) Data Set [https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)].

Also see: Breast Cancer Wisconsin (Diagnostic) Data Set [https://www.kaggle.com/uciml/breast-cancer-wisconsin-data].

Data preparation

Let's download the data and save it in the local folder with the name data.csv and take a look at it.

Key observations:

• The data has 569 observations and 32 columns.
• The first field is the 'id' attribute that we will want to drop before batch inference and add to the final inference output next to the probability of malignancy.
• Second field, 'diagnosis', is an indicator of the actual diagnosis ('M' = Malignant; 'B' = Benign).
• There are 30 other numeric features that we will use for training and inferencing.

Let's replace the M/B diagnosis with a 1/0 boolean value.

Let's split the data as follows: 80% for training, 10% for validation and let's set 10% aside for our batch inference job. In addition, let's drop the 'id' field on the training set and validation set as 'id' is not a training feature. For our batch set however, we keep the 'id' feature. We'll want to filter it out prior to running our inferences so that the input data features match the ones of training set and then ultimately, we'll want to join it with inference result. We are however dropping the diagnosis attribute for the batch set since this is what we'll try to predict.

Let's upload those data sets in S3

Training job and model creation

The below cell uses the SageMaker Python SDK to kick off the training job using both our training set and validation set. Not that the objective is set to 'binary:logistic' which trains a model to output a probability between 0 and 1 (here the probability of a tumor being malignant).

Batch Transform

In SageMaker Batch Transform, we introduced 3 new attributes - input_filter, join_source and output_filter. In the below cell, we use the SageMaker Python SDK to kick-off several Batch Transform jobs using different configurations of these 3 new attributes. Please refer to this page to learn more about how to use them.

1. Create a transform job with the default configurations

Let's first skip these 3 new attributes and inspect the inference results. We'll use it as a baseline to compare to the results with data processing.

Let's inspect the output of the Batch Transform job in S3. It should show the list probabilities of tumors being malignant.

2. Join the input and the prediction results

Now, let's associate the prediction results with their corresponding input records. We can also use the input_filter to exclude the ID column easily and there's no need to have a separate file in S3.

• Set input_filter to "$[1:]": indicates that we are excluding column 0 (the 'ID') before processing the inferences and keeping everything from column 1 to the last column (all the features or predictors)

• Set join_source to "Input": indicates our desire to join the input data with the inference results

• Leave output_filter to default ('$'), indicating that the joined input and inference results be will saved as output.

Let's inspect the output of the Batch Transform job in S3. It should show the list of tumors identified by their original feature columns and their corresponding probabilities of being malignant.

3. Update the output filter to keep only ID and prediction results

Let's change output_filter to "$[0,-1]", indicating that when presenting the output, we only want to keep column 0 (the 'ID') and the last column (the inference result i.e. the probability of a given tumor to be malignant)

Now, let's inspect the output of the Batch Transform job in S3 again. It should show 2 columns: the ID and their corresponding probabilities of being malignant.

In summary, we can use newly introduced 3 attributes - input_filter, join_source, output_filter to

1. Filter / select useful features from the input dataset. e.g. exclude ID columns.
2. Associate the prediction results with their corresponding input records.
3. Filter the original or joined results before saving to S3. e.g. keep ID and probability columns only.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.