Multiclass classification with Amazon SageMaker XGBoost algorithm

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Single machine and distributed training for multiclass classification with Amazon SageMaker XGBoost algorithm

Contents

1. Introduction
2. Prerequisites and Preprocessing
   1. Permissions and environment variables
   2. Data ingestion
   3. Data conversion
3. Training the XGBoost model
   1. Training on a single instance
   2. Training on multiple instances
   3. Training with Automatic Model Tuning (HPO)
4. Set up hosting for the model
   1. Import model into hosting
   2. Create endpoint configuration
   3. Create endpoint
5. Validate the model for use

Introduction

This notebook demonstrates the use of Amazon SageMaker’s implementation of the XGBoost algorithm to train and host a multiclass classification model. The MNIST dataset is used for training. It has a training set of 60,000 examples and a test set of 10,000 examples. To illustrate the use of libsvm training data format, we download the dataset and convert it to the libsvm format before training.

To get started, we need to set up the environment with a few prerequisites for permissions and configurations.

Prequisites and Preprocessing

This notebook was tested in Amazon SageMaker Studio on a ml.t3.medium instance with Python 3 (Data Science) kernel.

Permissions and environment variables

Here we set up the linkage and authentication to AWS services.

1. The roles used to give learning and hosting access to your data. See the documentation for how to specify these.
2. The S3 buckets that you want to use for training and model data and where the downloaded data is located.

Data ingestion

Next, we read the MNIST dataset [1] from an existing repository into memory, for preprocessing prior to training. It was downloaded from this link and stored in downloaded_data_bucket. Processing could be done in situ by Amazon Athena, Apache Spark in Amazon EMR, Amazon Redshift, etc., assuming the dataset is present in the appropriate location. Then, the next step would be to transfer the data to S3 for use in training. For small datasets, such as this one, reading into memory isn't onerous, though it would be for larger datasets.

[1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998.

Data conversion

Since algorithms have particular input and output requirements, converting the dataset is also part of the process that a data scientist goes through prior to initiating training. In this particular case, the data is converted from pickle-ized numpy array to the libsvm format before being uploaded to S3. The hosted implementation of xgboost consumes the libsvm converted data from S3 for training. The following provides functions for data conversions and file upload to S3 and download from S3.

Training the XGBoost model

Now that we have our data in S3, we can begin training. We'll use Amazon SageMaker XGboost algorithm, and will actually fit two models in order to demonstrate the single machine and distributed training on SageMaker. In the first job, we'll use a single machine to train. In the second job, we'll use two machines and use the ShardedByS3Key mode for the train channel. Since we have 5 part file, one machine will train on three and the other on two part files. Note that the number of instances should not exceed the number of part files.

First let's setup a list of training parameters which are common across the two jobs.

Now we'll create two separate jobs, updating the parameters that are unique to each.

Training on a single instance

Training on multiple instances

You can also run the training job distributed over multiple instances. For larger datasets with multiple partitions, this can significantly boost the training speed. Here we'll still use the small/toy MNIST dataset to demo this feature.

Let's submit these jobs, taking note that the first will be submitted to run in the background so that we can immediately run the second in parallel.

Let's confirm both jobs have finished.

Training with Automatic Model Tuning (HPO)

Instead of maunally configuring your hyper parameter values and training with SageMaker Training, you could also train with Amazon SageMaker Automatic Model Tuning. AMT, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose.

To create a tuning job using the AWS SageMaker Automatic Model Tuning API, you need to define 3 attributes.

1. the tuning job name (string)
2. the tuning job config (to specify settings for the hyperparameter tuning job - JSON object)
3. training job definition (to configure the training jobs that the tuning job launches - JSON object).

To learn more about that, refer to the Configure and Launch a Hyperparameter Tuning Job documentation.

In this notebook, the model that the HPO job creates is the one that is eventually hosted. You can instead choose to deploy the model created by the standalone training job by changing the below variable deploy_amt_model to False.

Note that the tuning job will 10-12 minutes to complete.

Set up hosting for the model

In order to set up hosting, we have to import the model from training to hosting. The step below demonstrated hosting the model generated from the standalone distributed training job scheduled directly against SageMaker Training. Same steps can be followed to host the model obtained from the single machine job, or the one from the tuning job. Note that for the tuning job, the best training (field accessible from the tuning job object) need to be selected first (see the comment in the code reference below).

Import model into hosting

Next, you register the model with hosting. This allows you the flexibility of importing models trained elsewhere.

Create endpoint configuration

SageMaker supports configuring REST endpoints in hosting with multiple models, e.g. for A/B testing purposes. In order to support this, customers create an endpoint configuration, that describes the distribution of traffic across the models, whether split, shadowed, or sampled in some way. In addition, the endpoint configuration describes the instance type required for model deployment.

Create endpoint

Lastly, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete.

Validate the model for use

Finally, the customer can now validate the model for use. They can obtain the endpoint from the client library using the result from previous operations, and generate classifications from the trained model using that endpoint.

In order to evaluate the model, we'll use the test dataset previously generated. Let us first download the data from S3 to the local host.

Start with a single prediction. Lets use the first record from the test file.

OK, a single prediction works. Let's do a whole batch and see how good is the predictions accuracy.

The following function helps us calculate the error rate on the batch dataset.

Here are a few predictions

and the corresponding labels

The following function helps us create the confusion matrix on the labeled batch test dataset.

The following helps us visualize the erros that the XGBoost classifier is making.

Delete Endpoint

Once you are done using the endpoint, you can use the following to delete it.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.