Targeting Direct Marketing with Amazon SageMaker XGBoost

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Deploy a trained Gradient Boosted Trees model in SageMaker: A Binary Prediction Problem With Unbalanced Classes

Deployments with bring your own model and custom inference script

With Amazon SageMaker, you can deploy your machine learning (ML) models to make predictions, also known as inference. SageMaker provides a broad selection of ML infrastructure and model deployment options to help meet all your ML inference needs. It is a fully managed service and integrates with MLOps tools, so you can scale your model deployment, reduce inference costs, manage models more effectively in production, and reduce operational burden.

After you’ve built and trained a machine learning model, you can use SageMaker Inference to start getting predictions, or inferences, from your model. With SageMaker Inference, you can either set up an endpoint that returns inferences or run Asynchronous inference workloads

To get started with SageMaker Inference, see the following sections and review the Inference options https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html#deploy-model-options to determine which feature best fits your use case.

Background for Model and data set

Direct marketing, either through mail, email, phone, etc., is a common tactic to acquire customers. Because resources and a customer's attention is limited, the goal is to only target the subset of prospects who are likely to engage with a specific offer. Predicting those potential customers based on readily available information like demographics, past interactions, and environmental factors is a common machine learning problem.

The data set is available at https://sagemaker-sample-data-us-west-2.s3-us-west-2.amazonaws.com/autopilot/direct_marketing/bank-additional.zip

For purpose of this notebook we wil execute the following steps

• Visualize the data set used to train the model
• Upload the test data set to S3 for leveraging during the test runs of the Inferencing
• Set up the following End points
  • Real-time Inference endpoint
  • Serverless Inference endpoint
  • Asynchronous Inference endpoint
• Investigate the model latency times and pros and cons of the approach

Optoinal section:

• Scaling options and showcase how to scale endpoints in SageMaker

Preparation

This notebook was created and tested on an ml.m4.xlarge notebook instance.

Let's start by specifying:

• The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.
• The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the boto regexp with a the appropriate full IAM role arn string(s).

Now let's bring in the Python libraries that we'll use throughout the analysis

Prerequisite

Upload the model and the data to S3 for deployments. We will simulate the bring your own model concept which assumes the model is already in S3

Visualize the Data we are going to use to run predictions on. The model will essentially predict the last column

Upload the Model and test data artifacts into S3 for simulations

Upload the Testing data and the Ground Truth

The full data set is a 4119, 58 matrix

Bring in the container with the specific version which will be used to run the model

Here we need to have the container version be the same as the one which was used to train the model

Create the Inference script. SageMaker allows you to combine a pre trained model with your own inference script

Hosting

Real-time Inference endpoint

Now that we've have loaded the pre trained xgboost model in S3, let's deploy a model that's hosted behind a real-time endpoint.

This cell can take a couple of minutes please be patient

Change the default serializers which is libsvm to CSV since we will be dealing with a CSV data set

Evaluation

There are many ways to compare the performance of a machine learning model, but let's start by simply comparing actual to predicted values. In this case, we're simply predicting whether the customer subscribed to a term deposit (1) or not (0), which produces a simple confusion matrix.

First we'll need to determine how we pass data into and receive data from our endpoint. Our data is currently stored as NumPy arrays in memory of our notebook instance. To send it in an HTTP POST request, we'll serialize it as a CSV string and then decode the resulting CSV.

Note: For inference with CSV format, SageMaker XGBoost requires that the data does NOT include the target variable.

We have a couple of options here , we can:

1. Loop over our test dataset and split it into mini batches and send that into the model
2. Or we could send the entire payload as is into the model. This is governed by the payload size which cannot exceed 6MB
3. Snce our payload is smaller we can use the entire batch as is to feed
4. This can be configured by the batch_size_to_run variable. For purpose of this lab we can keep it with a shorted value

Now we'll check our confusion matrix to see how well we predicted versus actuals.

So, (since out batch size is 20 ) of the ~20 potential customers, this would be on that batch size. Please see for more details on Confusion Matrix here.

Run the below to get a P95 latency numbers for our model

Serverless Invocations

Create a Serverless Inference endpoint and show the initial cold-start issue and then subsequent calls run

For serverless we need to specify only the concurrency and memory size of the model. For further reading you can refer to https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html

As with realtime we will change the serializers to match our data sets

Cold start for the serverless

The very first time the request is sent, the serverless endpoint will spin up an instance and then run predictions on that. To save cost for the customer serverless will spin down all the instances after a certain time period of in-activity. See this for more details https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-monitoring.html

Subsequent invocation will be a lot faster

Run invocations for the endpoint and get P95 numbers

Start with the Asynchronous inference endpoint for deployment

Upload Test data sets into multiple buckets. The idea behind this is to leverage Asynchronous inference by splitting the data sets into smaller data sets which can then be fed into the system for predictions. This simplifies the blast radius and makes it easier to debug vs a batch workload. The size of each payload should be ~1G maximum. We can simulate loading the Asynchronous inference queue using these location

Once you have a model, create an Asynchronous inference configuration. Amazon SageMaker hosting services uses this configuration to deploy models. In the configuration, you identify one or more model that were created, to deploy the resources that you want Amazon SageMaker to provision. Specify the AsyncInferenceConfig object and provide an output Amazon S3 location for OutputConfig. You can optionally specify Amazon SNS topics on which to send notifications about prediction results.

Asynchronous inference endpoint can spin down to all the way down to 0 Instances to save cost

Define where the output will go - it will be for each of the outputs we run through the system for prediction

Now create the Asynchronous inference endpoint

Change the serializers to match the model requirements

Input data for which you want the model to provide inference.

The Asynchronous inference needs the data to be in s3 and below is a helper method to take the payload from memory and save it to S3 and then run the prediciton on the Asynchronous inference Endpoint

If a serializer was specified in the encapsulated in the Predictor object, the result of the serializer is sent as input data. Otherwise the data must be sequence of bytes, and the predict method then upload the data to the S3 location

The predictions will be stored in S3 location at async_output_path

We can use the SDK API's to get the result or check from the S3 location

Run the below only in case you see the output from the aws ls for async_output_path variable location

View the results using Pandas

Run Predictions using the saved data sets

This sends in the request - and the endpoint will process from the queue. It is not garunteed that the results will be available right away

The saved data sets were the full 4119 rows of data and hence the length of predictions will not match the batch size we had specified earlier

Check Output Location

Check the output location to see if the inference has been processed. We make multiple requests (beginning of the while True statement in the get_output function) every two seconds until there is an output of the inference request:

Cross tab for the predictions

Congratulations End of Lab for deployment Options

Below section is optional and depends on service limits in your accounts

Now we create a autoscaling policy for the end point. For this we will leverage the boto3 API

This section describes how to configure autoscaling on your asynchronous endpoint using Application Autoscaling. You need to first register your endpoint variant with Application Autoscaling, define a scaling policy, and then apply the scaling policy. In this configuration, we use a custom metric, CustomizedMetricSpecification, called ApproximateBacklogSizePerInstance. Please refer to the SageMaker Developer guide for a detailed list of metrics available with your asynchronous inference endpoint.

The end point is now ready for invocations with burst capacity

Run the below only in case you see scaling activities have finished - this can takea couple of minutes

endpoint status should show as Inservice

(Optional) Clean-up

If you are done with this notebook, please run the cell below. This will remove the hosted endpoint you created and avoid any charges from a stray instance being left on.

Clean the bucket and delete contents

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.