Amazon SageMaker Asynchronous Inference

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

A new near real-time Inference option for generating machine learning model predictions

Table of Contents

• Background
• Notebook Scope
• Overview and sample end to end flow
• Section 1 - Setup
  • Create Model
  • Create EndpointConfig
  • Create Endpoint
  • Setup AutoScaling policy (Optional)
• Section 2 - Using the Endpoint
  • Invoke Endpoint
  • Check Output Location
  • Multiple Invocations
• Section 3 - Clean up

Background

Amazon SageMaker Asynchronous Inference is a new capability in SageMaker that queues incoming requests and processes them asynchronously. SageMaker currently offers two inference options for customers to deploy machine learning models: 1) a real-time option for low-latency workloads 2) Batch transform, an offline option to process inference requests on batches of data available upfront. Real-time inference is suited for workloads with payload sizes of less than 6 MB and require inference requests to be processed within 60 seconds. Batch transform is suitable for offline inference on batches of data.

Asynchronous inference is a new inference option for near real-time inference needs. Requests can take up to 15 minutes to process and have payload sizes of up to 1 GB. Asynchronous inference is suitable for workloads that do not have sub-second latency requirements and have relaxed latency requirements. For example, you might need to process an inference on a large image of several MBs within 5 minutes. In addition, asynchronous inference endpoints let you control costs by scaling down endpoints instance count to zero when they are idle, so you only pay when your endpoints are processing requests.

Notebook scope

This notebook provides an introduction to the SageMaker Asynchronous inference capability. This notebook will cover the steps required to create an asynchronous inference endpoint and test it with some sample requests.

Overview and sample end to end flow

Asynchronous inference endpoints have many similarities (and some key differences) compared to real-time endpoints. The process to create asynchronous endpoints is similar to real-time endpoints. You need to create: a model, an endpoint configuration, and then an endpoint. However, there are specific configuration parameters specific to asynchronous inference endpoints which we will explore below.

Invocation of asynchronous endpoints differ from real-time endpoints. Rather than pass request payload inline with the request, you upload the payload to Amazon S3 and pass an Amazon S3 URI as a part of the request. Upon receiving the request, SageMaker provides you with a token with the output location where the result will be placed once processed. Internally, SageMaker maintains a queue with these requests and processes them. During endpoint creation, you can optionally specify an Amazon SNS topic to receive success or error notifications. Once you receive the notification that your inference request has been successfully processed, you can access the result in the output Amazon S3 location.

The diagram below provides a visual overview of the end-to-end flow with Asynchronous inference endpoint.

Dataset

We're about to work with the Titanic dataset[1]. From the dataset documentation:

The original Titanic dataset, describing the survival status of individual passengers on the Titanic. The titanic data does not contain information from the crew, but it does contain actual ages of half of the passengers. The principal source for data about Titanic passengers is the Encyclopedia Titanica. The datasets used here were begun by a variety of researchers. One of the original sources is Eaton & Haas (1994) Titanic: Triumph and Tragedy, Patrick Stephens Ltd, which includes a passenger list created by many researchers and edited by Michael A. Findlay.

Thomas Cason of UVa has greatly updated and improved the Titanic data frame using the Encyclopedia Titanica and created the dataset here. Some duplicate passengers have been dropped, many errors corrected, many missing ages filled in, and new variables created.

1. Setup

First we ensure we have an updated version of boto3, which includes the latest SageMaker features:

Import the required python libraries:

Specify your IAM role. Go the AWS IAM console (https://console.aws.amazon.com/iam/home) and add the following policies to your IAM Role:

• SageMakerFullAccessPolicy

• (Optional) Amazon SNS access: Add sns:Publish on the topics you define. Apply this if you plan to use Amazon SNS to receive notifications.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sns:Publish"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:sns:<aws-region>:<account-id>:<topic-name>"
        }
    ]
}

• (Optional) KMS decrypt, encrypt if your Amazon S3 bucket is encrypte.

Specify your SageMaker IAM Role (sm_role) and Amazon S3 bucket (s3_bucket). You can optionally use a default SageMaker Session IAM Role and Amazon S3 bucket. Make sure the role you use has the necessary permissions for SageMaker, Amazon S3, and optionally Amazon SNS.

Next, you will create a model with CreateModel, an endpoint configuration with CreateEndpointConfig, and then an endpoint with the CreateEndpoint API.

1.1 Create Model

Specify the location of the pre-trained model stored in Amazon S3. This example uses a pre-trained XGBoost model name demo-xgboost-model.tar.gz. The full Amazon S3 URI is stored in a string variable model_url.

Specify a primary container. For the primary container, you specify the Docker image that contains inference code, artifacts (from prior training), and a custom environment map that the inference code uses when you deploy the model for predictions. In this example, we specify an XGBoost built-in algorithm container image.

Create a model by specifying the ModelName, the ExecutionRoleARN (the ARN of the IAM role that Amazon SageMaker can assume to access model artifacts/ docker images for deployment), and the PrimaryContainer.

1.2 Create EndpointConfig

Once you have a model, create an endpoint configuration with CreateEndpointConfig. Amazon SageMaker hosting services uses this configuration to deploy models. In the configuration, you identify one or more model that were created using with CreateModel API, to deploy the resources that you want Amazon SageMaker to provision. Specify the AsyncInferenceConfig object and provide an output Amazon S3 location for OutputConfig. You can optionally specify Amazon SNS topics on which to send notifications about prediction results.

1.3 Create Endpoint

Once you have your model and endpoint configuration, use the CreateEndpoint API to create your endpoint. The endpoint name must be unique within an AWS Region in your AWS account.

Validate that the endpoint is created before invoking it:

1.4 Setup AutoScaling policy (Optional)

This section describes how to configure autoscaling on your asynchronous endpoint using Application Autoscaling. You need to first register your endpoint variant with Application Autoscaling, define a scaling policy, and then apply the scaling policy. In this configuration, we use a custom metric, CustomizedMetricSpecification, called ApproximateBacklogSizePerInstance. Please refer to the SageMaker Developer guide for a detailed list of metrics available with your asynchronous inference endpoint.

The endpoint is now ready for invocation.

2. Using the Endpoint

2.1 Uploading the Request Payload

First, you need to upload the request to Amazon S3. We define a function called, upload_file, to make it easier to make multiple invocations in a later step.

2.1 Invoke Endpoint

Get inferences from the model hosted at your asynchronous endpoint with InvokeEndpointAsync. Specify the location of your inference data in the InputLocation field and the name of your endpoint for EndpointName. The response payload contains the output Amazon S3 location where the result will be placed.

2.2 Check Output Location

Check the output location to see if the inference has been processed. We make multiple requests (beginning of the while True statement in the get_output function) every two seconds until there is an output of the inference request:

2.3 Multiple Invocations

The following shows how you can invoke the endpoint with multiple requests:

3. Clean up

If you enabled auto-scaling for your endpoint, ensure you deregister the endpoint as a scalable target before deleting the endpoint. To do this, run the following:

Remember to delete your endpoint after use as you will be charged for the instances used in this Demo.

You may also want to delete any other resources you might have created such as SNS topics, S3 objects, etc.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.