Amazon SageMaker Multi-Model Endpoints using Linear Learner

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

With Amazon SageMaker multi-model endpoints, customers can create an endpoint that seamlessly hosts up to thousands of models. These endpoints are well suited to use cases where any one of a large number of models, which can be served from a common inference container, needs to be invokable on-demand and where it is acceptable for infrequently invoked models to incur some additional latency. For applications which require consistently low inference latency, a traditional endpoint is still the best choice.

At a high level, Amazon SageMaker manages the loading and unloading of models for a multi-model endpoint, as they are needed. When an invocation request is made for a particular model, Amazon SageMaker routes the request to an instance assigned to that model, downloads the model artifacts from S3 onto that instance, and initiates loading of the model into the memory of the container. As soon as the loading is complete, Amazon SageMaker performs the requested invocation and returns the result. If the model is already loaded in memory on the selected instance, the downloading and loading steps are skipped and the invocation is performed immediately.

Amazon SageMaker inference pipeline model consists of a sequence of containers that serve inference requests by combining preprocessing, predictions and post-processing data science tasks. An inference pipeline allows you to apply the same preprocessing code used during model training, to process the inference request data used for predictions.

To demonstrate how multi-model endpoints are created and used with inference pipeline, this notebook provides an example using a set of Linear Learner models that each predict housing prices for a single location. This domain is used as a simple example to easily experiment with multi-model endpoints.

This notebook showcases three MME capabilities:

• Native MME support with Amazon SageMaker Linear Learner algorithm. Because of the native support there is no need for you to create a custom container.
• Native MME support with Amazon SageMaker Inference Pipelines.
• Granular InvokeModel access to multiple models hosted on the MME using IAM condition key.

To demonstrate these capabilities, the notebook discusses the use case of predicting house prices in multiple cities using linear regression. House prices are predicted based on features like number of bedrooms, number of garages, square footage etc. Depending on the city, the features affect the house price differently. For example, small changes in the square footage cause a drastic change in house prices in New York when compared to price changes in Houston. For accurate house price predictions, we will train multiple linear regression models, a unique location specific model per city.

Contents

1. Generate synthetic data for housing models
2. Preprocess the raw housing data using Scikit Learn model
3. Train multiple house value prediction models for multiple cities
4. Create model entity with multi model support
5. Create an inference pipeline with sklearn model and MME linear learner model
6. Exercise the inference pipeline - Get predictions from the different linear learner models
7. Update Multi Model Endpoint with new models
8. Explore granular access to the target models of MME
9. Endpoint CloudWatch Metrics Analysis
10. Clean up

Section 1 - Generate synthetic data for housing models

In this section, you will generate synthetic data that will be used to train the linear learner models. The data generated consists of 6 numerical features - the year the house was built in, house size in square feet, number of bedrooms, number of bathroom, the lot size and number of garages and two categorial features - deck and front_porch.

Section 2 - Preprocess the raw housing data using Scikit Learn

In this section, the categorical features of the data (deck and porch) are pre-processed using sklearn to convert them to one hot encoding representation.

Section 3 : Train house value prediction models for multiple cities

In this section, you will use the preprocessed housing data to train multiple linear learner models.

Launch a single training job for a given housing location

There is nothing specific to multi-model endpoints in terms of the models it will host. They are trained in the same way as all other SageMaker models. Here we are using the Linear Learner estimator and not waiting for the job to complete.

Kick off a model training job for each housing location

Wait for all training jobs to finish

Section 4 - Create Sagemaker model with multi model support

Section 5 : Create an inference pipeline with sklearn model and MME linear learner model

Set up the inference pipeline using the Pipeline Model API. This sets up a list of models in a single endpoint; In this example, we configure our pipeline model with the fitted Scikit-learn inference model and the fitted Linear Learner model.

Section 6 : Exercise the inference pipeline - Get predictions from different linear learner models.

Section 7 - Add new model to the endpoint, simply by copying the model artifact to the S3 location

Section 8 - Endpoint CloudWatch Metrics Analysis

With MME, the models are dynamically loaded into the container’s memory of the instance hosting the endpoint when invoked. Therefore, the model invocation may take longer when it is invoked for the first time. And after the model is already in the container’s memory, the subsequent invocations will be faster. If an instance memory utilization is high and a new model needs to be loaded then unused models are unloaded. The unloaded models will remain in the instance’s storage volume and can be loaded into container’s memory later without being downloaded from the S3 bucket again. If the instance’s storage volume if full, unused models are deleted from storage volume.
Managing the loading/unloading of the models is completely handled by Amazon SageMaker behind the scenes without you having to take any specific actions. However, it is important to understand this behavior because it has implications on the model invocation latency.

Amazon SageMaker provides CloudWatch metrics for multi-model endpoints so you can determine the endpoint usage and the cache hit rate and optimize your endpoint. To analyze the endpoint and the container behavior, you will invoke multiple models in this order :

a. Create 200 copies of the original model and save with different names.
b. Starting with no models loaded into the container, Invoke the first 100 models
c. Invoke the same 100 models again
d. Invoke all 200 models

We use this order of invocations to observe the behavior of the CloudWatch metrics - LoadedModelCount, MemoryUtilization and ModelCacheHit. You are encouraged to experiment with loading varying number of models to use the CloudWatch charts to help make ongoing decisions on the optimal choice of instance type, instance count, and number of models that a given endpoint should host.

CloudWatch charts for LoadedModelCount,MemoryUtilization and ModelCacheHit metrics will be similar to charts below.

“LoadedModelCount” continuously increases, as more models are invoked, till it levels off at 121. “MemoryUtilization” of the container also increased correspondingly to around 79%. This shows that the instance chosen to host the endpoint, could only maintain 121 models in memory, when 200 model invocations are made.

As the number of models loaded to the container memory increase, the ModelCacheHit improves. When the same 100 models are invoked the second time, the ModelCacheHit reaches 1. When new models, not yet loaded are invoked the ModelCacheHit decreases again.

Section 9 - Explore granular access to the target models of MME

If the role attached to this notebook instance allows invoking SageMaker endpoints, it is able to invoke all models hosted on the MME. Using IAM conditional keys, you can restrict this model invocation access to specific models. To explore this, you will create a new IAM role and IAM policy with conditional key to restrict access to a single model. Assume this new role and verify that only a single target model can be invoked.

Note that to execute this section, the role attached to the notebook instance should allow the following actions : "iam:CreateRole", "iam:CreatePolicy", "iam:AttachRolePolicy", "iam:UpdateAssumeRolePolicy"

If this is not the case, please work with the Administrator of this AWS account to ensure this.

Clean up

Clean up the endpoint to avoid unneccessary costs.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.