SageMakerCore Inference, Async Inference, and Resource Chaining

Introductions

In this notebook, we will walkthrough the process of performing Inference using the SageMakerCore SDK. Additionaly, this notebook will highlight how to create an endpoint using the Resource Chaining feature.

Resource Chaining

Resource Chaining is a feature provided by SageMakerCore that aims to reduce the cognitive load for a user when performing operations with the SDK. The idea is to allow users to create an object, for example a Model resource object, and pass the object directly as a parameter to some other resource like EndpointConfig. An example of this chaining can be seen below:

key = f'xgboost-iris-{strftime("%H-%M-%S", gmtime())}'

model = Model.create(...) # Create model object

endpoint_config = ndpointConfig.create(
    endpoint_config_name=key,
    production_variants=[
        ProductionVariant(
            variant_name=key,
            initial_instance_count=1,
            instance_type='ml.m5.xlarge',
            model_name=model # Pass model object directly
        )
    ]
)

Pre-Requisites

Install Latest SageMakerCore

All SageMakerCore beta distributions will be released to a private s3 bucket. After being allowlisted, run the cells below to install the latest version of SageMakerCore from s3://sagemaker-core-beta-artifacts/sagemaker_core-latest.tar.gz

Ensure you are using a kernel with python version >=3.8

Install Additional Packages

Setup

Let's start by specifying:

• AWS region.
• The IAM role arn used to give learning and hosting access to your data. Ensure your enviornment has AWS Credentials configured.
• The S3 bucket that you want to use for storing training and model data.

Load and Prepare Dataset

For this example, we will be using the IRIS data set from sklearn.datasets to train our XGBoost container.

Upload Data to S3

In this step, we will upload the train and test data to the S3 bucket configured earlier using sagemaker_session.default_bucket()

Fetch the XGBoost Image URI

In this step, we will fetch the XGBoost Image URI we will use as an input parameter when creating an AWS TrainingJob

Train XGBoost Image using IRIS Data

Next, we will the SageMakerCore TrainingJob.create() to start a training job for an XGBoost Image using IRIS data and wait for it to complete.

Create Endpoint Using Resource Chaining

In the following cells, we will walkthrough the process of creating an Endpoint using the Resource Chaining feature of SageMakerCore. Resource Chaining aims to reduce the cognitive load for a user by autoresolving necessary attributes when chaing resource objects together during operations.

1. First, we will create a Model using the model data from the training job in the previous step.
2. We will create an EndpointConfig and pass the Model object directly as a parameter. SageMakerCore will autoresolve the neccessary attributes from the Model object.
3. We will create an Endpoint using the EndpointConfig object as a parameter. SageMakerCore will autoresolve the neccessary attributes from the EndpointConfig object.

Create and Wait for Endpoint

Create a Model by specifying the image and model_data_url. For the model_data_url we will use the S3 path of the model output from the TrainingJob we performed previously.

Notice that we are able to set the model_data_url directly by referencing the s3_model_artifacts from the nested ModelArtifacts object attribute. This is possible due to SageMakerCore's object-oriented programming experience.

Class Definitions example:

class TrainingJob(Base):
    ...
    model_artifacts: Optional[ModelArtifacts] = Unassigned()

class ModelArtifacts(Base):
    s3_model_artifacts: str

A user can then reference attributes for nested objects like:

model_data_url = training_job.model_artifacts.s3_model_artifacts

Create the Endpoint and wait for it to be InService

Endpoint Invoke

The below cells demonstrates how an endpoint would be invoked synchronously in SageMakerCore using endpoint.invoke() or endpoint.inoke_with_response_stream().

In these examples, we are using CSV data to train the model and to invoking the endpoint. We will rely on the CSVSerializer and CSVDeserializer from the the sagemaker-python-sdk to assist with serilizing and deserializing the invoke input and output.

Endpoint Invoke With Response Stream

Create Endpoint for Async Invoke

Now that we have gone through the process of creating and invoking endpoint synchronously using SageMakerCore. In the next section, we will create a new endpoint for asynchronous invocations and call endpoint.invoke_async().

Download the Input Files and Pre-Trained Model tar.gz from S3

Upload Data to S3

In this step, we will upload the input and model data to the S3 bucket configured earlier using sagemaker_session.default_bucket() and set the model_url variable that we will use to create a Model resource object.

Create and Wait for Endpoint

Create a Model by specifying the image and model_data_url. For the model_data_url we will use the S3 path of the pretrained model uploaded previously.

Create the Endpoint and wait for it to be InService

Upload the Async Invoke Payload

To invoke an endpoint asynchronously, we first must upload the request payload to S3.

Endpoint Async Invoke

Call endpoint.invoke_async() using the s3 path of the invoke request payload and store the "OutputLocation" of from the response.

Check the Output Location

Check the output location from the endpoint.invoke_async() response to get the async inference results.

Delete All SageMaker Resources

The following code block will call the delete() method for any SageMaker Core Resources created during the execution of this notebook which were assigned to local or global variables. If you created any additional deleteable resources without assigning the returning object to a unique variable, you will need to delete the resource manually by doing something like:

resource = Resource.get("resource-name")
resource.delete()