Amazon SageMaker Clarify Model Monitors

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook shows how to:

• Host a machine learning model in Amazon SageMaker and capture inference requests, results, and metadata
• Schedule Clarify bias monitor to monitor predictions for bias drift on a regular basis.
• Schedule Clarify explainability monitor to monitor predictions for feature attribution drift on a regular basis.

Background

Amazon SageMaker Model Monitor continuously monitors the quality of Amazon SageMaker machine learning models in production. It enables developers to set alerts for when there are deviations in the model quality. Early and pro-active detection of these deviations enables corrective actions, such as retraining models, auditing upstream systems, or fixing data quality issues without having to monitor models manually or build additional tooling.

Amazon SageMaker Clarify bias monitoring helps data scientists and ML engineers monitor predictions for bias on a regular basis. One way bias can be introduced in deployed ML models is when the data used in training differs from the data used to generate predictions. This is especially pronounced if the data used for training changes over time (e.g. fluctuating mortgage rates), and the model prediction will not be accurate unless the model is retrained with updated data. For example, model for predicting home prices can be biased if the mortgage rates used to train the model differ from the most current real-world mortgage rate.

Aamazon SageMaker Clarify explainability monitoring offers tools to provide global explanations of models and to explain the predictions of a deployed model producing inferences. Such model explanation tools can help ML modelers and developers and other internal stakeholders understand model characteristics as a whole prior to deployment and to debug predictions provided by the model once deployed. The current offering includes a scalable and efficient implementation of SHAP, based on the concept of the Shapley value from the field of cooperative game theory that assigns each feature an importance value for a particular prediction.

As the model is monitored, customers can view exportable reports and graphs detailing bias and feature attributions in SageMaker Studio and configure alerts in Amazon CloudWatch to receive notifications if violations are detected.

General Setup

To get started, make sure these prerequisites completed.

• Specify an AWS Region to host the model.
• An IAM role ARN exists that is used to give Amazon SageMaker access to data in Amazon Simple Storage Service (Amazon S3). See the documentation for how to fine tune the permissions needed.
• Create an S3 bucket used to store the test dataset, any additional model data, data captured from model invocations and ground truth data. For demonstration purposes, this notebook uses the same bucket for these. In reality, they could be separated with different security policies.

Imports

Import APIs to be used by the notebook.

Handful of configuration

Model files and data files

The prebuilt model, and a couple of dataset files.

To verify that the execution role for this notebook has the necessary permissions to proceed. Put a simple test object into the S3 bucket specified above. If this command fails, update the role to have s3:PutObject permission on the bucket and try again.

PART A: Capturing real-time inference data from Amazon SageMaker endpoints

Create an endpoint to showcase the data capture capability in action. (In next parts, model monitors will be created to process the data.)

Upload the pre-trained model to Amazon S3

As an example, this code uploads a pre-trained XGBoost model that is ready for to be deployed. This model was trained using the XGBoost Churn Prediction Notebook in SageMaker.

Deploy the model to Amazon SageMaker

Start with deploying a pre-trained churn prediction model. Here, create the SageMaker Model object with the image and model data.

To enable data capture for monitoring jobs, here specify the new capture option called DataCaptureConfig, it enables capturing the request payload and the response payload of the endpoint. The capture config applies to all variants. Go ahead with the deployment.

Invoke the deployed model

Now send data to this endpoint to get inferences in real time. Because data capture is enabled in the previous steps, the request and response payload, along with some additional metadata, is saved in the Amazon S3 location specified in the DataCaptureConfig.

View captured data

Now list the data capture files stored in Amazon S3. There should be different files from different time periods organized based on the hour in which the invocation occurred. The format of the Amazon S3 path is:

s3://{destination-bucket-prefix}/{endpoint-name}/{variant-name}/yyyy/mm/dd/hh/filename.jsonl

Next, view the content of a single capture file. Take a quick peek at the first few lines in the captured file.

Finally, the contents of a single line is present below in a formatted JSON file to observe a little better.

Start generating some artificial traffic

The cell below starts a thread to send some traffic to the endpoint. If there is no traffic, the monitoring jobs are marked as Failed since there is no data to process.

Notice the inferenceId attribute used above to invoke. If this is present, it will be used to join with ground truth data (otherwise eventId will be used):

Start generating some fake ground truth

Besides captures, model bias monitoring execution also requires ground truth data. In real use cases, ground truth data should be regularly collected and uploaded to designated S3 location. In this example notebook, below code snippet is used to generate fake ground truth data. The first-party merge container will combine captures and ground truth data, and the merged data will be passed to model bias monitoring job for analysis. Similar to captures, the model bias monitoring execution will fail if there's no data to merge.

PART B: Model Bias Monitor

Model bias monitor can detect bias drift of Machine Learning models in a regular basis. Similar to the other monitoring types, the standard procedure of creating a model bias monitor is first baselining and then monitoring schedule.

Create a baselining job

A baselining job runs predictions on training dataset and suggests constraints. suggest_baseline() method starts a SageMakerClarifyProcessor processing job using SageMaker Clarify container to generate the constraints.

The step is not mandatory, but providing constraints file to the monitor can enable violations file generation.

Configurations

Information about the input data need to be provided to the processor.

DataConfig stores information about the dataset to be analyzed, for example the dataset file, its format (CSV or JSONLines), headers (if any) and label.

BiasConfig is the configuration of the sensitive groups in the dataset. Typically, bias is measured by computing a metric and comparing it across groups. The group of interest is specified using the "facet." For post-training bias, the possitive label should also be taken into account.

ModelPredictedLabelConfig specifies how to extract a predicted label from the model output. This model returns probability that user will churn. Here choose an arbitrary 0.8 cutoff to consider that a customer will churn. For more complicated outputs, there are a few more options, like "label" is the index, name or JMESPath expression to locate predicted label in endpoint response payload.

ModelConfig is configuration related to model to be used for inferencing. In order to compute post-training bias metrics, the computation needs to get inferences for the model name provided. To accomplish this, the processing job will use the model to create an ephemeral endpoint (also known as "shadow endpoint"). The processing job will delete the shadow endpoint after the computations are completed. The configuration is also used by explainability monitor.

Kick off baselining job

Below cell waits until the baselining job is completed and then inspects the suggested constraints. This step can be skipped, because the monitor to be scheduled will automatically pick up baselining job name and wait for it before monitoring execution.

Schedule model bias monitor

With above constraints collected, now call create_monitoring_schedule() method to schedule a hourly monitor, to analyze the data with monitoring schedule. If a baselining job has been submitted, then the monitor will automatically pick up analysis configuration from the baselining job. But if the baselining step is skipped, or the capture dataset has different nature than the training dataset, then analysis configuration has to be provided.

BiasAnalysisConfig is a subset of the configuration of the baselining job, many options are not needed because,

• Model bias monitor will merge captures and ground truth data and use merged data as dataset. (DataConfig)
• Captures already include predictions, so there is no need to create shadow endpoint. (ModelConfig)
• Attributes like probability threshold are provided as part of EndpointInput. (ModelPredictedLabelConfig)

Wait for the first execution

The schedule starts jobs at the previously specified intervals. Code below wait util time crosses the hour boundary (in UTC) to see executions kick off.

Note: Even for an hourly schedule, Amazon SageMaker has a buffer period of 20 minutes to schedule executions. The execution might start in anywhere from zero to ~20 minutes from the hour boundary. This is expected and done for load balancing in the backend.

In real world, a monitoring schedule is supposed to be active all the time. But in this example, it can be stopped to avoid incurring extra charges. A stopped schedule will not trigger further executions, but the ongoing execution will continue. And if needed, the schedule can be restarted by start_monitoring_schedule().

Wait for the execution to finish

In the previous cell, the first execution has started. This section waits for the execution to finish so that its analysis results are available. Here are the possible terminal states and what each of them mean:

• Completed - This means the monitoring execution completed and no issues were found in the violations report.
• CompletedWithViolations - This means the execution completed, but constraint violations were detected.
• Failed - The monitoring execution failed, maybe due to client error (perhaps incorrect role permissions) or infrastructure issues. Further examination of FailureReason and ExitMessage is necessary to identify what exactly happened.
• Stopped - job exceeded max runtime or was manually stopped.

Inspect execution results

List the generated reports,

If there are violations compared to the baseline, they will be listed here.

The analysis results and CloudWatch metrics are visualized in SageMaker Studio. Select the Endpoints tab, then double click the endpoint to show the UI.

PART C: Model Explainability Monitor

Model explainability monitor can explain the predictions of a deployed model producing inferences and detect feature attribution drift on a regular basis.

Create a baselining job

Similary, a baselining job can be scheduled to suggest constraints for model explainability monitor.

Configuration

In this example, the explainability baselining job shares the test dataset with the bias baselining job, so here it uses the same DataConfig, the only difference is the job output URI.

Currently the Clarify explainer offers a scalable and efficient implementation of SHAP, so the explainability config is SHAPConfig, including

• baseline: A list of rows (at least one) or S3 object URI to be used as the baseline dataset in the Kernel SHAP algorithm. The format should be the same as the dataset format. Each row should contain only the feature columns/values and omit the label column/values.
• num_samples: Number of samples to be used in the Kernel SHAP algorithm. This number determines the size of the generated synthetic dataset to compute the SHAP values.
• agg_method: Aggregation method for global SHAP values. Valid values are
  • "mean_abs" (mean of absolute SHAP values for all instances),
  • "median" (median of SHAP values for all instances) and
  • "mean_sq" (mean of squared SHAP values for all instances).
• use_logit: Indicator of whether the logit function is to be applied to the model predictions. Default is False. If "use_logit" is true then the SHAP values will have log-odds units.
• save_local_shap_values (bool): Indicator of whether to save the local SHAP values in the output location. Default is True.

Kick off baselining job

The same model_config is required, because the explainability baselining job needs to create shadow endpoint to get predictions for generated synthetic dataset.

Wait for baselining job to finish (or skip this cell because the monitor to be scheduled will wait for it anyway).

Schedule model explainability monitor

Call create_monitoring_schedule() method to schedule a hourly monitor, to analyze the data with monitoring schedule. If a baselining job has been submitted, then the monitor will automatically pick up analysis configuration from the baselining job. But if the baselining step is skipped, or the capture dataset has different nature than the training dataset, then analysis configuration has to be provided.

ModelConfig is required by ExplainabilityAnalysisConfig for the same reason as it is required by the baselining job. Note that only features are required for computing feature attribution, so ground truth label should be excluded.

Wait for execution and inspect analysis results

Once created the schedule is started by default, here wait for the its first execution to start, then stop the schedule to avoid incurring charges.

Wait further for the execution to finish, then inspect its analysis results,

If there are any violations compared to the baseline, they will be listed here.

The analysis results and CloudWatch metrics are visualized in SageMaker Studio. Select the Endpoints tab, then double click the endpoint to show the UI.

PART D: Cleanup

The endpoint can keep running and capturing data, but if there is no plan to collect more data or use this endpoint further, it should be deleted to avoid incurring additional charges. Note that deleting endpoint does not delete the data that was captured during the model invocations.

First stop the worker threads,

Then stop all monitors scheduled for the endpoint

Finally delete the endpoint

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.