Comparing model metrics with SageMaker Pipelines and SageMaker Model Registry

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

SageMaker Model Registry

SageMaker Model Registry is a central location where customers can manage their machine learning models, compare different model versions and visualize metrics. It's a registry with which data scientists can register machine learning models with relevant metadata and metrics, and from which machine learning engineers or DevOps engineers can deploy machine learning models. In larger MLOps systems, a model registry is usually where the teams in charge of deploying machine learning models, meet the teams in charge of developing and training machine learning models.

MLOps

MLOps, or Machine Learning Operations, is the concept of applying DevOps practices on the lifecycle of a machine learning model. Among many other things, MLOps usually consists of two workflows that sits on either side of a machine learning model registry; one to train a model and one to deploy a model. A model registry is a central location to manage machine learning models, where ML engineers and data scientists can compare different model versions, visualize metrics, and decide which versions to accept and which to reject. Ideally, approving a new version of a model triggers a pipeline that ultimately deploys the model into production.

From a high-level perspective, it can look like this.

The deployment pipeline is what is most similar to traditional CI/CD methods, propagating an artifact through a pipeline that performs testing and if the tests are successful, deploys them. Any CI/CD tools can be used for the deployment, but to automate the entire machine learning model lifecycle, a pipeline on "the other side" of the model registry is required as well. A pipeline that ultimately produces a new machine learning model and registers it with the model registry. A SageMaker Pipeline.

This notebook, demonstrate how SageMaker Pipelines can be used to create a reusable machine learning pipeline that preprocesses, trains, evaluates and registers a machine learning model with the SageMaker Model Registry for visualization and comparison of different model versions.

Amazon SageMaker Pipelines

Amazon SageMaker Pipelines is a purpose-built, easy-to-use CI/CD service for machine learning. With SageMaker Pipelines, customers can create machine learning workflows with an easy-to-use Python SDK, and then visualize and manage workflows using Amazon SageMaker Studio.

SageMaker Pipeline steps and parameters

SageMaker pipelines works on the concept of steps. The order steps are executed in is inferred from the dependencies each step have. If a step has a dependency on the output from a previous step, it's not executed until after that step has completed successfully.

SageMaker Pipeline Parameters are input parameters specified when triggering a pipeline execution. They need to be explicitly defined when creating the pipeline and contain default values.

To know more about the type of steps and parameters supported, check out the SageMaker Pipelines Overview.

SageMaker Pipeline DAG

When creating a SageMaker Pipeline, SageMaker creates a Direct Acyclic Graph, DAG, that customers can visualize in Amazon SageMaker Studio. The DAG can be used to track pipeline executions, outputs and metrics. In this notebook, a SageMaker pipeline with the following DAG is created:

Predict customer churn with XGboost

Data

This notebook uses a synthetic dataset.

This dataset contains information about phone carrier customers, such as customer location, phone number, number of phone calls etc.

Overview

Disclaimer This notebook was created using Amazon SageMaker Studio and the Python3(DataScience) kernel. SageMaker Studio is required for the visualizations of the DAG and model metrics to work.

The purpose of this notebook is to demonstrate how SageMaker Pipelines can be used to preprocess, train, evaluate and push new machine learning models into the SageMaker model registry for version comparison. All scripts to preprocess the data and evaluate the trained model have been prepared in advance and are available here:

• preprocess.py
• evaluate.py.

Table of contents

• Define parameters In this section the parameters of the pipeline are defined.
• Preprocess step In this section an SKLearnProcessor is created and used in a Preprocess step.
• Train step In this section the SageMaker managed XGboost container is downloaded and an Estimator object and Training step are created.
• Evaluate model step In this section a ScriptProcessor is created, used in a Processing step to compute some evaluation metrics of the previously trained model.
• Condition step In this section a condition step is defined, using the metrics from the evaluation step.
• Register model step In this section a register model step is created, where the trained model is registered with the SageMaker Model Registry.
• Create SageMaker Pipeline In the last section, the SageMaker pipeline is created and all steps orchestrated before executing the pipeline.

Pipeline input parameters

Pipeline Parameters are input parameter when triggering a pipeline execution. They need to be explicitly defined when creating the pipeline and contain default values.

Preprocess data step

In the first step an sklearn processor is created, used in the ProcessingStep.

Train model step

In the second step, the train and validation output from the precious processing step are used to train a model. The XGBoost container is retrieved and then an XGBoost estimator is created, on which hyper parameters are specified before the training step is created.

Evaluate model step

When a model a trained, it's common to evaluate the model on unseen data before registering it with the model registry. This ensures the model registry isn't cluttered with poorly performing model versions. To evaluate the model, create a ScriptProcessor object and use it in a ProcessingStep.

Note that a separate preprocessed test dataset is used to evaluate the model, and not the output of the processing step. This is only for demo purposes, to ensure the second run of the pipeline creates a model with better performance. In a real-world scenario, the test output of the processing step would be used.

Register model step

If the trained model meets the model performance requirements a new model version is registered with the model registry for further analysis. To attach model metrics to the model version, create a ModelMetrics object using the evaluation report created in the evaluation step. Then, create the RegisterModel step.

Accuracy condition step

Adding conditions to the pipeline is done with a ConditionStep. In this case, we only want to register the new model version with the model registry if the new model meets an accuracy condition.

Pipeline Creation: Orchestrate all steps

Now that all pipeline steps are created, a pipeline is created.

Submit pipeline, and start it.

Visualize SageMaker Pipeline DAG

In SageMaker Studio, choose SageMaker Components and registries in the left pane and under Pipelines, click the pipeline that was created. Then all pipeline executions are shown and the one just created should have a status of Executing. Selecting that execution, the different pipeline steps can be tracked as they execute.

Visualize model performance metrics

Once the pipeline has completed successfully, metrics attached to the model version can be visualized. In SageMaker Studio, choose SageMaker Components and registries in the left pane and under Model registry, select the model package that was created. If a new model package group was created, only one model version should be visible. Click that version and visualize the model performance metrics.

Compare model performance metrics

When there are more than one model version, they can be visualized side by side.

Run the pipeline again, but with a larger dataset.

Visualize

Select both versions and right-click. Choose Compare model versions.

Clean up (optional)

Delete the model registry and the pipeline to keep the studio environment tidy.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.