Hyperparameter Tuning with the SageMaker TensorFlow Container

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This tutorial focuses on how to create a convolutional neural network model to train the MNIST dataset using the SageMaker TensorFlow container. It leverages hyperparameter tuning to run multiple training jobs with different hyperparameter combinations, to find the one with the best model training result.

Runtime

This notebook takes approximately 10 minutes to run.

Contents

1. Set Up the Environment
2. Data
3. Run a TensorFlow Training Job
4. Set Up Channels for Training and Testing Data
5. Run a Hyperparameter Tuning Job
6. Deploy the Best Model
7. Evaluate
8. Cleanup

Set Up the Environment

Set up a few things before starting the workflow:

1. A boto3 session object to manage interactions with the Amazon SageMaker APIs.
2. An execution role which is passed to SageMaker to access your AWS resources.

Data

Download the MNIST data from a public S3 bucket and save it in a temporary directory.

Run a TensorFlow Training Job

A TensorFlow training job is defined by using the TensorFlow estimator class. It lets you run your training script on SageMaker infrastructure in a containerized environment. For more information on how to instantiate it, see the example Train an MNIST model with TensorFlow.

Set Up Channels for Training and Testing Data

Upload the MNIST data to the default bucket of your AWS account and pass the S3 URI as the channels of training and testing data for the TensorFlow estimator class.

Run a Hyperparameter Tuning Job

Now that you have set up the training job and the input data channels, you are ready to train the model with hyperparameter search.

Set up the hyperparameter tuning job with the following steps:

• Define the ranges of hyperparameters we plan to tune. In this example, we tune the learning rate.
• Define the objective metric for the tuning job to optimize.
• Create a hyperparameter tuner with the above setting, as well as tuning resource configurations.

For a typical ML model, there are three kinds of hyperparamters:

• Categorical parameters need to take one value from a discrete set. We define this by passing the list of possible values to CategoricalParameter(list)
• Continuous parameters can take any real number value between the minimum and maximum value, defined by ContinuousParameter(min, max)
• Integer parameters can take any integer value between the minimum and maximum value, defined by IntegerParameter(min, max)

Learning rate is a continuous variable, so we define its range by ContinuousParameter.

Next we specify the objective metric that we'd like to tune and its definition, which includes the regular expression (regex) needed to extract that metric from the CloudWatch logs of the training job. In this particular case, our script emits average loss value and we use it as the objective metric. We set the objective_type to Minimize, so that hyperparameter tuning seeks to minimize the objective metric when searching for the best hyperparameter value.

Now, you'll create a HyperparameterTuner object. It takes the following parameters:

• The TensorFlow estimator you previously created.
• Your hyperparameter ranges.
• Objective metric name and definition.
• Tuning resource configurations such as the number of training jobs to run in total, and how many training jobs to run in parallel.

Deploy the Best Model

After training with hyperparameter optimization, you can deploy the best-performing model (by the objective metric you defined) to a SageMaker endpoint. For more information about deploying a model to a SageMaker endpoint, see the example Deploy a Trained TensorFlow V2 Model.

Evaluate

Now, you can evaluate the best-performing model by invoking the endpoint with the MNIST test set. The test data needs to be readily consumable by the model, so we arrange them into the correct shape that is accepted by a TensorFlow model. We also normalize them so that the pixel values have mean 0 and standard deviation 1, since this is the convention used to train the model.

Cleanup

If you do not plan to continue using the endpoint, delete it to free up resources.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.