Hyperparameter Tuning using Your Own Keras/Tensorflow Container

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook shows how to build your own Keras(Tensorflow) container, test it locally using SageMaker Python SDK local mode, and bring it to SageMaker for training, leveraging hyperparameter tuning.

The model used for this notebook is a ResNet model, trainer with the CIFAR-10 dataset. The example is based on https://github.com/keras-team/keras/blob/master/examples/cifar10_cnn.py

Set up the notebook instance to support local mode

Currently you need to install docker-compose in order to use local mode (i.e., testing the container in the notebook instance without pushing it to ECR).

Permissions

Running this notebook requires permissions in addition to the normal SageMakerFullAccess permissions. This is because it creates new repositories in Amazon ECR. The easiest way to add these permissions is simply to add the managed policy AmazonEC2ContainerRegistryFullAccess to the role that you used to start your notebook instance. There's no need to restart your notebook instance when you do this, the new permissions will be available immediately.

Set up the environment

We will set up a few things before starting the workflow.

1. get the execution role which will be passed to sagemaker for accessing your resources such as s3 bucket
2. specify the s3 bucket and prefix where training data set and model artifacts are stored

Complete source code

• trainer/start.py: Keras model
• trainer/environment.py: Contain information about the SageMaker environment

Building the image

We will build the docker image using the Tensorflow versions on dockerhub. The full list of Tensorflow versions can be found at https://hub.docker.com/r/tensorflow/tensorflow/tags/

Prepare the data

Testing the container locally (optional)

You can test the container locally using local mode of SageMaker Python SDK. A training container will be created in the notebook instance based on the docker image you built. Note that we have not pushed the docker image to ECR yet since we are only running local mode here. You can skip to the tuning step if you want but testing the container locally can help you find issues quickly before kicking off the tuning job.

Setting the hyperparameters

Create a training job using local mode

Pushing the container to ECR

Now that we've tested the container locally and it works fine, we can move on to run the hyperparmeter tuning. Before kicking off the tuning job, you need to push the docker image to ECR first.

The cell below will create the ECR repository, if it does not exist yet, and push the image to ECR.

Specify hyperparameter tuning job configuration

Note, with the default setting below, the hyperparameter tuning job can take 20~30 minutes to complete. You can customize the code in order to get better result, such as increasing the total number of training jobs, epochs, etc., with the understanding that the tuning time will be increased accordingly as well.

Now you configure the tuning job by defining a JSON object that you pass as the value of the TuningJobConfig parameter to the create_tuning_job call. In this JSON object, you specify:

• The ranges of hyperparameters you want to tune
• The limits of the resource the tuning job can consume
• The objective metric for the tuning job

Specify training job configuration

Now you configure the training jobs the tuning job launches by defining a JSON object that you pass as the value of the TrainingJobDefinition parameter to the create_tuning_job call. In this JSON object, you specify:

• Metrics that the training jobs emit
• The container image for the algorithm to train
• The input configuration for your training and test data
• Configuration for the output of the algorithm
• The values of any algorithm hyperparameters that are not tuned in the tuning job
• The type of instance to use for the training jobs
• The stopping condition for the training jobs

This example defines one metric that Tensorflow container emits: loss.

Create and launch a hyperparameter tuning job

Now you can launch a hyperparameter tuning job by calling create_tuning_job API. Pass the name and JSON objects you created in previous steps as the values of the parameters. After the tuning job is created, you should be able to describe the tuning job to see its progress in the next step, and you can go to SageMaker console->Jobs to check out the progress of each training job that has been created.

Let's just run a quick check of the hyperparameter tuning jobs status to make sure it started successfully and is InProgress.

Analyze tuning job results - after tuning job is completed

Please refer to "HPO_Analyze_TuningJob_Results.ipynb" to see example code to analyze the tuning job results.

Deploy the best model

Now that we have got the best model, we can deploy it to an endpoint. Please refer to other SageMaker sample notebooks or SageMaker documentation to see how to deploy a model.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.