Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

Train and hyperparameter tune on Iris Dataset with Scikit-learn

In this tutorial, we demonstrate how to use the Azure ML Python SDK to train a support vector machine (SVM) on a single-node CPU with Scikit-learn to perform classification on the popular Iris dataset. We will also demonstrate how to perform hyperparameter tuning of the model using Azure ML's HyperDrive service.

Prerequisites

• Go through the Configuration notebook to install the Azure Machine Learning Python SDK and create an Azure ML Workspace

Diagnostics

Opt-in diagnostics for better experience, quality, and security of future releases.

Initialize workspace

Initialize a Workspace object from the existing workspace you created in the Prerequisites step. Workspace.from_config() creates a workspace object from the details stored in config.json.

Create AmlCompute

You will need to create a compute target for training your model. In this tutorial, we use Azure ML managed compute (AmlCompute) for our remote training compute resource.

Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.

As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota.

If we could not find the cluster with the given name, then we will create a new cluster here. We will create an AmlCompute cluster of STANDARD_D2_V2 CPU VMs. This process is broken down into 3 steps:

1. create the configuration (this step is local and only takes a second)
2. create the cluster (this step will take about 20 seconds)
3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about 3-5 minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell

The above code retrieves a CPU compute target. Scikit-learn does not support GPU computing.

Train model on the remote compute

Now that you have your data and training script prepared, you are ready to train on your remote compute. You can take advantage of Azure compute to leverage a CPU cluster.

Create a project directory

Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on.

Prepare training script

Now you will need to create your training script. In this tutorial, the training script is already provided for you at train_iris.py. In practice, you should be able to take any custom training script as is and run it with Azure ML without having to modify your code.

However, if you would like to use Azure ML's tracking and metrics capabilities, you will have to add a small amount of Azure ML code inside your training script.

In train_iris.py, we will log some metrics to our Azure ML run. To do so, we will access the Azure ML Run object within the script:

	from azureml.core.run import Run
run = Run.get_context()

Further within train_iris.py, we log the kernel and penalty parameters, and the highest accuracy the model achieves:

	run.log('Kernel type', np.string(args.kernel))
run.log('Penalty', np.float(args.penalty))

run.log('Accuracy', np.float(accuracy))

These run metrics will become particularly important when we begin hyperparameter tuning our model in the "Tune model hyperparameters" section.

Once your script is ready, copy the training script train_iris.py into your project directory.

Create an experiment

Create an Experiment to track all the runs in your workspace for this Scikit-learn tutorial.

Create an environment

Create an Azure ML environment.

Configure the training job

Create a ScriptRunConfig object to specify the configuration details of your training job, including your training script, environment to use, and the compute target to run on.

Submit job

Run your experiment by submitting your ScriptRunConfig object. Note that this call is asynchronous.

Monitor your run

You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes.

Tune model hyperparameters

Now that we've seen how to do a simple Scikit-learn training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's hyperparameter tuning capabilities.

Start a hyperparameter sweep

First, we will define the hyperparameter space to sweep over. Let's tune the kernel and penalty parameters. In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, Accuracy.

Finally, lauch the hyperparameter tuning job.

Monitor HyperDrive runs

You can monitor the progress of the runs with the following Jupyter widget.

Warm start a Hyperparameter Tuning experiment and resuming child runs

Often times, finding the best hyperparameter values for your model can be an iterative process, needing multiple tuning runs that learn from previous hyperparameter tuning runs. Reusing knowledge from these previous runs will accelerate the hyperparameter tuning process, thereby reducing the cost of tuning the model and will potentially improve the primary metric of the resulting model. When warm starting a hyperparameter tuning experiment with Bayesian sampling, trials from the previous run will be used as prior knowledge to intelligently pick new samples, so as to improve the primary metric. Additionally, when using Random or Grid sampling, any early termination decisions will leverage metrics from the previous runs to determine poorly performing training runs.

Azure Machine Learning allows you to warm start your hyperparameter tuning run by leveraging knowledge from up to 5 previously completed hyperparameter tuning parent runs.

Additionally, there might be occasions when individual training runs of a hyperparameter tuning experiment are cancelled due to budget constraints or fail due to other reasons. It is now possible to resume such individual training runs from the last checkpoint (assuming your training script handles checkpoints). Resuming an individual training run will use the same hyperparameter configuration and mount the storage used for that run. The training script should accept the "--resume-from" argument, which contains the checkpoint or model files from which to resume the training run. You can also resume individual runs as part of an experiment that spends additional budget on hyperparameter tuning. Any additional budget, after resuming the specified training runs is used for exploring additional configurations.

For more information on warm starting and resuming hyperparameter tuning runs, please refer to the Hyperparameter Tuning for Azure Machine Learning documentation

Find and register best model

When all jobs finish, we can find out the one that has the highest accuracy.

Now, let's list the model files uploaded during the run.

We can then register the folder (and all files in it) as a model named sklearn-iris under the workspace for deployment