Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

04. Train in a remote Linux VM

• Create Workspace
• Create train.py file
• Create and Attach a Remote VM (eg. DSVM) as compute resource
• Upload data files into default datastore
• Configure & execute a run in a few different ways
  • Use system-built conda
  • Use existing Python environment
  • Use Docker
• Find the best model in the run

Prerequisites

If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the configuration notebook first if you haven't already to establish your connection to the AzureML Workspace.

Initialize Workspace

Initialize a workspace object from persisted configuration.

Create Experiment

Experiment is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments.

Let's also create a local folder to hold the training script.

Upload data files into datastore

Every workspace comes with a default datastore (and you can register more) which is backed by the Azure blob storage account associated with the workspace. We can use it to transfer data from local to the cloud, and access it from the compute target.

Load diabetes data from scikit-learn and save it as 2 local files.

Now let's upload the 2 files into the default datastore under a path named diabetes:

Create a Dataset for Files

A Dataset can reference single or multiple files in your datastores or public urls. The files can be of any format. Dataset provides you with the ability to download or mount the files to your compute. By creating a dataset, you create a reference to the data source location. The data remains in its existing location, so no extra storage cost is incurred. Learn More

View train.py

For convenience, we created a training script for you. It is printed below as a text, but you can also run %pfile ./train.py in a cell to show the file. Please pay special attention on how we are loading the features and labels from files in the data_folder path, which is passed in as an argument of the training script (shown later).

Create and Attach a DSVM as a compute target

The DSVM can be created using the below single line command and then attached(like any VM) using the sample code below. Also note, that we only support Linux VMs for remote execution from AML and the commands below will spin a Linux VM only.

	# create a DSVM in your resource group
# note you need to be at least a contributor to the resource group in order to execute this command successfully
(myenv) $ az vm create --resource-group <resource_group_name> --name <some_vm_name> --image microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:latest --admin-username <username> --admin-password <password> --generate-ssh-keys --authentication-type password

Note: You can also use this url to create the VM using the Azure Portal

Note: By default SSH runs on port 22 and you don't need to specify it. But if for security reasons you switch to a different port (such as 5022), you can specify the port number in the provisioning configuration object.

Configure & Run

Now we can try a few different ways to run the training script in the VM.

Conda run

You can ask the system to build a conda environment based on your dependency specification, and submit your script to run there. Once the environment is built, and if you don't change your dependencies, it will be reused in subsequent runs.

Note: if you need to cancel a run, you can follow these instructions.

Show the run object. You can navigate to the Azure portal to see detailed information about the run.

Native VM run

You can also configure to use an exiting Python environment in the VM to execute the script without asking the system to create a conda environment for you.

The below run will likely fail because train.py needs dependency azureml, scikit-learn and others, which are not found in that Python environment.

You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script train2.py that doesn't have azureml or data store dependencies, and submit it instead.

Now let's try again. And this time it should work fine.

Note even in this case you get a run record with some basic statistics.

Configure a Docker run with new conda environment on the VM

You can execute in a Docker container in the VM. If you choose this option, the system will pull down a base Docker image, build a new conda environment in it if you ask for (you can also skip this if you are using a customer Docker image when a preconfigured Python environment), start a container, and run your script in there. This image is also uploaded into your ACR (Azure Container Registry) assoicated with your workspace, an reused if your dependencies don't change in the subsequent runs.

Submit the Experiment

Submit script to run in the Docker image in the remote VM. If you run this for the first time, the system will download the base image, layer in packages specified in the conda_dependencies.yml file on top of the base image, create a container and then execute the script in the container.

Use a custom Docker image instead

You can also specify a custom Docker image if you don't want to use the default image provided by Azure ML.

	# use an image available in Docker Hub without authentication
conda_env.docker.base_image = "continuumio/miniconda3"

# or, use an image available in a private Azure Container Registry
conda_env.docker.base_image = "mycustomimage:1.0"
conda_env.docker.base_image_registry.address = "myregistry.azurecr.io"
conda_env.docker.base_image_registry.username = "username"
conda_env.docker.base_image_registry.password = "password"

When you are using a custom Docker image, you might already have your environment setup properly in a Python environment in the Docker image. In that case, you can skip specifying conda dependencies, and just use user_managed_dependencies option instead:

	conda_env.python.user_managed_dependencies = True
# path to the Python environment in the custom Docker image
conda_env.python.interpreter_path = '/opt/conda/bin/python'

View run history details

Find the best model

Now we have tried various execution modes, we can find the best model from the last run.

Clean up compute resource

Use detach() to detach an existing DSVM from Workspace without deleting it. Use delete() if you created a new DsvmCompute and want to delete it.