Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

How to use CommandStep in Azure ML Pipelines

This notebook shows how to use the CommandStep with Azure Machine Learning Pipelines for running commands in steps. The example shows running distributed TensorFlow training from within a pipeline.

Prerequisite:

• Understand the architecture and terms introduced by Azure Machine Learning
• If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the configuration notebook to:
  • install the Azure ML SDK
  • create a workspace and its configuration file (config.json)

Let's get started. First let's import some Python libraries.

Initialize workspace

Initialize a Workspace object from the existing workspace you created in the Prerequisites step. Workspace.from_config() creates a workspace object from the details stored in config.json.

Create or Attach existing AmlCompute

You will need to create a compute target for training your model. In this tutorial, you create AmlCompute as your training compute resource.

Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.

If we could not find the cluster with the given name, then we will create a new cluster here. We will create an AmlCompute cluster of Standard_NC6s_v3 GPU VMs. This process is broken down into 3 steps:

1. create the configuration (this step is local and only takes a second)
2. create the cluster (this step will take about 20 seconds)
3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about 3-5 minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell

Now that you have created the compute target, let's see what the workspace's compute_targets property returns. You should now see one entry named 'gpu-cluster' of type AmlCompute.

Create a CommandStep

CommandStep adds a step to run a command in a Pipeline. For the full set of configurable options see the CommandStep reference docs.

• name: Name of the step
• runconfig: ScriptRunConfig object. You can configure a ScriptRunConfig object as you would for a standalone non-pipeline run and pass it in to this parameter. If using this option, you do not have to specify the command, source_directory, compute_target parameters of the CommandStep constructor as they are already defined in your ScriptRunConfig.
• runconfig_pipeline_params: Override runconfig properties at runtime using key-value pairs each with name of the runconfig property and PipelineParameter for that property
• command: The command to run or path of the executable/script relative to source_directory. It is required unless the runconfig parameter is specified. It can be specified with string arguments in a single string or with input/output/PipelineParameter in a list.
• source_directory: A folder containing the script and other resources used in the step.
• compute_target: Compute target to use
• allow_reuse: Whether the step should reuse previous results when run with the same settings/inputs. If this is false, a new run will always be generated for this step during pipeline execution.
• version: Optional version tag to denote a change in functionality for the step

The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the source_directory for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the source_directory would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the source_directory of the step.

First define the environment that you want to step to run in. This example users a curated TensorFlow environment, but in practice you can configure any environment you want.

This example will first create a ScriptRunConfig object that configures the training job. Since we are running a distributed job, specify the distributed_job_config parameter. If you are just running a single-node job, omit that parameter.

If you have an input dataset you want to use in this step, you can specify that as part of the command. For example, if you have a FileDataset object called dataset and a --data-dir script argument, you can do the following: command=['python train.py --epochs 30 --data-dir', dataset.as_mount()].

For detailed guidance on how to move data in pipelines for input and output data, see the documentation Moving data into and between ML pipelines.

Now create a CommandStep and pass in the ScriptRunConfig object to the runconfig parameter.

Build and Submit the Pipeline

View Run Details