Amazon SageMaker Processing jobs

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

With Amazon SageMaker Processing jobs, you can leverage a simplified, managed experience to run data pre- or post-processing and model evaluation workloads on the Amazon SageMaker platform.

A processing job downloads input from Amazon Simple Storage Service (Amazon S3), then uploads outputs to Amazon S3 during or after the processing job.

This notebook shows how you can:

1. Run a processing job to run a scikit-learn script that cleans, pre-processes, performs feature engineering, and splits the input data into train and test sets.
2. Run a training job on the pre-processed training data to train a model
3. Run a processing job on the pre-processed test data to evaluate the trained model's performance
4. Use your own custom container to run processing jobs with your own Python libraries and dependencies.

The dataset used here is the Census-Income KDD Dataset. You select features from this dataset, clean the data, and turn the data into features that the training algorithm can use to train a binary classification model, and split the data into train and test sets. The task is to predict whether rows representing census responders have an income greater than $50,000, or less than $50,000. The dataset is heavily class imbalanced, with most records being labeled as earning less than $50,000. After training a logistic regression model, you evaluate the model against a hold-out test dataset, and save the classification evaluation metrics, including precision, recall, and F1 score for each label, and accuracy and ROC AUC for the model.

Data pre-processing and feature engineering

To run the scikit-learn preprocessing script as a processing job, create a SKLearnProcessor, which lets you run scripts inside of processing jobs using the scikit-learn image provided.

Before introducing the script you use for data cleaning, pre-processing, and feature engineering, inspect the first 20 rows of the dataset. The target is predicting the income category. The features from the dataset you select are age, education, major industry code, class of worker, num persons worked for employer, capital gains, capital losses, and dividends from stocks.

This notebook cell writes a file preprocessing.py, which contains the pre-processing script. You can update the script, and rerun this cell to overwrite preprocessing.py. You run this as a processing job in the next cell. In this script, you

• Remove duplicates and rows with conflicting data
• transform the target income column into a column containing two labels.
• transform the age and num persons worked for employer numerical columns into categorical features by binning them
• scale the continuous capital gains, capital losses, and dividends from stocks so they're suitable for training
• encode the education, major industry code, class of worker so they're suitable for training
• split the data into training and test datasets, and saves the training features and labels and test features and labels.

Our training script will use the pre-processed training features and labels to train a model, and our model evaluation script will use the trained model and pre-processed test features and labels to evaluate the model.

Run this script as a processing job. Use the SKLearnProcessor.run() method. You give the run() method one ProcessingInput where the source is the census dataset in Amazon S3, and the destination is where the script reads this data from, in this case /opt/ml/processing/input. These local paths inside the processing container must begin with /opt/ml/processing/.

Also give the run() method a ProcessingOutput, where the source is the path the script writes output data to. For outputs, the destination defaults to an S3 bucket that the Amazon SageMaker Python SDK creates for you, following the format s3://sagemaker-<region>-<account_id>/<processing_job_name>/output/<output_name/. You also give the ProcessingOutputs values for output_name, to make it easier to retrieve these output artifacts after the job is run.

The arguments parameter in the run() method are command-line arguments in our preprocessing.py script.

Now inspect the output of the pre-processing job, which consists of the processed features.

Training using the pre-processed data

We create a SKLearn instance, which we will use to run a training job using the training script train.py.

The training script train.py trains a logistic regression model on the training data, and saves the model to the /opt/ml/model directory, which Amazon SageMaker tars and uploads into a model.tar.gz file into S3 at the end of the training job.

Run the training job using train.py on the preprocessed training data.

Model Evaluation

evaluation.py is the model evaluation script. Since the script also runs using scikit-learn as a dependency, run this using the SKLearnProcessor you created previously. This script takes the trained model and the test dataset as input, and produces a JSON file containing classification evaluation metrics, including precision, recall, and F1 score for each label, and accuracy and ROC AUC for the model.

Now retrieve the file evaluation.json from Amazon S3, which contains the evaluation report.

Running processing jobs with FrameworkProcessor to include custom dependencies

Above, you used a processing container that has scikit-learn installed, but there was no way to add dependencies to the processing container.

A new sub-class of existing ScriptProcessor called FrameworkProcessor has been added to SageMaker SDK which allows to specify source_dir using which requirements.txt file can be included. The processor object will first install listed dependencies inside the target container prior to triggering processing job.

Below is a simple example of how to create a processing container, and how to use a FrameworkProcessor to run your own code within a container. Create a scikit-learn container and run a processing job using the same preprocessing.py script you used above. You can provide your own dependencies inside this container to run your processing script with source_dir and requirements.txt.

Run the same preprocessing.py script you ran above, but now, this code is running inside the container that now has additional dependencies installed. You can update the processing script to use these dependencies and perform custom processing. This approach can be applied to your own pre-processing, feature-engineering, and model evaluation scripts inside of this container.

First let's copy the previously created preprocessing.py script to code folder for next run method

Summary

As you can see all the dependencies from requirements.txt file have been installed on the target container prior to starting the processing script.
FrameworkProcessor makes it easy to use existing built-in framework processors like SciKit Learn, MXNet, PyTorch etc along with specifying custom dependencies for processing script.

(Optional)Next Steps

As an optional step, this notebook shows an example below to build custom container from scratch using docker.

By default optional steps run automatically, set run_optional_steps to False if you don't want to execute optional steps

(Optional) Running processing jobs with your own dependencies

Above, you used a processing container that has scikit-learn installed, but you can run your own processing container in your processing job as well, and still provide a script to run within your processing container.

Below, you walk through how to create a processing container, and how to use a ScriptProcessor to run your own code within a container. Create a scikit-learn container and run a processing job using the same preprocessing.py script you used above. You can provide your own dependencies inside this container to run your processing script with.

This is the Dockerfile to create the processing container. Install pandas and scikit-learn into it. You can install your own dependencies.

This block of code builds the container using the docker command, creates an Amazon Elastic Container Registry (Amazon ECR) repository, and pushes the image to Amazon ECR.

The ScriptProcessor class lets you run a command inside this container, which you can use to run your own script.

Run the same preprocessing.py script you ran above, but now, this code is running inside of the Docker container you built in this notebook, not the scikit-learn image maintained by Amazon SageMaker. You can add the dependencies to the Docker image, and run your own pre-processing, feature-engineering, and model evaluation scripts inside of this container.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.