SageMaker Data Wrangler Job Notebook

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook uses the Data Wrangler .flow file to submit a SageMaker Data Wrangler Job with the following steps:

• Push Data Wrangler .flow file to S3
• Parse the .flow file inputs, and create the argument dictionary to submit to a boto client
• Submit the ProcessingJob arguments and wait for Job completion

Optionally, the notebook also gives an example of starting a SageMaker XGBoost TrainingJob using the newly processed data.

Parameters

The following lists configurable parameters that are used throughout this notebook.

Upload Data

Upload the dataset that we will be transforming and training on to Amazon S3.

Push Flow to S3

Use the following cell to upload the Data Wrangler .flow file to Amazon S3 so that it can be used as an input to the processing job.

Create boto3 Processing Job arguments

This notebook submits a Processing Job using boto, which will require an argument dictionary to submit to the boto client. Below, utility methods are defined for creating Processing Job Inputs for the following sources: S3, Athena, and Redshift. Then the argument dictionary is generated using the parsed inputs and job configurations such as instance type.

Start ProcessingJob

Now, the Processing Job is submitted to a boto client. The status of the processing job is monitored with the boto client, and this notebook waits until the job is no longer 'InProgress'.

Kick off SageMaker Training Job (Optional)

Data Wrangler is a SageMaker tool for processing data to be used for Machine Learning. Now that the data has been processed, users will want to train a model using the data. The following shows an example of doing so using a popular algorithm XGBoost.

It is important to note that the following XGBoost objective ['binary', 'regression', 'multiclass'], hyperparameters, or content_type may not be suitable for the output data, and will require changes to train a proper model. Furthermore, for CSV training, the algorithm assumes that the target variable is in the first column. For more information on SageMaker XGBoost, please see https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html.

Find Training Data path

The below demonstrates how to recursively search the output directory to find the data location.

Next, the Training Job hyperparameters are set. For more information on XGBoost Hyperparameters, see https://xgboost.readthedocs.io/en/latest/parameter.html.

The TrainingJob configurations are set using the SageMaker Python SDK Estimator, and which is fit using the training data from the ProcessingJob that was run earlier.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.