Parameterize spark configuration in pipeline PySparkProcessor execution

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Overview

In this example, we demonstrate how we can parameterize spark-configuration in different pipeline PySparkProcessor executions. This example is an extended version of Specifying additional Spark configuration example in Distributed Data Processing using Apache Spark and SageMaker Processing. Here we are creating a simple pipeline with one processing step to demonstrate spark-configuration parameterization capabilities in sagemaker pipeline PySparkProcessor. This could be useful to pipeline users who want to define different spark-configuraitons for different pipeline PySparkProcessor executions.

Prerequisites

To learn about how we can create pipeline, follow this tutorial

Pipeline Creation

The following is the step-by-step process to demonstrate parameterization capabilities in pipeline PySparkProcessor

Step-1: Install the latest SageMaker Python SDK

Restart your notebook kernel after upgrading the SDK

Step-2: Setup Environment

Create prefix for parametrize-spark-config-pysparkprocessor-demo

Step-3: Prepare Input Data

In this example, we process Abalone Data Set using PySpark script. We download the data locally and upload it to our Amazon S3 bucket for data processing.

Step-4: Upload default spark-configuraiton

Upload default spark-configuration to Amazon S3

Step-5: Define Pipeline Parameters

If no SparkConfigS3Uri is provided to the pipeline execution, the pipeline uses the pre-uploaded default_spark_configuration as a default spark-config.

Step-6: Write the PySpark script

We create a PySpark script similar to this example. The source for a preprocessing script is in the cell below. This script does some basic feature engineering on a raw input dataset. In this example, the dataset is the Abalone Data Set and the code below performs string indexing, one hot encoding, vector assembly, and combines them into a pipeline to perform these transformations in order. The script then does an 80-20 split to produce training and validation datasets as output.

Step-7: Create PySparkProcessor

Create an instance of an PySparkProcessor to pass in to the processing step.

Step-8: Create ProcessingStep and Pipeline

Create a processing step. This step takes in the PySparkProcessor, the input and output channels, and the ./code/preprocess.py script that we created.

Now, we have successfully created a sagemaker pipeline with a PySparkProcessor.

Pipeline Executions

Execute pipeline with default spark-configuration

If no SparkConfigS3Uri parameter value is provided, pipeline execution uses default_spark_configuration_object_s3_uri as a default spark-configuration. In the following execution example, we execute PySparkProcessor with default spark-configuration.

We can verify that PySparkProcessor is using the default spark-configuration by looking into the CloudWatch logs.

Execute pipeline with a new spark-configuraiton

We upload a new spark-configuration to Amazon S3 and use it in the next pipeline execution

We can verify that PySparkProcessor is using the newly provided spark-configuration by looking into the CloudWatch logs.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.