SageMaker Pipelines with Amazon SageMaker Geospatial Capabilities

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook demonstrates how to create a SageMaker Pipeline to automate geospatial data processing workflows.

Amazon SageMaker Pipelines is a tool for building machine learning pipelines that take advantage of direct SageMaker integration. In this notebook, we will combine Amazon SageMaker geospatial capabilities with Amazon SageMaker Pipelines to create a reproducible and automated workflow which executes a chain of Earth Observation Jobs.

Prerequisites

General setup

This notebook runs with Kernel Geospatial 1.0. Note that the following policies need to be attached to the execution role that you used to run this notebook:

• AmazonSageMakerFullAccess
• AmazonSageMakerGeospatialFullAccess

You can see the policies attached to the role in the IAM console under the permissions tab. If required, add the roles using the 'Add Permissions' button.

Interactions with AWS Lambda, Amazon S3, and Amazon SQS

To allow deployment of the Lambda function and use the SageMaker geospatial services from within the notebook, the notebook's execution role needs to allow to assume the role. This can be done by adding the following trust policy using the 'Trust relationships' tab:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "sagemaker.amazonaws.com",
                    "sagemaker-geospatial.amazonaws.com",
                    "lambda.amazonaws.com",
                    "s3.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Additionaly the notebook's AWS IAM role needs to have the proper permissions for interacting with AWS Lambda, Amazon S3, and Amazon SQS as required.

For demo purposes, the following inline policy can be added. Please note that this policy should be scoped down to improve security for any production deployment, following the least privilege principle:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:CreateQueue"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:*",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Import SageMaker geospatial capabilities SDK

Setup Infrastructure

The following steps will setup the necessary infrastructure to execute a SageMaker Pipeline using geospatial capabilities. Earth Observation Jobs (EOJs) are executed asynchronous and the pipeline needs to be able to check the status via callbacks.

For this, we'll setup a Lambda function and a SQS queue which are both later used in the pipeline.

This infrastructure needs to be created only once and can then be referenced in multiple pipelines.

Create Lambda layer

Before we create the Lambda function, we'll create a Lambda layer which contains a recent boto3 version which is necessary to use the sagemaker-geospatial API.

The following steps are packaging the layer code, uploading the code package to S3 and creating the actual Lambda layer.

Create Lambda function

After the layer has been created, we will create the actual Lambda function. The code for Lambda function is located at assets/eoj_pipeline_lambda.py.

The Lambda function will provide functionality to start EOJs and also check their status. It will be used in combination with an Amazon SQS queue.

The following cell will create the Lambda function geospatial-eoj-pipeline-lambda.

Create SQS queue

After creating the Lambda function to invoke EOJs, we will create an Amazon SQS queue for asynchronous callbacks.

As Earth Observation Jobs (EOJs) are executed in an asynchronous fashion, the SQS queue is needed to check the EOJ status within the pipeline asynchronously.

The following cell will create an Amazon SQS queue which will be later used in the pipeline.

Now we need to associate the SQS queue as an input trigger for our Lambda function, in this way whenever the Callback Step pushes a message to the queue it would run our Lambda function for checking the status of the EOJ. We do this by creating an Event Source Mapping between the Lambda function and the SQS queue.

Pipeline Creation

After the necessary infrastructure has been deployed, we can build the pipeline.

We will define the parameters which are used to configure our pipeline dynamically whenever we execute the pipeline.

The code below allows the creation of a desired amount of EOJ steps within the pipeline. In this example, we will create 3 EOJ steps and one additional export step. By adapting the parameter eoj_step_count, the amount of EOJ steps can be changed.

Next, we will create as many EOJ steps, as provided in eoj_step_count. Each iteration will create a LambdaStep and a CallbackStep which depends on the former.

Then, we will create additional steps which enable the export of the EOJ.

Finally, we can define our pipeline based on the steps and parameters created before.

After the pipeline has been created, you are able to inspect the created pipeline.

For this, you can navigate to the SageMaker Studio Resources tab in the left menu and click on Pipelines.

You should be able to see the "GeospatialEarthObservationPipeline" in the list. You can click on it and then navigate to the Graph tab to see a visual representation of the created pipeline.

Pipeline execution

The following cell is defining the configuration for the pipeline and finally runs a pipeline execution.

If you adapted the eoj_step_count in the previous cells, you must ensure that eoj_configs contain as many EOJ configurations as you have steps.

After the pipeline execution has been started, you can see it as well in the Pipelines UI.

Navigate back to the "GeospatialEarthObservationPipeline" in the Pipelines UI and you can see the execution in the Executions tab. You can double-click it to see the details of this execution.

Alternative way to execute the created Pipeline

The Pipeline exection example in the previous cell needs an initialized Pipeline object to run a pipeline execution. If you want to execute the pipeline from within other notebooks, this can be done in the following way.

Clean-up

When the created pipeline and underlying infrastructure is no longer needed, uncomment and run the following cells for deleting the previously created resources.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.