Amazon SageMaker Feature Store: Feature Processor Introduction

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook demonstrates how to get started with Feature Processor using SageMaker python SDK, create feature groups, perform batch transformation and ingest processed input data to feature groups.

We first demonstrate how to use @feature-processor decorator to run the job locally and then show how to use @remote decorator to execute large batch transform and ingestion on SageMaker training job remotely. Besides, the SDK provides APIs to create scheduled pipelines based on transformation code.

If you would like to learn more about Feature Processor, see documentation Feature Processing for more info and examples.

Setup For Notebook

Setup Runtime Environment

First we create a new kernel to execute this notebook.

1. Launch a new terminal in the current image (the '$_' icon at the top of this notebook).
2. Execute the commands:

conda create --name feature-processing-py-3.9 python=3.9 -y
conda activate feature-processing-py-3.9
conda install ipykernel -y
conda install openjdk -y

3. Return to this notebook and select the kernel with Image: 'Data Science' and Kernel: 'feature-processing-py-3.9'

Alternatively If you run this notebook on SageMaker Studio, you can execute the following cell to install runtime dependencies.

To get the Feature Processor module, we need to reinstall the SageMaker python SDK along with extra dependencies.

Create Feature Groups

First we start by creating two feature groups. One feature group is used for storing raw car sales dataset which is located in data/car_data.csv. We create another feature group to store aggregated feature values after feature processing, for example average value of mileage, price and msrp.

@feature_processor

The following example demonstrates how to use the @feature_processor decorator to load data from Amazon S3 to a SageMaker Feature Group.

A @feature_processor decorated function automatically loads data from the configured inputs, applies the feature processing code and ingests the transformed data to a feature group.

@feature_processor + @remote

The following example demonstrates how to run your feature processing code remotely.

This is useful if you are working with large data sets that require hardware more powerful than locally available. You can decorate your code with the @remote decorator to run your local Python code as a single or multi-node distributed SageMaker training job. For more information on running your code as a SageMaker training job, see Run your local code as a SageMaker training job.

to_pipeline and schedule

The following example demonstrates how to operationalize your feature processor by promoting it to a SageMaker Pipeline and configuring a schedule to execute it on a regular basis. This example uses the aggregate function defined above. Note, in order to create a pipeline, please make sure your method is annotated by both @remote and @feature-processor decorators.

In the following example, we will create and schedule the pipeline using to_pipeline and schedule method. If you want to test the job before scheduling, you can use execute to start only one execution.

The SDK also provides two extra methods describe and list_pipelines for you to get insights about the pipeline info.

Explorating feature processing pipelines and ML Lineage.

You can track scheduled SageMaker Pipelines with SageMaker Lineage in Amazon SageMaker Studio. This includes tracking scheduled executions, visualizing lineage to trace features back to their data sources, and viewing shared feature processing code all in one environment.

Find the feature groups that were created in this notebook and view the Pipeline Executions and Lineage tabs.

Clean up Resources

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.