Sm App Autoscaling Realtime Endpoints Step Scaling
Faster autoscaling on Amazon SageMaker realtime endpoints (Step Scaling)
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
In this notebook we show how the new faster autoscaling feature helps scale sagemaker inference endpoints by almost 6x faster than earlier.
We deploy Meta's Llama3-8B-Instruct model to an Amazon SageMaker realtime endpoint using Text Generation Inference (TGI) Deep Learning Container (DLC) and apply Step Scaling autoscaling policies to the endpoint.
Please use at least m5.2xlarge or larger instance types if running this on Amazon SageMaker Notebook Instance.
Prerequisites
- Step 1: Create user access token in HuggingFace (HF). Refer here on how to create HF tokens.
- Step 2: Login to HuggingFace and navigate to Meta-Llama-3-8B-Instruct* home page.
- Step 3: Accept META LLAMA 3 COMMUNITY LICENSE AGREEMENT by following the instructions here
- Step 4: Wait for the approval email from META (Approval may take any where b/w 1-3 hrs)
Ensure python version of kernel is 3.10
Install packages using uv, an extremely fast python package installer. Read more about uv here https://astral.sh/blog/uv
Restart kernel after installing packages
Initiate sagemaker session
Deploy model
Create and deploy model using Amazon SageMaker HuggingFace TGI DLC
https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy
Refer here to learn about creating HF tokens.
Inference
Invoke and test endpoint using messages API. Refer to HF Messages API for more info.
Baseline average latency at various concurrency levels (Optional)
By capturing average latency across various concurrency levels, we can get a fair idea on after how many concurrent request does endpoint performance would degrade significantly.
Having this information can help define values for scaling policy accordingly.
At this concurrency level the endpoint gets overloaded and cannot serve requests in a timely fashion.
We use these values to set as threshold values for autoscaling.
Apply Step-Scaling autoscaling policies to endpoint
- Step 1: Register Scalable Target
- Step 2: Create Scale-Out Policy
- Step 3: Create Scale-In Policy
- Step 4: Create CloudWatch Alarms
Define and apply the step-scaling policy for scaling out.
Create StepScaling Scale-out Policy
Create StepScaling Scale-In Policy
Create CloudWatch alarms (Step-Scaling)
Create CloudWatch Alarms using new ConcurrentRequestsPerModel high-resolution Metric.
Trigger autoscaling action
Use LLMPerf to generate traffic to the endpoint
Refer to https://github.com/philschmid/llmperf for more details on LLMPerf.
Run the LLMPerf traffic generation script in the background using subprocess.Popen
Monitor Alarm Trigger times and Scaling event times
As llmperf generates traffic to the endpoint continuously this trigger auto-scaling.
The monitor_scaling_events function does the following:
- Calculates time taken for alarm to go into InAlarm state.
- checks if alarm is InAlarm state. If yes, then starts the scaling timer
- continuously monitors the
DesiredInstanceCountproperty of the endpoint- waits till
CurrentInstanceCount == DesiredInstanceCountandEndpointStatusisInService
- waits till
- Calculates time taken to scale out instances prints the times in a table
The below cell triggers auto scaling action and calls the monitor_scaling_events immediately on the AlarmHigh
Monitor if the background process (llmperf) is completed.
Print LLMPerf results
LLMPerf writes the results to "results/" directory. summary.json file has the endpoint benchmarking data.
Monitor Scale-in action scaling times (Optional)
Cleanup
- Delete cloudwatch alarms
- Delete scaling policies
- Deregister scalable target
- Delete model
- Delete endpoint
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.