Deploy a GPTQ Quantized model on SageMaker with Hugging Face TGI

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook we will deploy a pre-quantized 13 billion parameter Llama 2 Chat model.

The original models is stored and served in half-precision fp16 format which translates to 2 bytes per parameter. Given that the model has 13 billion parameters, the model size translates 26GB which is too large to fit in the memory of a single A10 GPU which has only 24GB of memory. This requires us to use a more expensive multi-gpu instance such as a ml.g5.12xlarge. An alternative is to quantize the model which can significantly reduce the amount of VRAM required to host the model.

In this notebook, we will deploy a 13 billion parameter model that has been pre-quantized to 4-bits using GPTQ algorithm. With 4bit quantization the amount of memory per parameter is reduced from 2 bytes to 4 bits (0.5 bytes) which translates to a 75% reduction in memory footprint. This allows us to host the model on a single A10 GPU instance, such as a ml.g5.xlarge, which is significantly cheaper than a multi-gpu instance. As a disclaimer, quantization does result in a slight drop in model accuracy. However, the drop in accuracy is small and the model is still able to generate coherent responses, but it is important to evaluate the model on your use case.

*Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved

We will be using the Hugging Face Text Generation Inference (TGI) Container which runs the optimized TGI LLM hosting solution from HuggingFace

The helper function below will deploy the model to a SageMaker endpoint.

The repo maintained by TheBloke contains many pre-quantized models. You can also easily quantize your own models with GPTQ by using the AutoGPTQ library.

Once the endpoint is deployed we can invoke it using the boto3 SDK. We will use the Llama2 recommended prompt format along with a user instruction to create the final prompt which we will pass as a payload to the model.

Run the code below to delete the endpoint and avoid any additional charges.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.