Mistral AI Gptq

Gptq

mistral-cookbookmethodsquantizationconcept-deep-dive

alph-notebooks/mistral-cookbook / gptq.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

GPTQ

GPTQ is a quantization format optimized for GPU inference. It makes use of a calibration dataset to improve its quantizations.

Quantizing with AutoGPTQ

Let's do a short demo and quantize Mistral 7B.

First, we install auto-gptq. It will allow us to easily quantize and infer GPTQ models.

[4]

Requirement already satisfied: auto-gptq in /usr/local/lib/python3.10/dist-packages (0.7.1)
Requirement already satisfied: accelerate>=0.26.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (0.32.1)
Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (2.21.0)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (0.1.99)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (1.26.4)
Requirement already satisfied: rouge in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (1.0.1)
Requirement already satisfied: gekko in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (1.2.1)
Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (2.3.1+cu121)
Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (0.4.4)
Requirement already satisfied: transformers>=4.31.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (4.42.4)
Requirement already satisfied: peft>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (0.12.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from auto-gptq) (4.66.5)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->auto-gptq) (24.1)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->auto-gptq) (5.9.5)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->auto-gptq) (6.0.2)
Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->auto-gptq) (0.23.5)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (1.13.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (2024.6.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (12.1.105)
Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq) (2.3.1)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.13.0->auto-gptq) (12.6.20)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.31.0->auto-gptq) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.31.0->auto-gptq) (2.32.3)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.31.0->auto-gptq) (0.19.1)
Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (17.0.0)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (0.3.8)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (2.1.4)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (3.5.0)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (0.70.16)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq) (3.10.3)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from rouge->auto-gptq) (1.16.0)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (2.3.5)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (24.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq) (4.0.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.31.0->auto-gptq) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.31.0->auto-gptq) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.31.0->auto-gptq) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.31.0->auto-gptq) (2024.7.4)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.13.0->auto-gptq) (2.1.5)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->auto-gptq) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->auto-gptq) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->auto-gptq) (2024.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.0->auto-gptq) (1.3.0)

Once we're done, we can download the model we want to quantize. First, let's log in with a read access token so we have access to the models.

Note: You need to first accept the terms in the repo.

[5]

The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: read).
Your token has been saved to /root/.cache/huggingface/token
Login successful

Now everything is ready, so we can load the model and quantize it! Here, we will quantize the model to 4-bit!

[7]

tokenizer_config.json:   0%|          | 0.00/141k [00:00<?, ?B/s]

tokenizer.model:   0%|          | 0.00/587k [00:00<?, ?B/s]

tokenizer.json:   0%|          | 0.00/1.96M [00:00<?, ?B/s]

special_tokens_map.json:   0%|          | 0.00/414 [00:00<?, ?B/s]

config.json:   0%|          | 0.00/601 [00:00<?, ?B/s]

/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(

model.safetensors.index.json:   0%|          | 0.00/23.9k [00:00<?, ?B/s]

Downloading shards:   0%|          | 0/3 [00:00<?, ?it/s]

model-00001-of-00003.safetensors:   0%|          | 0.00/4.95G [00:00<?, ?B/s]

model-00002-of-00003.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]

model-00003-of-00003.safetensors:   0%|          | 0.00/4.55G [00:00<?, ?B/s]

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]

generation_config.json:   0%|          | 0.00/116 [00:00<?, ?B/s]

INFO - Start quantizing layer 1/32
INFO:auto_gptq.modeling._base:Start quantizing layer 1/32
INFO - Quantizing self_attn.k_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 1/32...
INFO - Quantizing self_attn.v_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 1/32...
INFO - Quantizing self_attn.q_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 1/32...
INFO - Quantizing self_attn.o_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 1/32...
INFO - Quantizing mlp.up_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 1/32...
INFO - Quantizing mlp.gate_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 1/32...
INFO - Quantizing mlp.down_proj in layer 1/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 1/32...
INFO - Start quantizing layer 2/32
INFO:auto_gptq.modeling._base:Start quantizing layer 2/32
INFO - Quantizing self_attn.k_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 2/32...
INFO - Quantizing self_attn.v_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 2/32...
INFO - Quantizing self_attn.q_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 2/32...
INFO - Quantizing self_attn.o_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 2/32...
INFO - Quantizing mlp.up_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 2/32...
INFO - Quantizing mlp.gate_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 2/32...
INFO - Quantizing mlp.down_proj in layer 2/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 2/32...
INFO - Start quantizing layer 3/32
INFO:auto_gptq.modeling._base:Start quantizing layer 3/32
INFO - Quantizing self_attn.k_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 3/32...
INFO - Quantizing self_attn.v_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 3/32...
INFO - Quantizing self_attn.q_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 3/32...
INFO - Quantizing self_attn.o_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 3/32...
INFO - Quantizing mlp.up_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 3/32...
INFO - Quantizing mlp.gate_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 3/32...
INFO - Quantizing mlp.down_proj in layer 3/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 3/32...
INFO - Start quantizing layer 4/32
INFO:auto_gptq.modeling._base:Start quantizing layer 4/32
INFO - Quantizing self_attn.k_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 4/32...
INFO - Quantizing self_attn.v_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 4/32...
INFO - Quantizing self_attn.q_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 4/32...
INFO - Quantizing self_attn.o_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 4/32...
INFO - Quantizing mlp.up_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 4/32...
INFO - Quantizing mlp.gate_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 4/32...
INFO - Quantizing mlp.down_proj in layer 4/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 4/32...
INFO - Start quantizing layer 5/32
INFO:auto_gptq.modeling._base:Start quantizing layer 5/32
INFO - Quantizing self_attn.k_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 5/32...
INFO - Quantizing self_attn.v_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 5/32...
INFO - Quantizing self_attn.q_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 5/32...
INFO - Quantizing self_attn.o_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 5/32...
INFO - Quantizing mlp.up_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 5/32...
INFO - Quantizing mlp.gate_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 5/32...
INFO - Quantizing mlp.down_proj in layer 5/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 5/32...
INFO - Start quantizing layer 6/32
INFO:auto_gptq.modeling._base:Start quantizing layer 6/32
INFO - Quantizing self_attn.k_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 6/32...
INFO - Quantizing self_attn.v_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 6/32...
INFO - Quantizing self_attn.q_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 6/32...
INFO - Quantizing self_attn.o_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 6/32...
INFO - Quantizing mlp.up_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 6/32...
INFO - Quantizing mlp.gate_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 6/32...
INFO - Quantizing mlp.down_proj in layer 6/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 6/32...
INFO - Start quantizing layer 7/32
INFO:auto_gptq.modeling._base:Start quantizing layer 7/32
INFO - Quantizing self_attn.k_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 7/32...
INFO - Quantizing self_attn.v_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 7/32...
INFO - Quantizing self_attn.q_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 7/32...
INFO - Quantizing self_attn.o_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 7/32...
INFO - Quantizing mlp.up_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 7/32...
INFO - Quantizing mlp.gate_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 7/32...
INFO - Quantizing mlp.down_proj in layer 7/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 7/32...
INFO - Start quantizing layer 8/32
INFO:auto_gptq.modeling._base:Start quantizing layer 8/32
INFO - Quantizing self_attn.k_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 8/32...
INFO - Quantizing self_attn.v_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 8/32...
INFO - Quantizing self_attn.q_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 8/32...
INFO - Quantizing self_attn.o_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 8/32...
INFO - Quantizing mlp.up_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 8/32...
INFO - Quantizing mlp.gate_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 8/32...
INFO - Quantizing mlp.down_proj in layer 8/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 8/32...
INFO - Start quantizing layer 9/32
INFO:auto_gptq.modeling._base:Start quantizing layer 9/32
INFO - Quantizing self_attn.k_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 9/32...
INFO - Quantizing self_attn.v_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 9/32...
INFO - Quantizing self_attn.q_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 9/32...
INFO - Quantizing self_attn.o_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 9/32...
INFO - Quantizing mlp.up_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 9/32...
INFO - Quantizing mlp.gate_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 9/32...
INFO - Quantizing mlp.down_proj in layer 9/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 9/32...
INFO - Start quantizing layer 10/32
INFO:auto_gptq.modeling._base:Start quantizing layer 10/32
INFO - Quantizing self_attn.k_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 10/32...
INFO - Quantizing self_attn.v_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 10/32...
INFO - Quantizing self_attn.q_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 10/32...
INFO - Quantizing self_attn.o_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 10/32...
INFO - Quantizing mlp.up_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 10/32...
INFO - Quantizing mlp.gate_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 10/32...
INFO - Quantizing mlp.down_proj in layer 10/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 10/32...
INFO - Start quantizing layer 11/32
INFO:auto_gptq.modeling._base:Start quantizing layer 11/32
INFO - Quantizing self_attn.k_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 11/32...
INFO - Quantizing self_attn.v_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 11/32...
INFO - Quantizing self_attn.q_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 11/32...
INFO - Quantizing self_attn.o_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 11/32...
INFO - Quantizing mlp.up_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 11/32...
INFO - Quantizing mlp.gate_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 11/32...
INFO - Quantizing mlp.down_proj in layer 11/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 11/32...
INFO - Start quantizing layer 12/32
INFO:auto_gptq.modeling._base:Start quantizing layer 12/32
INFO - Quantizing self_attn.k_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 12/32...
INFO - Quantizing self_attn.v_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 12/32...
INFO - Quantizing self_attn.q_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 12/32...
INFO - Quantizing self_attn.o_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 12/32...
INFO - Quantizing mlp.up_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 12/32...
INFO - Quantizing mlp.gate_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 12/32...
INFO - Quantizing mlp.down_proj in layer 12/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 12/32...
INFO - Start quantizing layer 13/32
INFO:auto_gptq.modeling._base:Start quantizing layer 13/32
INFO - Quantizing self_attn.k_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 13/32...
INFO - Quantizing self_attn.v_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 13/32...
INFO - Quantizing self_attn.q_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 13/32...
INFO - Quantizing self_attn.o_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 13/32...
INFO - Quantizing mlp.up_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 13/32...
INFO - Quantizing mlp.gate_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 13/32...
INFO - Quantizing mlp.down_proj in layer 13/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 13/32...
INFO - Start quantizing layer 14/32
INFO:auto_gptq.modeling._base:Start quantizing layer 14/32
INFO - Quantizing self_attn.k_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 14/32...
INFO - Quantizing self_attn.v_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 14/32...
INFO - Quantizing self_attn.q_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 14/32...
INFO - Quantizing self_attn.o_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 14/32...
INFO - Quantizing mlp.up_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 14/32...
INFO - Quantizing mlp.gate_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 14/32...
INFO - Quantizing mlp.down_proj in layer 14/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 14/32...
INFO - Start quantizing layer 15/32
INFO:auto_gptq.modeling._base:Start quantizing layer 15/32
INFO - Quantizing self_attn.k_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 15/32...
INFO - Quantizing self_attn.v_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 15/32...
INFO - Quantizing self_attn.q_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 15/32...
INFO - Quantizing self_attn.o_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 15/32...
INFO - Quantizing mlp.up_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 15/32...
INFO - Quantizing mlp.gate_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 15/32...
INFO - Quantizing mlp.down_proj in layer 15/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 15/32...
INFO - Start quantizing layer 16/32
INFO:auto_gptq.modeling._base:Start quantizing layer 16/32
INFO - Quantizing self_attn.k_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 16/32...
INFO - Quantizing self_attn.v_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 16/32...
INFO - Quantizing self_attn.q_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 16/32...
INFO - Quantizing self_attn.o_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 16/32...
INFO - Quantizing mlp.up_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 16/32...
INFO - Quantizing mlp.gate_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 16/32...
INFO - Quantizing mlp.down_proj in layer 16/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 16/32...
INFO - Start quantizing layer 17/32
INFO:auto_gptq.modeling._base:Start quantizing layer 17/32
INFO - Quantizing self_attn.k_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 17/32...
INFO - Quantizing self_attn.v_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 17/32...
INFO - Quantizing self_attn.q_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 17/32...
INFO - Quantizing self_attn.o_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 17/32...
INFO - Quantizing mlp.up_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 17/32...
INFO - Quantizing mlp.gate_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 17/32...
INFO - Quantizing mlp.down_proj in layer 17/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 17/32...
INFO - Start quantizing layer 18/32
INFO:auto_gptq.modeling._base:Start quantizing layer 18/32
INFO - Quantizing self_attn.k_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 18/32...
INFO - Quantizing self_attn.v_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 18/32...
INFO - Quantizing self_attn.q_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 18/32...
INFO - Quantizing self_attn.o_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 18/32...
INFO - Quantizing mlp.up_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 18/32...
INFO - Quantizing mlp.gate_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 18/32...
INFO - Quantizing mlp.down_proj in layer 18/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 18/32...
INFO - Start quantizing layer 19/32
INFO:auto_gptq.modeling._base:Start quantizing layer 19/32
INFO - Quantizing self_attn.k_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 19/32...
INFO - Quantizing self_attn.v_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 19/32...
INFO - Quantizing self_attn.q_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 19/32...
INFO - Quantizing self_attn.o_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 19/32...
INFO - Quantizing mlp.up_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 19/32...
INFO - Quantizing mlp.gate_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 19/32...
INFO - Quantizing mlp.down_proj in layer 19/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 19/32...
INFO - Start quantizing layer 20/32
INFO:auto_gptq.modeling._base:Start quantizing layer 20/32
INFO - Quantizing self_attn.k_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 20/32...
INFO - Quantizing self_attn.v_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 20/32...
INFO - Quantizing self_attn.q_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 20/32...
INFO - Quantizing self_attn.o_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 20/32...
INFO - Quantizing mlp.up_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 20/32...
INFO - Quantizing mlp.gate_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 20/32...
INFO - Quantizing mlp.down_proj in layer 20/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 20/32...
INFO - Start quantizing layer 21/32
INFO:auto_gptq.modeling._base:Start quantizing layer 21/32
INFO - Quantizing self_attn.k_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 21/32...
INFO - Quantizing self_attn.v_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 21/32...
INFO - Quantizing self_attn.q_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 21/32...
INFO - Quantizing self_attn.o_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 21/32...
INFO - Quantizing mlp.up_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 21/32...
INFO - Quantizing mlp.gate_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 21/32...
INFO - Quantizing mlp.down_proj in layer 21/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 21/32...
INFO - Start quantizing layer 22/32
INFO:auto_gptq.modeling._base:Start quantizing layer 22/32
INFO - Quantizing self_attn.k_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 22/32...
INFO - Quantizing self_attn.v_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 22/32...
INFO - Quantizing self_attn.q_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 22/32...
INFO - Quantizing self_attn.o_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 22/32...
INFO - Quantizing mlp.up_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 22/32...
INFO - Quantizing mlp.gate_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 22/32...
INFO - Quantizing mlp.down_proj in layer 22/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 22/32...
INFO - Start quantizing layer 23/32
INFO:auto_gptq.modeling._base:Start quantizing layer 23/32
INFO - Quantizing self_attn.k_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 23/32...
INFO - Quantizing self_attn.v_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 23/32...
INFO - Quantizing self_attn.q_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 23/32...
INFO - Quantizing self_attn.o_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 23/32...
INFO - Quantizing mlp.up_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 23/32...
INFO - Quantizing mlp.gate_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 23/32...
INFO - Quantizing mlp.down_proj in layer 23/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 23/32...
INFO - Start quantizing layer 24/32
INFO:auto_gptq.modeling._base:Start quantizing layer 24/32
INFO - Quantizing self_attn.k_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 24/32...
INFO - Quantizing self_attn.v_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 24/32...
INFO - Quantizing self_attn.q_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 24/32...
INFO - Quantizing self_attn.o_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 24/32...
INFO - Quantizing mlp.up_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 24/32...
INFO - Quantizing mlp.gate_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 24/32...
INFO - Quantizing mlp.down_proj in layer 24/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 24/32...
INFO - Start quantizing layer 25/32
INFO:auto_gptq.modeling._base:Start quantizing layer 25/32
INFO - Quantizing self_attn.k_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 25/32...
INFO - Quantizing self_attn.v_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 25/32...
INFO - Quantizing self_attn.q_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 25/32...
INFO - Quantizing self_attn.o_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 25/32...
INFO - Quantizing mlp.up_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 25/32...
INFO - Quantizing mlp.gate_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 25/32...
INFO - Quantizing mlp.down_proj in layer 25/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 25/32...
INFO - Start quantizing layer 26/32
INFO:auto_gptq.modeling._base:Start quantizing layer 26/32
INFO - Quantizing self_attn.k_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 26/32...
INFO - Quantizing self_attn.v_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 26/32...
INFO - Quantizing self_attn.q_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 26/32...
INFO - Quantizing self_attn.o_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 26/32...
INFO - Quantizing mlp.up_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 26/32...
INFO - Quantizing mlp.gate_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 26/32...
INFO - Quantizing mlp.down_proj in layer 26/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 26/32...
INFO - Start quantizing layer 27/32
INFO:auto_gptq.modeling._base:Start quantizing layer 27/32
INFO - Quantizing self_attn.k_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 27/32...
INFO - Quantizing self_attn.v_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 27/32...
INFO - Quantizing self_attn.q_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 27/32...
INFO - Quantizing self_attn.o_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 27/32...
INFO - Quantizing mlp.up_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 27/32...
INFO - Quantizing mlp.gate_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 27/32...
INFO - Quantizing mlp.down_proj in layer 27/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 27/32...
INFO - Start quantizing layer 28/32
INFO:auto_gptq.modeling._base:Start quantizing layer 28/32
INFO - Quantizing self_attn.k_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 28/32...
INFO - Quantizing self_attn.v_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 28/32...
INFO - Quantizing self_attn.q_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 28/32...
INFO - Quantizing self_attn.o_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 28/32...
INFO - Quantizing mlp.up_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 28/32...
INFO - Quantizing mlp.gate_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 28/32...
INFO - Quantizing mlp.down_proj in layer 28/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 28/32...
INFO - Start quantizing layer 29/32
INFO:auto_gptq.modeling._base:Start quantizing layer 29/32
INFO - Quantizing self_attn.k_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 29/32...
INFO - Quantizing self_attn.v_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 29/32...
INFO - Quantizing self_attn.q_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 29/32...
INFO - Quantizing self_attn.o_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 29/32...
INFO - Quantizing mlp.up_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 29/32...
INFO - Quantizing mlp.gate_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 29/32...
INFO - Quantizing mlp.down_proj in layer 29/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 29/32...
INFO - Start quantizing layer 30/32
INFO:auto_gptq.modeling._base:Start quantizing layer 30/32
INFO - Quantizing self_attn.k_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 30/32...
INFO - Quantizing self_attn.v_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 30/32...
INFO - Quantizing self_attn.q_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 30/32...
INFO - Quantizing self_attn.o_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 30/32...
INFO - Quantizing mlp.up_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 30/32...
INFO - Quantizing mlp.gate_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 30/32...
INFO - Quantizing mlp.down_proj in layer 30/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 30/32...
INFO - Start quantizing layer 31/32
INFO:auto_gptq.modeling._base:Start quantizing layer 31/32
INFO - Quantizing self_attn.k_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 31/32...
INFO - Quantizing self_attn.v_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 31/32...
INFO - Quantizing self_attn.q_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 31/32...
INFO - Quantizing self_attn.o_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 31/32...
INFO - Quantizing mlp.up_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 31/32...
INFO - Quantizing mlp.gate_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 31/32...
INFO - Quantizing mlp.down_proj in layer 31/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 31/32...
INFO - Start quantizing layer 32/32
INFO:auto_gptq.modeling._base:Start quantizing layer 32/32
INFO - Quantizing self_attn.k_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.k_proj in layer 32/32...
INFO - Quantizing self_attn.v_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.v_proj in layer 32/32...
INFO - Quantizing self_attn.q_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.q_proj in layer 32/32...
INFO - Quantizing self_attn.o_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing self_attn.o_proj in layer 32/32...
INFO - Quantizing mlp.up_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.up_proj in layer 32/32...
INFO - Quantizing mlp.gate_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.gate_proj in layer 32/32...
INFO - Quantizing mlp.down_proj in layer 32/32...
INFO:auto_gptq.modeling._base:Quantizing mlp.down_proj in layer 32/32...

Now that the model is quantized, we can save it so we can share it or load it later! Since quantizing with GPTQ takes a while and some resources, it's advised to always save them.

[8]

Model quantized and saved to GPTQ 4-bit precision!

You can also load it for inference using auto-gptq as follows:

[10]

WARNING - Exllamav2 kernel is not installed, reset disable_exllamav2 to True. This may because you installed auto_gptq using a pre-build wheel on Windows, in which exllama_kernels are not compiled. To use exllama_kernels to further speedup inference, you can re-install auto_gptq from source.
WARNING:auto_gptq.modeling._base:Exllamav2 kernel is not installed, reset disable_exllamav2 to True. This may because you installed auto_gptq using a pre-build wheel on Windows, in which exllama_kernels are not compiled. To use exllama_kernels to further speedup inference, you can re-install auto_gptq from source.
WARNING - CUDA kernels for auto_gptq are not installed, this will result in very slow inference speed. This may because:
1. You disabled CUDA extensions compilation by setting BUILD_CUDA_EXT=0 when install auto_gptq from source.
2. You are using pytorch without CUDA support.
3. CUDA and nvcc are not installed in your device.
WARNING:auto_gptq.modeling._base:CUDA kernels for auto_gptq are not installed, this will result in very slow inference speed. This may because:
1. You disabled CUDA extensions compilation by setting BUILD_CUDA_EXT=0 when install auto_gptq from source.
2. You are using pytorch without CUDA support.
3. CUDA and nvcc are not installed in your device.
WARNING - ignoring unknown parameter in quantize_config.json: quant_method.
WARNING:auto_gptq.modeling._base:ignoring unknown parameter in quantize_config.json: quant_method.
INFO - The layer lm_head is not quantized.
INFO:auto_gptq.modeling._base:The layer lm_head is not quantized.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

How are you today? I don't have feelings, but I'm here and ready to help you with your questions or problems! How can I assist you today? 😊