Efficient-ML/GPTQ-for-Qwen3

Guidelines for Loading Qwen3 (GPTQ) Quantized Models

Installation Setup

Download the GPTQ-for-Qwen_hf folder.

File Replacement

If you need to use the tests we provide, please download the files in the eval_my directory on GitHub and pay attention to the "Attention" section in the README:

• Add eval_my directory: Place the eval_my directory under the GPTQ-for-Qwen directory.

Load the model

Group-wise Quantization

1. Perform GPTQ search

CUDA_VISIBLE_DEVICES=0 python path_of_qwen.py your_model_path \
--wbits model_wbit  --groupsize 128  \
--load path_of_.pth

2. Evaluate the quantized model

CUDA_VISIBLE_DEVICES=0 python path_of_qwen.py your_model_path \
--wbits model_wbit  --groupsize 128  \
--load path_of_.pth --eval

Per-channel Quantization

1. Perform GPTQ search

CUDA_VISIBLE_DEVICES=0 python path_of_qwen.py your_model_path \
--wbits model_wbit  --groupsize -1  \
--load path_of_.pth

2. Evaluate the quantized model

CUDA_VISIBLE_DEVICES=0 python path_of_qwen.py your_model_path \
--wbits model_wbit  --groupsize -1  \
--load path_of_.pth --eval

Notes

• You need to input the corresponding wbit and groupsize parameters for the model; otherwise, loading errors may occur.
• Set the groupsize parameter to -1 for per-channel quantization.
• Make sure you have sufficient GPU memory to run a 32B-sized model
• Check GitHub for more information.