#Cant load models

1 messages · Page 1 of 1 (latest)

plain nova
#

Need help, cant load models

01:35:40-792758 INFO Loading "TheBloke_dolphin-2.6-mistral-7B-AWQ"
01:35:40-810710 INFO TRANSFORMERS_PARAMS=
{ 'low_cpu_mem_usage': True,
'torch_dtype': torch.float16,
'device_map': 'auto',
'max_memory': {0: '2900MiB', 'cpu': '14800MiB'}}

01:35:40-816694 ERROR Failed to load the model.
Traceback (most recent call last):
File "G:\text-generation-webui-main\text-generation-webui-main\modules\ui_model_menu.py", line 232, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 93, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 263, in huggingface_loader
model = LoaderClass.from_pretrained(path_to_model, **params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3657, in from_pretrained
hf_quantizer.validate_environment(
File "G:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\transformers\quantizers\quantizer_awq.py", line 50, in validate_environment
raise ImportError("Loading an AWQ quantized model requires auto-awq library (pip install autoawq)")
ImportError: Loading an AWQ quantized model requires auto-awq library (pip install autoawq)

I have installed autoawg

image.png
haughty solar
#

Try GGUF version instead, AWQ is a never ending source of headache.
And AWQ support was dropped one version ago because it uses outdated libraries and incompatible with new version of this app.

Also, it's a fairly old model, newer models are a lot better.

plain nova
#

im a total noob, just follow a walktrue. if you know of a better model i would say no 😄

haughty solar
#

What's your graphics card and how much memory it has?

plain nova
#

ASUS TUF Gaming GeForce RTX 4090 OC Edition 24GB GDDR6X

#

intel 9900k with 64 gb ram

#

i know im cpu bottlenecked

haughty solar
#

Nah, you are not.

#

Put this into downloader field
LoneStriker/Qwen2.5-32B-Instruct-4.65bpw-h6-exl2
Once it downloads, set max_sex_len to 16384 before loading.
It might be one of the best model you could try with your GPU.

plain nova
#

Thanks you do much for your help

haughty solar
#

Your GPU is quite amazing for language models.
You should look for models within 34B range at least at 4bit, these could run purely on your GPU at a pretty decent speed.

#

But the one I mentioned before is probably the best choice for general purpose stuff.

plain nova
#

swwet, first I need to learn the problem and just play arround with it 🙂 thanks you so much again for the help 😄