Anyone know how to get this model to load or what this error means. I think the model should fit on my 3090 since its only 20.1 GB?
#Mistral MOE loading?
patent wind
near hatch
update to latest ooba and use llama.cpp (gguf) or autogptq instead
patent wind
Yeah Gguf is so slow though. I was hoping to get qptq working for some of the 3bit models since they are less than 24 GB. Im guessing they take more than that to hold them in memory?
frigid chasm
update to latest ooba and use llama.cpp (gguf) or autogptq instead
I understand correctly that now you don’t have to install anything manually for mixtral to work in text ui? Do I just need to update the interface via update_windows.bat? It would be enough?
compact wave
