Gemma3 gguf conversion doesn't work | Unsloth AI | Page 1

viscid sundial May 6, 2025, 8:30 AM

#

Hi! I'm trying to fine tune gemma3 4b and then quantize it to Q8_0. But I have the errors you see in the screens. About the packeges versions I'm using the official unsloth gemma3 notebook as starting point. It doesn't work even updating unsloth and unsloth_zoo to the last versions. Can someone help?

craggy thunder May 6, 2025, 9:01 AM

#

we're updating unsloth to fully support the new architecture of the llama.cpp conversion class.

#

until then you can try to do the conversion on the console using the script that comes with llama.cpp, convert_hf_to_gguf.py

viscid sundial May 6, 2025, 9:33 AM

#

Will try, thanks!

viscid sundial May 6, 2025, 10:39 AM

#

craggy thunder we're updating unsloth to fully support the new architecture of the llama.cpp co...

Still have problems, don't know if I'm using it rightly

weak relic May 7, 2025, 5:30 AM

#

Interesting so I had it work today with with: if True: # Change to True to save to GGUF
model.save_pretrained_gguf(
"fhi-2025-may-0.2-float16",
quantization_type = "Q8_0", # For now only Q8_0, BF16, F16 supported
)

#

After doing if True: # Change to True to save finetune!
model.save_pretrained_merged("fhi-2025-may-0.2-float16", tokenizer)

#

/content# pip freeze |grep unsloth
unsloth==2025.4.7
unsloth_zoo==2025.4.4

craggy thunder May 7, 2025, 6:18 AM

#

wait what

#

actually you'd need to merge first before you can convert to gguf

weak relic May 10, 2025, 4:56 AM

#

Yeah sorry my “after” should have been first

#Gemma3 gguf conversion doesn't work