tender carbon
My code which works well for unsloth/gemma-3-1b-it fails with the following error when I switch to unsloth/gemma-3-4b-it.
----> 2 doc_tokens = tokenizer.tokenize(doc)
AttributeError: 'Gemma3Processor' object has no attribute 'tokenize'
I load the model and tokenizer as prescribed in the Gemma 3 notebook for 4bit finetuning.
Can now confirm this happens with the 12B and 27B parameter models, in addition to the 4B model.
humble wadi
@tender carbon any chance you can give a code snippet to help me reproduce? Maybe it’s something unique to the dataset/formatting and the order in which things are done.
The unsloth Gemma3 4b notebook works for me. If I can reproduce I can figure out what’s happening
Ok wait I misinterpreted earlier. You’re trying to manually tokenize which is what’s different from the notebook example? I think I can track this down
The 4b + variants are returning the processor instead of the tokenizer. The quick workaround would be to do tokenizer.tokenizer.tokenize