#How to enable embed_tokens training in Gemma 3 with Unsloth?

3 messages · Page 1 of 1 (latest)

grim pumice
#

How can I enable embed_tokens training in Unsloth when fine-tuning a Gemma 3 model? I need this for a low-resource language.

grim pumice
#

Hi, would anyone be able to assist with this? I’d really appreciate any guidance!

rapid pecan
#

try
model = FastModel.get_peft_model(
model,

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj","embed_tokens", "lm_head","language", "text","self_attn", "attention", "attn","mlp", "feed_forward", "ffn", "dense",],
r = 8,           # Larger = higher accuracy, but might overfit
lora_alpha = 8,  # Recommended alpha == r at least
lora_dropout = 0,
bias = "none",
random_state = 3407,

)