Target all modules

[tranquil hare]

model = FastLanguageModel.get_peft_model(
    model,
    r = 8, 
    target_modules = "all-linear",
    lora_alpha = 32,
    lora_dropout = 0, 
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    use_rslora = False,  # rank stabilized LoRA
    loftq_config = None, # LoftQ
)

I was trying to deviate from the notebook as I heard targetting all modules is benefital by replacing:

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],```
with 

`target_modules = "all-linear",`

However this gave the error: 

```Traceback (most recent call last):
  File "/home/volts/AI/ModelCreation/modelcreation/training/unslothtrainer.py", line 32, in <module>
    model = FastLanguageModel.get_peft_model(
  File "/home/volts/AI/ModelCreation/.venv/lib/python3.10/site-packages/unsloth/models/llama.py", line 1465, in get_peft_model
    assert(module in accepted_modules)
AssertionError```

I'm aware I'm wrong but I'm not sure how, I think I'm getting confused on syntax across multiple articles and would really appreciate some help. Thankyou.

[compact dirge]

["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",]
This config already targets all linear modules (apart from embed and head), don't change it unless you know what you do

[tranquil hare]

👍 Thankyou, and this will apply to most model?

https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms

The article I was reading mentioned head was useful, so why would this not be included in the list?

[compact dirge]

This will apply to most models. Including head may help and may not, but it will surely slowdown finetuning

[tranquil hare]

Ok thanks and same applies to embed I presume? It may not have an effect but will not be detrimental to model peformamce

I do wonder why all examples I see do not include those modules and if there is a reason or just a trend

those modules being as mentioned by Nyan: ["embed_tokens", "lm_head"]

[dense raft]

oh i havent added all yet

u can edit it and simply add "lm_head", "embed_tokens"

but i do not suggest it as well

itll make training slower and ur loss will be higher

[tranquil hare]

👍 ye thanks Sloth after a bit of searching in this discord I found them. An all method would be great.... ohh?

Ok is there a resource explaining this, not that I doubt you at all just curious why

[dense raft]

yes yes

i can add that in!

yes training will be slower

uses more VRAM

and from first hand experience, higher loss

and not worht it

i dont suggest it

[tranquil hare]

👍 Ok got it boss, ty for your help as always

[real aurora]

Mr volts hopefully we solved your issue? 🙏 Also a huge thank you to @compact dirge for helping out!