Attempt unsloth library on Mamba-2.8b

[halcyon sandal]

I've read the introduction blog post, and the blog post about apply unsloth on gemma. Where do I start 🧐 ?

[tropic stratus]

Hi have you tried using our free Google Colab notebooks? https://github.com/unslothai/unsloth#-finetune-for-free

Unfortunately if you want to use Mamba, it will not work with unsloth, you'll need to make a lot of changes

[halcyon sandal]

is it precisely due to architecture difference so I have to redo the manual autograd?

[tropic stratus]

Yes unfortunately. Mamba is quite different from transformers

[mint viper]

Had a look yesterday - its exttremely different, like a whole new beast than transformers 😦

[halcyon sandal]

Im happy to do it as personal project but I kinda need to know that code-wise, how did you approach manual autograd with transformer in the first place.

p.s. im bad at jacobian derivative

[mint viper]

oh ye sadly derivatives

but as a start, just try coding up Mamba from start to end

and compare losses 🙂

[halcyon sandal]

my interpretation of your suggestion so far is go to fast_lora.py and write manual autograd to Mamba architecture

[mint viper]

Ohh no so the issue is Mamba isnt doing attention nor MLP anymore

fast lora will not work

[halcyon sandal]

https://huggingface.co/state-spaces/mamba-2.8b-hf this mamba can be Lora, so at least your def get_lora_parameters can work

on the other hand, their philosophy and yours seem awfully similar already

[mint viper]

Oh maybe https://github.com/huggingface/transformers/blob/main/src/transformers/models/mamba/modeling_mamba.py will be useful

[halcyon sandal]

bump

[mint viper]

hmmm mamba will sadly have to wait - it looks overly complex to optimize

ill have to get myself more well versed with it