so, i keep getting this message whenever i try to load a model. i've done some light research, but i've found nothing that i can make sense of, please help me
#im trying to load some models to do some light testing but i keep getting a warning message:
oblique remnant
snow whale
This warning message doesn't really mean anything useful...
But I find it a bit strange that you load something in 5 shards. You're probably loading some uncompressed model. And it seems the app dies.
Try using models with -GGUF suffix.
What's your hardware setup?
oblique remnant
unsure, been a while since i got this thing
snow whale
Hm, 11GB GTX card, you definitely need to use -GGUF models.
Something within 12B range, I think. Perhaps more if you ok with sacrificing a lot of speed.
oblique remnant
ye, thing is, i used to mess around with llms and got it all to work, just can't remember how for the life of me
oblique remnant
Hm, 11GB GTX card, you definitely need to use -GGUF models.
Something within 12B...
ight
ill download one and get back to you
oblique remnant
alright
so
still not working
different problem though
ill post the cmd message i got and then head to bed for now, it's late
snow whale
- Use llama.cpp loader
- Ensure you limit context to some reasonable value, for example 8192.
Not sure how it's field called now, maybe still n-ctx. The field where value is set to ~1 million.
oblique remnant
update
did that
no binaries
how do i fix this
also
sorry if this is annoying to deal with for you man
snow whale
cmd_windows.bat -> pip install -r requirements/full/requirements.txt
snow whale
hm, do you have several copies of the app installed?
if you do, ensure you do this command for the same copy you're running
and restart the app
snow whale
Try then cmd_windows.bat -> pip install https://github.com/oobabooga/llama-cpp-binaries/releases/download/v0.14.0/llama_cpp_binaries-0.14.0+cu124-py3-none-win_amd64.whl