silk heart
Hi, appreciate this fantastic community so far.
I published a model to https://huggingface.co/pacozaa/lora_model_tinyllama_alpaca_first now if I want to deploy it to say "production" for inference api call, what do I do next? I see some model has deploy button and other options as well. Maybe not relate to Unsloth directly but any documentation recommended read would be perfect!
regal geyser
You have only pushed adapters by the way.
It would be easier if you quantised it to GGUF or merge it to 16 Bit.
Loading adapters onto the base model requires a lot of ram and if you don't have good hardware on HuggingFace, it is not possible.
silk heart
@regal geyser we need GGUF or 16bit of pytorch model to deploy? Will try
regal geyser
<@805550380334579772> we need GGUF or 16bit of pytorch model to deploy? Will try
Yes.
regal geyser
<@805550380334579772> we need GGUF or 16bit of pytorch model to deploy? Will try
Unless you have good hardware and can load the model with LORA. But you need good code.
If you convert to GGUF, then I can deploy it, then you can clone my Space.
regal geyser
<@805550380334579772> we need GGUF or 16bit of pytorch model to deploy? Will try
Actually, don't do 16 Bit, you need good hardware.
For free Spaces, do GGUF.
silk heart
alright will try!
regal geyser
Cool.
Just send the link to the GGUF model and I will make you a inference space that you can duplicate!
When I have time.
silk heart
I just pay for " pay as you go" in Google Colab, mayb it will do
regal geyser
No, for deploying on Spaces.
regal geyser
Actually, don't do 16 Bit, you need good hardware.
Colab is fine for converting 16 bit.
regal geyser
I just pay for " pay as you go" in Google Colab, mayb it will do
I meant on deploying on Spaces lol.
silk heart
but I can push gguf from unsloth example colab right?
regal geyser
Yes.
regal geyser
but I can push gguf from unsloth example colab right?
Both 16 Bit and GGUF are possbile to push from Colab , it is just that free Spaces from HuggingFace doesn't have enough RAM for 16 Bit.
silk heart
okok lol
regal geyser
Hahahah... Q8_0 is way too big for Spaces.
silk heart
ok q4_k_m right now
silk heart
will have to go in a bit tho, moving house with two newborns, maybe seee you in like 12 hours
silk heart
By the way, is this a chatbot?
Nah man, just my first try of the whole pipeline
regal geyser
I am only 14, no hassle of children lol.
regal geyser
will have to go in a bit tho, moving house with two newborns, maybe seee you in ...
Same.
regal geyser
@silk heart See ya!
silk heart
Done done
rotund hazel
Hopefully resolved guys? 👏
regal geyser
Not yet.
regal geyser
Here you go!
Duplicate it.
By the way TinyLlama isn't that smart. So the responses are really bad.
@silk heart
silk heart
Thankss. Let me close this.
regal geyser
Cool.
silk heart
yeah it's really bad lol
regal geyser
Yep...
