hello how can i serve models with unsloth over opan ai like api ? and is there a gui for inferencing weith unsloth ?
#Serve over internet with / without gui ?
gritty cedar
sturdy cloud
Whoops sorry missed this!
GUI inferencing yes! Use GPT4All for eg
save the model to GGUF with Unsloth, then use GPT4All
You can also try Ooba via Colab: https://colab.research.google.com/github/oobabooga/text-generation-webui/blob/main/Colab-TextGen-GPU.ipynb
For an API - try vLLM
vivid cape
hello how can i serve models with unsloth over opan ai like api ? and is there a...
you can setup SGlang on runpod to provide an API that supports batched inference.
gritty cedar
you can setup SGlang on runpod to provide an API that supports batched inference...
Do you have an examplary code?
gritty cedar
You can also try Ooba via Colab: https://colab.research.google.com/github/oobabo...
Thank you
gritty cedar
For an API - try vLLM
Can you provide code for how to do that?
vivid cape
Do you have an examplary code?
the sglang github repo has example code
rugged trench
@gritty cedar hopefully we solved your issue?