serverless runpod

[mortal girder]

I am new to serverless hosting and i have some pains to host my model on runpod. On my computer unsloth runs perfectly, but when it comes to host it on runpod i am totally lost. Does anyone have faced such tasks?

[ocean mesa]

Hey 🙂

I have some experience with runpod. How familiar are you with docker and linux?

Also are you looking to use the model for training or inference?

[mortal girder]

Hello @ocean mesa , I am quite familiar with Linux, but totally new on docker.
I am looking to use the model for inference.

[wise granite]

have a look at runpod's templates, there might be a useful one there for hosting a custom model

[ocean mesa]

@mortal girder if you start with one of the pytorch containers on runpod you should be able to ssh into the container.

Then you will need to install unsloth and run it using the cli.

How far have you been able to get so far?

[mortal girder]

@ocean mesa actually i just understood that i could connect to the pod container via ssh. So far, i finally succeeded making a docker image and get my model on a serverless, the model is taken from my private huggingface repo. Although i still have a bug cuz of my docker image (i made a mistake such that it doesn't find python==> i forgot to explicitly use python3.11). also it was relying on a runpod docker image of 9GB so i'll try to find a smaller docker image as base and install on top of it unsloth and runpod packages

if I succeed i'll try to share the github code

[ocean mesa]

Which docker image are you using as the base image?

[mortal girder]

For the moment I use this one: runpod/base:0.6.2-cuda12.6.2 but it is pretty large.

[ocean mesa]

If you are just trying to run unsloth I would probably recommend starting from one of their (runpods) pytorch containers, it will have pytorch and python pre-installed

You should be able to just run pip install unsloth then run the cli python unsloth-cli.py (with whatever parameters you like)

[mortal girder]

thanks!! i'll try!