Hi. I am trying to run a serverless endpoint with the Omost model, which requires more VRAM. When I accidentally started it with a 20GB all works fine, except the expected CUDA OoM. Configuring to use 80GB VRAM in EU-RO-1 the endpoint is created but the workers end prematurely constantly. Is there any way to figure out what and why this is happening? The logs do not really seem to help me here.
#Higherend GPU Worker Stop Prematurely
formal kindle
lilac tundra
maybe post the log here?
visual kite
Hi. I am trying to run a serverless endpoint with the Omost model, which require...
can you send the handler code here
and how do you start the handler
maybe your dockerfile entrypoint or cmd too will help
formal kindle
That's the whole code (sorry, I have no publicly accessible Git repo). I start it via the Serverless + Endpoint UI using a template: Checking the two 80 GB GPU options, selecting the template and our Network Volume in the EU-RO-1 region. The template's Container Image is runpod/base:0.6.1-cuda12.1.0 and its Container Start Command is bash -c ". /runpod-volume/b426d666/prod/venv/bin/activate && python -u /runpod-volume/b426d666/prod/omost/src/handler.py" with 100 GB Container Disk.
visual kite
Ends prematurely?
like it doesn't return any output?
Whats the job status like
your handler file looks normal
Try making output a list
formal kindle
The worker itself doesn't start, e.g., its box turns red and the tooltip says "Worker stopped prematurely". But this does not occure when running on 20GB GPUs.
visual kite
Oh
Then it might be oom
Worker stopped prematurely means it stopped even before starting to handle the job, so I guess it failed to load your model
visual kite
Then it might be oom
Oh nvm, try to contact runpod to report this problem