#GPU not detected on RunPod serverless - HELP!!

8 messages · Page 1 of 1 (latest)

civic belfry
#

Hey everyone,

I'm running into an issue on RunPod serverless endpoint. Despite having CUDA 12.4.1 set up in my Docker container, my models are initializing on CPU instead of GPU.

My logs show: "Initializing pipeline on cpu"

I'm trying to run Whisper model which would really benefit from GPU acceleration. I've included all the CUDA requirements in my Dockerfile and have torch 2.4.1 in my requirements.txt.

Has anyone encountered a similar issue with RunPod serverless? Is there something specific I need to configure to make it recognize the GPU?

Any help would be greatly appreciated!

Thanks in advance

past hedgeBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

honest gorge
#

or the config for your appliction

#

maybe there's some configuration you need to do for that specific "pipeline"?

mild hawk
#

maybe check if torch.cuda.is_available() returns true

civic belfry
#

maybe check if torch.cuda.is_available() returns true

Yeah this returns false and that why it goes to cpu

honest gorge
#

Try to launch a new pod with cuda 12.4 and later selected