Failed to initialize NVML: Unknown Error | Runpod | Page 1

fading rune Jul 19, 2024, 12:08 PM

#

(compress) root@1908bfec7b85:/workspace# nvidia-smi

Failed to initialize NVML: Unknown Error

Every hour or so on my runpod instance, I get the above nvidia error.

I'm not changing anything with the machine -- I have to restart it to fix it. Any ideas?

Thanks

full oarBOT Jul 19, 2024, 12:08 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

vast depot Jul 19, 2024, 12:10 PM

#

https://stackoverflow.com/questions/72932940/failed-to-initialize-nvml-unknown-error-in-docker-after-few-hours

Stack Overflow

Failed to initialize NVML: Unknown Error in Docker after Few hours

I am having interesting and weird issue.
When I start docker container with gpu it works fine and I see all the gpus in docker. However, few hours or few days later, I can't use gpus in docker.
Whe...

cedar zenith Jul 19, 2024, 1:34 PM

#

looks like something need to be changed for the host server, might want to create a support ticket with pod id and attach the link nerdylive just post, hope they can fix it.

#Failed to initialize NVML: Unknown Error