Cuda Driver Version issues

[torpid wing]

The error below shows up on various machines. I am using the following base images:
nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.6.0-cudnn-devel-ubuntu22.04
nvidia/cuda:12.5.1-cudnn-devel-ubuntu22.04

Machine ID: lmrw66dp9b9b6r

For some reason when I change the cuda version, the error message changes based on the base version
cuda:12.8 shows cuda>=12.8
cuda:12.6 shows cuda>=12.6
cuda:12.5 shows cuda>=12.5

error starting container: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.8, please update your driver to a newer version, or use an earlier cuda container: unknown

[tribal lion]

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

[torpid wing]

I see this randomly from time to time. This machine also has the issue: 8akpfues8csotz

Oh, looks like the only cuda versions allowed on those machines are 12.4, that must be the issue. Any reason that Cuda Versions aren't kept somewhat up to date? 12.4 is a year old

[slow siren]

if you use nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04 then you should filter the pod that has cuda 12.8 in the host

the same thing like other version of base cuda image that you're using

select 12.8 here

[torpid wing]

Yeah, I found that, but is there a reason cuda versions aren't kept up to date?

[slow siren]

i think it is to support older images, compability, stability and they are rolling update so not all machines are updated at the same time

[torpid wing]

Most cuda versions are pretty backwards compatible (if cuda 12.8 is available, as far back as 11.8 or farther should work fine in container), but I guess it probably comes down to not having the resourcing to update them super often

[slow siren]

yeah true probably

if you want you can post a #1185337232517759028 about updating cuda in hosts and see what the staffs say

[torpid wing]

Thanks for your help!

[slow siren]

your welcome

[torpid wing]

It used to be okay when the 4090 had been out for a while, but since 5090 only works with 12.8, it makes it very confusing for users on which template they should be using, they can't just use any one and have it work