Unable to upgrade linux kernel version from 5.4.0 to 5.15.0 - RunPod A40 GPU

[late vessel]

I'm trying to upgrade my linux kernel from version 5.4.0 to 5.15.0. This is required for me to train deep learning models.

Here's what I tried

1. I tried to manually upgrade it with apt command. however I'm still getting the same kernel version
2. I tried stopping and restarting the pod but the issue persists
3. Tried with a new pod by deploying the same A40 configuration and still the issue persists.

Below is the log from the terminal

root@9301aaa036f0:/# uname -r
5.4.0-107-generic
root@9301aaa036f0:/# apt install linux-generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
...

....
Creating config file /etc/default/grub with new version
Setting up linux-image-5.15.0-113-generic (5.15.0-113.123) ...
I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-113-generic
I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-113-generic
I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-113-generic
I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-113-generic
Setting up linux-modules-5.15.0-113-generic (5.15.0-113.123) ...
Setting up linux-modules-extra-5.15.0-113-generic (5.15.0-113.123) ...
Setting up linux-image-generic (5.15.0.113.113) ...
Setting up linux-generic (5.15.0.113.113) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for initramfs-tools (0.140ubuntu13.4) ...
Processing triggers for linux-image-5.15.0-113-generic (5.15.0-113.123) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.15.0-113-generic
root@9301aaa036f0:/# uname -r
5.4.0-107-generic

Unable to upgrade linux kernel version from 5.4.0 to 5.15.0 - RunPod A40 GPU

[boreal belfry]

Not sure if this is really possible

Is it @naive garnet

[naive garnet]

You cant as containers use host kernel

[late vessel]

Can we please upgrade the host kernel to any stable version greater than or equal to 5.5.0

?

If not, is it possible to temporarily upgrade the host kernel until we keep the pod active ?

[distant wolf]

Not possible as kernel update requires full host machine reboot

[late vessel]

oh ok I got more clarity. And, is there a solution to the problem?

[distant wolf]

Not sure if it's secure cloud or community cloud

[late vessel]

it's a secure cloud

[cobalt garden]

Just curious what deep learning model you’re training that requires newer versions of kernel?😀

[late vessel]

i'm training a deep learning model that sees (via camera), identifies the objects/persons in the world, see what people are doing etc.. using PyTorch

with reasoning and explanation

[boreal belfry]

wow interesting

For now, i'd suggest to use other libraries that doesn't require this linux kernel update

[late vessel]

ok and is there a timeline where the kernel can be updated? I just need an approximate timeline so that I can wait for the kernel upgrade

[cobalt garden]

That's really cool~Not sure how fast RunPod can upgrade kernel, but since they have so many data centers, it seems pretty hard.😆