ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memor | Runpod | Page 1

dry lintel Apr 11, 2024, 6:12 PM

#

Hi I keep getting ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). when trying to train a model on RunPod with a large batch size. I can't reproduce the error locally.
I found this https://github.com/pytorch/pytorch#docker-image and this https://pytorch.org/docs/stable/multiprocessing.html#strategy-management but I'm not sure how to fix the problem.

GitHub

Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch

compact haven Apr 11, 2024, 6:19 PM

#

its not possible to edit shm on runpod

dry lintel Apr 11, 2024, 6:20 PM

#

Okay, do you know how to avoid the problem?

compact haven Apr 11, 2024, 6:21 PM

#

umm lower batch size mayby

dry lintel Apr 11, 2024, 6:22 PM

#

Also changing the number dataloader seems to have an effect

#

But now I'm only utilizing 20% of GPU memory

compact haven Apr 11, 2024, 6:23 PM

#

im not sure what you train and what you use

dry lintel Apr 11, 2024, 6:23 PM

#

Sorry, I'm training an image transformer model with pytorch. RunPod Pytorch 2.2.0 image

dry lintel Apr 11, 2024, 6:51 PM

#

@compact haven Is shm the same for all pod types?

compact haven Apr 11, 2024, 6:52 PM

#

yes it's static

dry lintel Apr 11, 2024, 7:29 PM

#

Can't train anything with a batch size larger than 16 😦
I just get Killed now

icy glacier Jan 7, 2025, 11:15 PM

#

bumping this again, setting shm size is essential for distributed training on GPU

compact haven Jan 8, 2025, 10:42 AM

#

shm size cant be changed on pod level

#ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memor