VRAM fully allocated with no visible processes

[fading goblet]

RTX 4090 host has VRAM fully allocated with no visible processes (nvidia-smi empty). on A1111 I'm getting CUDA OOM notices - with processes listed that I'm not apparently running.

OutOfMemoryError: CUDA out of memory. Tried to allocate 720.00 MiB. GPU 0 has a total capacity of 23.64 GiB of which 261.19 MiB is free. Process 3231365 has 14.35 GiB memory in use. Process 33993 has 9.00 GiB memory in use. Of the allocated memory 8.29 GiB is allocated by PyTorch, and 254.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

What's going on?

[cerulean jay]

in my opinion it can be either that zombie CUDA context leaked VRAM or the processes exist, but you can’t see them

try more explicit nvidia-smi - with these parameters;

nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv nvidia-smi pmon -c 1

also try to see who has the gpu open:

fuser -v /dev/nvidia* lsof /dev/nvidia*

(you may need to use sudo based on your image - try without first)

if pytorch gave you PID(s) check them using ps -fp