#🔒 CUDA error

45 messages · Page 1 of 1 (latest)

broken mapleBOT
#

@orchid phoenix

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
orchid phoenix
#

650-ish MB of data and 280 M parameters.
It shouldn't be a oom error either

#

i have tried clearing cache.
There is no version mismatch.
Yes, the gpu is available (43-ish gigs approx. therefore no oom).
No other running processes.

thorny pine
#
  1. what GPU ar you using? (specifically how much VRAM do you have?)
  2. how big is your dataset?
  3. how are you training the model? (paste code if possible)
orchid phoenix
#

i don't think its a code level issue, because it runs on my other pc but not on this particular one

#

or maybe i fucked somethin up, what do you think

thorny pine
#

Have you tried checking nvidia-smi / nvtop as it's running?

orchid phoenix
#

thats the thing it doesnt run

#

it just gives me the total batch no. with tqdm and just stops

#

this thing

image.png
orchid phoenix
thorny pine
#

🥴 I'm pretty sure you can pass a command line arg to auto refresh

orchid phoenix
thorny pine
#

But either way, we know the oom is real

orchid phoenix
#

alright man, thanks for the help. i really appreciate it

thorny pine
#

Maybe the code isn't deallocating memory somewhere?

orchid phoenix
#

you think 285m param and 600 mb ish data needs only 15-20ish gigs of vram ?

thorny pine
orchid phoenix
#

FP32 precision

thorny pine
#

That should be around 3.2GB then

#

Although still nowhere near 40gb

orchid phoenix
#

even if the dataset is rgb, it still shouldnt 💀

#

its just 64 images each batch

thorny pine
#

mhm

#

Can you send the code again?

orchid phoenix
#

its just the train fn tho not the entire architecture that cool ?

orchid phoenix
thorny pine
#

Though I doubt I'll understand any of it :p

#

Check how much vram each .to() is consuming with torch.cuda.memory_allocated(config.device)

#

that should give a rough idea of where all the memory is going

orchid phoenix
#

just wrote some additional stuff to make it a bit clearer

#

one sec

orchid phoenix
#

IT dies here

cycle_encrypted = gen_E(fake_original)
        cycle_original = gen_O(fake_encrypted)
        cycle_encrypted_loss = l1(encrypted, cycle_encrypted)
        cycle_original_loss = l1(original, cycle_original)
        allocated_memory_bytes = torch.cuda.memory_allocated(config.device)
        allocated_memory_gb = allocated_memory_bytes / (1024 ** 3)
        print(f"Memory allocated on {config.device}:{allocated_memory_gb:.2f} GB")
        print('cycle loss')
#

@thorny pine

orchid phoenix
#

.

broken mapleBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.