Zluda keeps freezing while compiling | AI HUB | Page 1

gray violet Jul 25, 2025, 1:31 PM

#

New forum for Noobies

proven glacier Jul 25, 2025, 1:31 PM

#

okay, lets try a different method

#

download https://github.com/IAHispano/Applio/archive/refs/tags/3.2.9.zip

#

unzip it into C:\

#

so you have C:\Applio-3.2.9

#

drop this inside

📎 run-install.bat

#

and run

#

this should install applio using python 3.11

#

download these into Applio folder

#

https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110x/torch-2.7.0a0+rocm_git3f903c3-cp311-cp311-win_amd64.whl

#

https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110x/torchaudio-2.7.0a0+52638ef-cp311-cp311-win_amd64.whl

#

one sec, checking myself

#

you need to nuke C:\users\user\miniconda3 folder before you run install

gray violet Jul 25, 2025, 1:37 PM

#

proven glacier you need to nuke C:\users\user\miniconda3 folder before you run install

how to?

proven glacier Jul 25, 2025, 1:37 PM

#

just delete it

#

or rename to .old

gray violet Jul 25, 2025, 1:37 PM

#

check i did!

#

busy!

#

proven glacier Jul 25, 2025, 1:39 PM

#

it will download miniconda installer, then install it

#

then the rest

gray violet Jul 25, 2025, 1:41 PM

#

It's installing torch

#

Oke it's finished installing

gray violet Jul 25, 2025, 1:46 PM

#

proven glacier https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110...

Then putt these in the folder?

proven glacier Jul 25, 2025, 1:50 PM

#

yes

gray violet Jul 25, 2025, 1:50 PM

#

i did

proven glacier Jul 25, 2025, 1:51 PM

#

then open cmd in Applio's folder

#

and run

#

env\python -m pip install torch-2.7.0a0+rocm_git3f903c3-cp311-cp311-win_amd64.whl

gray violet Jul 25, 2025, 1:51 PM

#

do i need to put the downloaded files in the env folder or in the root?

proven glacier Jul 25, 2025, 1:51 PM

#

env\python -m pip install torchaudio-2.7.0a0+52638ef-cp311-cp311-win_amd64.whl

#

into C:\Applio-3.2.9

gray violet Jul 25, 2025, 1:52 PM

#

proven glacier Jul 25, 2025, 1:52 PM

#

how about env\python

gray violet Jul 25, 2025, 1:53 PM

#

proven glacier Jul 25, 2025, 1:53 PM

#

delete env

#

re-run run-install.bat

gray violet Jul 25, 2025, 1:54 PM

#

finished

#

now again terminal?

proven glacier Jul 25, 2025, 1:55 PM

#

yes

#

verify python is correct one

gray violet Jul 25, 2025, 1:55 PM

#

again same error

#

proven glacier Jul 25, 2025, 1:55 PM

#

env\python

#

not python

#

exit() to exit from it

gray violet Jul 25, 2025, 1:55 PM

#

proven glacier Jul 25, 2025, 1:56 PM

#

did you save the updated run-install from above into this applio folder?

gray violet Jul 25, 2025, 1:56 PM

#

yes

#

this is in the bat, it says 3.10

proven glacier Jul 25, 2025, 1:57 PM

#

ah, should be 3.11

#

sorry, nuke env again, change to 3.11, re-run

gray violet Jul 25, 2025, 1:59 PM

#

proven glacier sorry, nuke env again, change to 3.11, re-run

Don't worry, happends to the best! Already happy that you help me 🙂

#

proven glacier Jul 25, 2025, 2:05 PM

#

okay, not the pip install

gray violet Jul 25, 2025, 2:05 PM

#

it works

#

even with pip

#

#

ow..

proven glacier Jul 25, 2025, 2:06 PM

#

torchaudio too

gray violet Jul 25, 2025, 2:07 PM

#

Nope error

proven glacier Jul 25, 2025, 2:07 PM

#

?

gray violet Jul 25, 2025, 2:08 PM

#

gray violet ow..

this is the error at the end of: env\python -m pip install torch-2.7.0a0+rocm_git3f903c3-cp311-cp311-win_amd64.whl

proven glacier Jul 25, 2025, 2:09 PM

#

I mean run second install for torchaudio

gray violet Jul 25, 2025, 2:09 PM

#

proven glacier Jul 25, 2025, 2:10 PM

#

okay, run-applio.bat now

gray violet Jul 25, 2025, 2:11 PM

#

proven glacier Jul 25, 2025, 2:12 PM

#

not a problem

#

once it opens, go to training tab and see advanced settings

gray violet Jul 25, 2025, 2:14 PM

#

#

proven glacier Jul 25, 2025, 2:15 PM

#

okay, give it a try

#

change the value to 0

gray violet Jul 25, 2025, 2:16 PM

#

is it oke to get that much warnings?

#

the rest is working xd

proven glacier Jul 25, 2025, 2:17 PM

#

that's just librosa warnings, it can be updated to silence them

#

env\python -m pip install librosa==0.11.0

#

question is whether the training works

gray violet Jul 25, 2025, 2:23 PM

#

we are gonna find out now

#

nope

#

#

proven glacier Jul 25, 2025, 2:31 PM

#

hm

proven glacier Jul 25, 2025, 2:55 PM

#

can you post this as a text?

gray violet Jul 25, 2025, 2:57 PM

#

i started again

#

now i get this

#

To create a public link, set share=True in launch().
Using HiFi-GAN vocoder
Using HiFi-GAN vocoder
Process Process-2:
Traceback (most recent call last):
File "C:\Applio-3.2.9\env\Lib\multiprocessing\process.py", line 314, in _bootstrap
self.run()
File "C:\Applio-3.2.9\env\Lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\Applio-3.2.9\rvc\train\train.py", line 452, in run
net_g = DDP(net_g, device_ids=[device_id])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\nn\parallel\distributed.py", line 837, in init
_sync_module_states(
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\distributed\utils.py", line 311, in _sync_module_states
_sync_params_and_buffers(process_group, module_states, broadcast_bucket_size, src)
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\distributed\utils.py", line 322, in _sync_params_and_buffers
dist._broadcast_coalesced(
RuntimeError: HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with TORCH_USE_HIP_DSA to enable device-side assertions.

proven glacier Jul 25, 2025, 2:57 PM

#

you need to use 0 device

gray violet Jul 25, 2025, 2:59 PM

#

📎 message.txt

#

''D:/jam/TheRock/ml-libs/MIOpen/src/ocl/convolutionocl.cpp:275'' i dont have a D:/jam...?

proven glacier Jul 25, 2025, 3:02 PM

#

that's normal

#

could you set env variable

#

MIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD=1

#

and restart applio from a new window?

gray violet Jul 25, 2025, 3:06 PM

#

#

where?

proven glacier Jul 25, 2025, 3:06 PM

#

user variables is fine

gray violet Jul 25, 2025, 3:07 PM

#

at path or a total new one?

proven glacier Jul 25, 2025, 3:07 PM

#

new entry

#

name: MIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD

#

value: 1

gray violet Jul 25, 2025, 3:07 PM

#

proven glacier Jul 25, 2025, 3:07 PM

#

yeah

gray violet Jul 25, 2025, 3:09 PM

#

📎 message.txt

proven glacier Jul 25, 2025, 3:11 PM

#

okay, few more

#

MIOPEN_ENABLE_LOGGING=1
MIOPEN_ENABLE_LOGGING_CMD=1

gray violet Jul 25, 2025, 3:12 PM

#

proven glacier Jul 25, 2025, 3:13 PM

#

it may require some extra work on dev side, seems that they've made some shortcuts in this torch build

#

go ahead and run training again

gray violet Jul 25, 2025, 3:14 PM

#

proven glacier Jul 25, 2025, 3:15 PM

#

i need the whole thing as text

gray violet Jul 25, 2025, 3:15 PM

#

its busy

#

it's compiling

proven glacier Jul 25, 2025, 3:20 PM

#

no it is not

gray violet Jul 25, 2025, 3:20 PM

#

only i see everthing haha

proven glacier Jul 25, 2025, 3:20 PM

#

it just logs every operation

gray violet Jul 25, 2025, 3:20 PM

#

the balk with 37% was the same what i see in the beginning

proven glacier Jul 25, 2025, 3:21 PM

#

unselect the text, you're freezing it

#

esc

gray violet Jul 25, 2025, 3:22 PM

#

i did

proven glacier Jul 25, 2025, 3:32 PM

#

well?

gray violet Jul 25, 2025, 3:32 PM

#

still busy

proven glacier Jul 25, 2025, 3:33 PM

#

that can't be right

gray violet Jul 25, 2025, 3:35 PM

#

this is what he did till now

proven glacier Jul 25, 2025, 3:35 PM

#

so now it suddenly works???

gray violet Jul 25, 2025, 3:35 PM

#

📎 message.txt

#

can't even select all of it tho

proven glacier Jul 25, 2025, 3:36 PM

#

46%|█████████████████████████████████████▌ | 57/123 [01:57<02:15, 2.05s/it]

#

it is working

#

not failing

gray violet Jul 25, 2025, 3:36 PM

#

yeah but somewhere it was already at 118... 98&

#

%

proven glacier Jul 25, 2025, 3:37 PM

#

can you change all those the env values to 0

#

and restart from new window?

gray violet Jul 25, 2025, 3:37 PM

#

should i stop this one?

proven glacier Jul 25, 2025, 3:37 PM

#

yes

gray violet Jul 25, 2025, 3:38 PM

#

#

which one

proven glacier Jul 25, 2025, 3:38 PM

#

all 3

gray violet Jul 25, 2025, 3:40 PM

#

📎 message.txt

proven glacier Jul 25, 2025, 3:40 PM

#

what the fuk

#

lets set enable logging cmd to 1

#

and re-try

gray violet Jul 25, 2025, 3:42 PM

#

📎 message.txt

proven glacier Jul 25, 2025, 3:42 PM

#

lets see if it progresses to 1/

gray violet Jul 25, 2025, 3:43 PM

#

what do i need to do :)?

proven glacier Jul 25, 2025, 3:43 PM

#

watch the log, if it gets past to next step it is really weird

gray violet Jul 25, 2025, 3:44 PM

#

It doesnt do anything now (see the end of the txt... its the same error

proven glacier Jul 25, 2025, 3:45 PM

#

so it only managed one pass

#

set to 1

#

and try again

gray violet Jul 25, 2025, 3:47 PM

#

same error

proven glacier Jul 25, 2025, 3:48 PM

#

are you runnin them in new window every time?

gray violet Jul 25, 2025, 3:48 PM

#

i close the cmd and the applio tap in chrome

proven glacier Jul 25, 2025, 3:48 PM

#

okay

gray violet Jul 25, 2025, 3:48 PM

#

i changed all of the variable to 1 and now it's logging again

#

without error after a few lines

proven glacier Jul 25, 2025, 3:48 PM

#

and keeps going past 1st?

gray violet Jul 25, 2025, 3:49 PM

#

📎 message.txt

#

this is till now

proven glacier Jul 25, 2025, 3:50 PM

#

are you using a different batch size?

#

7 instead of 6?

#

gray violet Jul 25, 2025, 3:51 PM

#

gray violet

I used 6 each time , only ones it slide to 7

#

but with the other errors it was 6 aswell

proven glacier Jul 25, 2025, 3:52 PM

#

set all variables to 0, use batch 6

#

you used a different batch here too

#

figure out which batch sizes work

gray violet Jul 25, 2025, 3:55 PM

#

All on 0, batch 6

📎 message.txt

#

5 gave same error

proven glacier Jul 25, 2025, 3:56 PM

#

okay, but if you set all 3 to 1 and use batch 6 again?

gray violet Jul 25, 2025, 3:56 PM

#

then it starts logging like crazy

#

without any error

proven glacier Jul 25, 2025, 3:57 PM

#

but does not break...

gray violet Jul 25, 2025, 3:57 PM

#

till then (when i needed to stop) no

gray violet Jul 25, 2025, 3:58 PM

#

gray violet 5 gave same error

4 gave same error

proven glacier Jul 25, 2025, 3:58 PM

#

with all 3 flags to 1?

#

go up, 8, 12, 16

gray violet Jul 25, 2025, 3:58 PM

#

nope still 0

#

10 on 0 gave same error

proven glacier Jul 25, 2025, 3:58 PM

#

no, set all flags to 1

gray violet Jul 25, 2025, 3:59 PM

#

oke!

proven glacier Jul 25, 2025, 3:59 PM

#

then try different batches

gray violet Jul 25, 2025, 4:00 PM

#

8 is logging aswell

#

📎 message.txt

proven glacier Jul 25, 2025, 4:01 PM

#

I assume if you try 7 it fails

gray violet Jul 25, 2025, 4:01 PM

#

Should we try?

proven glacier Jul 25, 2025, 4:01 PM

#

yes

gray violet Jul 25, 2025, 4:02 PM

#

its logging

#

📎 message.txt

proven glacier Jul 25, 2025, 4:05 PM

#

okay

#

okay, lets set variables to 0

#

then open rvc/train/train.py, find line torch.backends.cudnn.benchmark = True

#

and set it to False

gray violet Jul 25, 2025, 4:07 PM

#

#

batch 6 or 7?

proven glacier Jul 25, 2025, 4:08 PM

#

any

gray violet Jul 25, 2025, 4:10 PM

#

error

#

📎 message.txt

proven glacier Jul 25, 2025, 4:10 PM

#

try a few times

#

same batch

gray violet Jul 25, 2025, 4:11 PM

#

i did 6 now, so 6 again?

proven glacier Jul 25, 2025, 4:11 PM

#

i mean re-try a few times

#

I wanna see if it just randomply fails

gray violet Jul 25, 2025, 4:15 PM

#

second = error
Third = error
fourth = error
fifth = error
Sixth = error
seventh = error

#

eigth = error
Ninth = error

proven glacier Jul 25, 2025, 4:17 PM

#

okay

#

in train.py, after the line you changed, add

#

torch.backends.cudnn.enabled = False

#

with the same spacing

#

restart and try to run training again

gray violet Jul 25, 2025, 4:19 PM

#

#

mmh no error yet.........

#

proven glacier Jul 25, 2025, 4:21 PM

#

how big is the set?

gray violet Jul 25, 2025, 4:22 PM

#

data? 35 min

#

proven glacier Jul 25, 2025, 4:22 PM

#

minutes

#

ok

gray violet Jul 25, 2025, 4:22 PM

#

35

proven glacier Jul 25, 2025, 4:22 PM

#

that's really slow

#

it should be much faster on 7800xty

gray violet Jul 25, 2025, 4:23 PM

#

proven glacier Jul 25, 2025, 4:23 PM

#

that's better

gray violet Jul 25, 2025, 4:24 PM

#

proven glacier Jul 25, 2025, 4:25 PM

#

you need to clean up eval folder

#

from all old attempts

#

keep the last file

gray violet Jul 25, 2025, 4:27 PM

#

proven glacier Jul 25, 2025, 4:29 PM

#

scalars tab shows the charts you need

gray violet Jul 25, 2025, 4:29 PM

#

#

and then what is the thing to look for when you overtrain or?

proven glacier Jul 25, 2025, 4:29 PM

#

expand loss_avg_50

#

and close 'grad'

gray violet Jul 25, 2025, 4:30 PM

#

proven glacier Jul 25, 2025, 4:31 PM

#

you can click this to load more data, then blue square under each chart to resize

#

let me know if it fails again

#

i've created a ticket for toch devs, they may ask for some more data

gray violet Jul 25, 2025, 4:32 PM

#

proven glacier i've created a ticket for toch devs, they may ask for some more data

Sure happy to help!

#

for now it's busy

#

and when do i know to stop? of like overtraining?

proven glacier Jul 25, 2025, 4:33 PM

#

g_total gonna converge

gray violet Jul 25, 2025, 4:34 PM

#

And how can i see it's overtraining>

proven glacier Jul 25, 2025, 4:34 PM

#

#

you can start testing models when it flattens

#

watch for fm loss

#

if norm_g does this

#

the model is wack

#

that's too high

gray violet Jul 25, 2025, 6:26 PM

#

can i make an index file in the middle of training (forgot to make it before is started)

proven glacier Jul 25, 2025, 6:28 PM

#

yes

proven glacier Aug 11, 2025, 8:31 PM

#

@gray violet lmk when you're online, i have one test

gray violet Aug 11, 2025, 8:32 PM

#

I am in like half an hour!

#

At the bar now drinking beer

gray violet Aug 11, 2025, 9:58 PM

#

Sorry took me a little longer! I’m online now

proven glacier Aug 11, 2025, 10:18 PM

#

gray violet Sorry took me a little longer! I’m online now

remind me, did we install Zluda last time?

#

what does the training/advanced settings show?

gray violet Aug 11, 2025, 10:18 PM

#

The last time, we got it working we didnt use Zluda

proven glacier Aug 11, 2025, 10:19 PM

#

could you do me a favor, save this in Applio's folder, then open cmd and run env\python bench.py

📎 bench.py

#

i wanna see the speed

#

it just runs 1000 loops of each operation

gray violet Aug 11, 2025, 10:20 PM

#

C:\Applio-3.2.9>env\python bench.py
Using cuda
torch.float32
linear : 0.0335s
conv1d 192x192x1 : 0.0565s
conv1d 192x768x3 : 0.2310s
conv1d 768x768x1 : 0.2482s
up_0 : 0.4821s
up_1 : 0.5306s
up_2 : 0.3540s
up_3 : 0.2720s
dn_0 : 0.1580s
dn_1 : 0.1630s
dn_2 : 0.1595s
dn_3 : 0.0740s
res1a : 0.2180s
MIOpen(HIP): Warning [IsEnoughWorkspace] [GetSolutionsFallback WTI] Solver <GemmFwdRest>, workspace required: 1228800, provided ptr: 0000000000000000 size: 0
MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 1228800, provided ptr: 0000000000000000 size: 0
MIOpen Error: D:/jam/TheRock/ml-libs/MIOpen/src/ocl/convolutionocl.cpp:275: No suitable algorithm was found to execute the required convolution
Traceback (most recent call last):
File "C:\Applio-3.2.9\bench.py", line 75, in <module>
t = benchmark_op(layer.to(dtype), x.to(dtype))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Applio-3.2.9\bench.py", line 23, in benchmark_op
_ = op(x)
^^^^^
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\nn\modules\conv.py", line 375, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Applio-3.2.9\env\Lib\site-packages\torch\nn\modules\conv.py", line 370, in _conv_forward
return F.conv1d(
^^^^^^^^^
RuntimeError: miopenStatusUnknownError

proven glacier Aug 11, 2025, 10:20 PM

#

weird

#

did we use wheels from scott or from https://d25kgig7rdsyks.cloudfront.net/v2/gfx110X-dgpu/ ?

gray violet Aug 11, 2025, 10:22 PM

#

No never heard of wheels

proven glacier Aug 11, 2025, 10:23 PM

#

env\python -m pip install torch-2.7.0a0+rocm_git3f903c3-cp311-cp311-win_amd64.whl

#

whl = wheel

gray violet Aug 11, 2025, 10:25 PM

#

we did

proven glacier Aug 11, 2025, 10:25 PM

#

okay, wanna try a newer build?

gray violet Aug 11, 2025, 10:26 PM

#

proven glacier https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110...

we installed these

proven glacier Aug 11, 2025, 10:26 PM

#

yeah, there's newer build

gray violet Aug 11, 2025, 10:26 PM

#

we can 🙂

#

i'm happy to help!

#

Just tell me what to do!

proven glacier Aug 11, 2025, 10:27 PM

#

env\python -m pip install --upgrade --index-url https://d2awnip2yjpvqn.cloudfront.net/v2/gfx110X-dgpu torchaudio```

#

then try the bench again

gray violet Aug 11, 2025, 10:30 PM

#

#

is this oke?

proven glacier Aug 11, 2025, 10:30 PM

#

okay, torchaudio then next

gray violet Aug 11, 2025, 10:31 PM

#

#

that one was oke

proven glacier Aug 11, 2025, 10:31 PM

#

okay, try env\python bench.py now

gray violet Aug 11, 2025, 10:31 PM

#

proven glacier Aug 11, 2025, 10:33 PM

#

could edit env\lib\site-packages\torch\__init__.py

#

find 0x0001 and change it to 0x0000 then save the file and try the bench again

gray violet Aug 11, 2025, 10:34 PM

#

this one?

proven glacier Aug 11, 2025, 10:34 PM

#

yes

gray violet Aug 11, 2025, 10:35 PM

#

same error

proven glacier Aug 11, 2025, 10:35 PM

#

did it show a popup?

#

it should've shown a window with an actual error

gray violet Aug 11, 2025, 10:36 PM

#

#

no only this

proven glacier Aug 11, 2025, 10:37 PM

#

dir c:\windows\system32\libomp140.x86_64.dll

#

maybe this file is missing

gray violet Aug 11, 2025, 10:38 PM

#

yeah dont see that one

#

proven glacier Aug 11, 2025, 10:39 PM

#

i mean run that command

#

#

should show this

#

if not, here's the file

📎 libomp140.x86_64.dll

gray violet Aug 11, 2025, 10:40 PM

#

proven glacier Aug 11, 2025, 10:40 PM

#

okay, that file should go to windows\system32

#

once you copy it over run the bench again

gray violet Aug 11, 2025, 10:41 PM

#

📎 message.txt

proven glacier Aug 11, 2025, 10:42 PM

#

are those env variables still set?

gray violet Aug 11, 2025, 10:43 PM

#

yeah didnt change after it worked

proven glacier Aug 11, 2025, 10:44 PM

#

performance-wise the numbers seem fine... except this failure

#

okay, I'm gonna make a ticket for the devs and see what they say

#

this is really weird

gray violet Aug 11, 2025, 10:51 PM

#

Oke! Happy to help 🙂

proven glacier Aug 11, 2025, 10:51 PM

#

the same wheels and bench works on 6800 just fine

#

sorry, I should've asked you to make a separate environment

#

if you need to run applio again this may work.. or not

#

you can revert back to the previous wheels

#Zluda keeps freezing while compiling