#Training speed too fast

1 messages · Page 1 of 1 (latest)

unique pawn
#

I have not trained a model in about 3 months. Before that, my training speed would be around 30 seconds.
Now suddenly, my training speed is around 5 seconds, with the same dataset.

I have checked my dataset in Applio (/assets/datasets) and it contains all files (329) with a total of 460mb.
All files are 32fp wav with 40k, used audacity audio labelling for cutting.

The dataset is around 50 minutes long.

graph for g/total, g/mel and g/kl are attached.

No idea why this is happening. I have reinstalled Applio, also tried another computer.

Hardware is a 4090 and an i9 14th gen

image.png image.png image.png
#

At Batch Size 12, I have a step size of 18, in case that matters.

#

What I noticed is that my grad_norm_d and g go crazy, they go up and down alot.

untold patio
#

"my dataset in Applio (/assets/datasets) and it contains all files (329)"

#

but how much is in the model's folder sliced/audios?

#

how many files are included in the filelist.txt?

unique pawn
#

sliced_autios contains 511 cause it contains .pt files and sliced_audios_16k contains 329

untold patio
#

looks like the files are too big to be used in the batch

unique pawn
#

filelist has 331 lines.

obtuse skyBOT
#

Ayo? @unique pawn level 4 !!! lfg

unique pawn
#

does that mean I should change batch size or what would be the best way to solve it

#

grad_g looks totally crazy, not sure what it means though:

image.png
untold patio
#

depending on your labeling settings, you may just have like 30+ sec long file

unique pawn
#

yeah I have many small files, like 5-20 seconds, and a few 45+

#

so I manually cut them down?

untold patio
#

yeah, 3-7 seconds max

unique pawn
#

alright, I will get right on it and report back, thank you very much 🙂

untold patio
#

also make sure you dont have very small ones that are mostly silence

#

<100kb

unique pawn
#

i always thought cutting onlarger files was done automatically by rvc, cause u can also input a single large file

untold patio
#

if you do cut it there

obtuse skyBOT
#

Ayo? @untold patio level 32 !!! lfg

untold patio
#

in Applio you have a choice - cut, normalize/filter, noise filter

unique pawn
#

ok that would be easier because i cannot set a maximum label length in audacity.

untold patio
#

I use this with some success

#

that is if you really want out of the box audio cutting

unique pawn
#

you mean because out of the box leads to cutting mid-sentence?

untold patio
#

the settings above do not cut mid-words

#

built-in slicer can

dusk cedar