#not sure how to read the graphs

1 messages · Page 1 of 1 (latest)

lunar sinew
#

One message removed from a suspended account.

faint quest
#

you're using an advanced version of the application without knowing how to use even the basic

#

I assume you started with some warmup, that's why losses are flat

lunar sinew
lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

so 500 steps is 3 epochs as you said, the results shoudl not be flat lines

#

show the training settings

#

and what's the number of steps used for averaging

#

in applio it is 50 steps, so 550 steps surely has some variety

lunar sinew
faint quest
#

okay, nothing unsual here

lunar sinew
#

One message removed from a suspended account.

faint quest
#

are you training on CPU?

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

how big is the dataset?

lunar sinew
#

One message removed from a suspended account.

faint quest
#

does task manager/performance actually show this used?

image.png
lunar sinew
#

One message removed from a suspended account.

faint quest
#

35 min audio should be going ~30s/epoch, not 30 minutes

#

okay.. you have a very large batch size, so it spilled into the shared memory

#

or you may be playing some 3d game in parallel

lunar sinew
#

One message removed from a suspended account.

faint quest
#

when it spill over the shared memory you get a terrible performance

lunar sinew
faint quest
#

what's the batch size?

lunar sinew
#

One message removed from a suspended account.

faint quest
#

well, stop this and re-start with [x] checkpointing

lunar sinew
#

One message removed from a suspended account.

faint quest
#

well, something is eating the vram

#

either a second process is running or something else

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

when you stopped it, did the vram used drop to 0?

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

lunar sinew
faint quest
#

well, you have some other process using GPU

#

another python.exe in details tab?

lunar sinew
faint quest
#

kill all of them, make sure gpu use is 0, restart the application

lunar sinew
faint quest
#

hardware acceleration enabled in discord or browser?

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

well, as I said checkpointing is always an option, at least you dont be training at 30m/epoch

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

if everything is good, charts go down like this

image.png
#

fm is finicky and may go up, as long as it is not too much it may be fine

#

+1 / 10k steps is fine

#

but it should be going down

#

norm_g going higher than 1k is concerning, over 3k is bad, over 10k (1e4) prerry much indicates the model is a toast

#

once total/g stabilizes and conveges down to a low value you can start testing models saved around those steps

#

like here 15.5k steps

image.png
lunar sinew
#

One message removed from a suspended account.

faint quest
#

but it may still go further

image.png
#

if the model sounds good, you can stop the training

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

test a few others, see which one works best without distortions

lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

lunar sinew
spring tinsel
#

or when you're trying experimental stuff (maybe you tried training a spin model and you didnt used spin's mute files?)

#

i'd recommend you try the regular applio, codename's fork is for more advanced users

lunar sinew
lunar sinew
spring tinsel
#

stick with applio and the original pretrain for the moment

#

after you learn stuff you can try codename's fork

lunar sinew
lunar sinew
#

One message removed from a suspended account.

spring tinsel
lunar sinew
lunar sinew
#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

#

One message removed from a suspended account.

faint quest
#

once it stops updating you may want to start looking at the charts

lunar sinew
#

One message removed from a suspended account.

faint quest
#

avg_50 is better, just dont smooth too much, 0.5 is good enough

lunar sinew
#

One message removed from a suspended account.

faint quest
#

norm_g is not something you need to use for determining the best model

lunar sinew
faint quest
#

it is just to see whether things went wrong

lunar sinew
#

One message removed from a suspended account.