How do i identify overtraining?

[tepid dust]

Hey gamers, I've been making models for a little bit, but i've had trouble deciding when overtraining has taken place. I understand you would choose the lowest point in your loss graphs prior to them starting to shift upwards, but in a situation like this, where would the overtraining have taken place? 11k, 17k, 23k, or 31k?

[languid finch]

Overtraining is honestly hard to come by for the most part. You should test and see these low points, but that doesn't really mean the later points are necessarily overtrained. In fact, sometimes, I would say the latest point is the best.

Anyways, the main way to actually see overtraining (or, better yet, model quality being screwed up) is by checking g/mel and g/kl. Mel being how well spectrogram recreation is going in the model vs the original data, and KL being the model vs the original data.

Normally, both of them will be going down, and that's generally when model improvements are still happening. If it's stagnated or going up, it's not improving or getting worse at it's job, respectively.

The lowest G/total point is normally just the best average model from the generator, and not necessarily the best sounding model. This is why testing is basically essential. Try picking up on the models' differences by ear and see what you prefer, and which one has the least amount of noise if it even has noise.

Hope this helps you to identify overtraining and all that stuff.

[tepid dust]

Huh, interesting, okay. It looks like the g/mel and g/kl are still trending downwards. I've trained this model to 1000 epocs at this point, i think i'd like to train further than 1000, even if improvements are unnoticable at best. Is there a way to train >1000 in rvc gui?

[stiff urchin]

huh? still using that old one without rmvpe?

-gui

[mighty dust]

https://cdn.discordapp.com/attachments/1122285248844144733/1203460490475343953/caption.gif?ex=65d12cec&is=65beb7ec&hm=bd2fb8d010006dd7c6e3c1c67d3ae846fd1478e1a3124c544c31b43086fe54aa&

[languid finch]

Yeah, you'd have to edit the webui file from RVC, iirc

[languid finch]

no, not THAT one

pretty sure they're just using Mainline

[tepid dust]

no lmao im currently using rmvpe

[stiff urchin]

hina's kaggle right? you'd better test some weight files before deciding to continue training

[chrome birch]

I have a fix for the file explorer

[tepid dust]

im listening to the latest .pth that was saved and it sounds pretty good

[loud patrol]

Ayo? @tepid dust level 1 !!!

[chrome birch]

a modified Imjoy by Vidal

[tepid dust]

just change this to 10000 or whatever i want the max to be?

[languid finch]

Yeah, that should be it I believe

[tepid dust]

cool, thanks! I've never actually resumed training after i reached the training completion. How can i resume training from that 1000 epoch? Or should i just start a new model from scratch

[languid finch]

Just insert the same settings, name and stuff, you can change total epochs and save frequency, just keep everything else the same and don't preprocess or feature extract again.