Overtraining?

[radiant sentinel]

is my model overtraining?

[obsidian rover]

g/total is just average iirc. check specially g/mel and g/kl (i may summon lyery about the g/kl thingy xd)

[obsidian rover]

Yea not overtraining ig.

[glass oracle]

How to know if its overtrained

[radiant sentinel]

How can you know

[glass oracle]

Idk

[obsidian rover]

tensorboard

[radiant sentinel]

I know but how can you tell by the graph

[obsidian rover]

if g/mel or g/kl rise up, even if both rise up, stop training and export the weight that's similar to the lowest point before OT

or keep training to see if the issue continues

[radiant sentinel]

So g/total isn't the one that I should look at

[obsidian rover]

Probably yeah, it's just average of model

[glass oracle]

That doednt even explain anything

[radiant sentinel]

The g/mel g/kl are going down but g/total is going up

[whole anvil]

g/total is an avg that is true but mel and kl are always gonna improve, there's no point of watching those

[whole anvil]

is going up forever?

like not wanting to go down again

[radiant sentinel]

Yeah i think so

[whole anvil]

can u post the graph here?

the one u believe is overtraining

[radiant sentinel]

Sure 1 sec

[whole anvil]

yup seems to start overtraining

u can wait a bit more to see if it goes down again or stop training now

[dense oyster]

pick weight files around 46k

[whole anvil]

tho i advise to stop training now since even if it goes down, the improvement will not be worth the wait (barely/not noticeable at all)

[radiant sentinel]

Thats what i suspected too but people told me to check g/mel and g/kl

[radiant sentinel]

Its still training so im gonna wait a bit

[whole anvil]

those are always gonna go down, they only go up if something went is very wrong (super extremely rare cases)

[radiant sentinel]

I see

[whole anvil]

mel is the clarity of your model, kl is how well the model learns the voice

g/total is the avg of fm, mel and kl
it shows u the real improvement of the model

[radiant sentinel]

What's fm?

[whole anvil]

feature matching
mostly the timbre and how natural your model will sound

[radiant sentinel]

Thank you for explaining it so well! I really appreciate it!

Also, do I pick the model before the lowest point or do I pick the one that is the lowest point

[whole anvil]

ideally pick the lowest point, if you can't pick the lowest point itself use the most close to it before overtraining

[dense oyster]

mostly you dont need to train for more than 500 or even 1000 epochs once you starts seeing the overtraining point

[radiant sentinel]

The model is at 803 epochs now

Wish that there was a way to detect overtraining faster

[whole anvil]

automatic overtraining detectors are unreliable yea

[radiant sentinel]

I saw a lot of graphs of people where the lines weren't going crazy up and down like mine

[whole anvil]

but after you train more models is going to be easier to spot overtraining

[whole anvil]

this is caused by batch size

[radiant sentinel]

Yeah this like my 5th model

[radiant sentinel]

Really? How does it affect the model?

[whole anvil]

too low causes crazy graphs like u said
too high causes very smooth graphs

[dense oyster]

again I said you could pick some weight files based on the optimal/overtraining point, also you can stop training anytime

[whole anvil]

for rvc training use batch 8, is a balance between the two

very safe and works

[radiant sentinel]

I don't think my pc can handle that

Lol

[whole anvil]

ow i see, anything below 8 gives you those crazy graphs

[radiant sentinel]

I use batch size 2

Didn't really know if using a lower batch size had a bad impact on the voice of the model

[whole anvil]

this is very unstable and not recommended for training, might degrade the model perfomance as well

[radiant sentinel]

Oh

Hmm

[whole anvil]

it doesn't really impact the quality, is more of a stability setting
when the graphs are unstable, things can go wrong

like your model overfitting too fast

or robotic feeling in the voice

[dense oyster]

if your gpu vram is too low or weak, you can opt for some colab/kaggle notebook that uses T4 with 15 gb vram that surely enables to use batch size 8-16

[whole anvil]

in simple words, your model can't really improve too much

[radiant sentinel]

I tried colab but i didnt understand it

It was so hard

[whole anvil]

have you tried hina's mainline colab?

[radiant sentinel]

I haven't

[whole anvil]

https://colab.research.google.com/github/hinabl/RVC-Online/blob/main/Mainline_Colab_Full.ipynb

don't use custom pretrains as they have multiple issues the original doesn't have

and disable extra

no need to run the theme loader cell

you first run these two

after that you put your ngrok token here

when you do that, you can now run the start running cell and it should work fine

this colab uses google drive as their file storage
you can upload your dataset's folder into google drive

your dataset location would be something like this /content/drive/MyDrive/mydataset

[radiant sentinel]

I'll try this when I'm home

Does that mean it won't use my pcs gpu

[whole anvil]

exactly, is going to use google's gpu
which can do batch 8

[obsidian rover]

what are you saying is wrong. u can't predict a graph, silly.

it was always a requirement to check g/mel and g/kl

iirc the guides even said that you needed to check these graphs

[whole anvil]

if you're training for mel yeah
if you're training for g/total nop
coz lowest g/total already has best mel

[obsidian rover]

g/total is just average tho

we check g/mel and g/kl, you can't predict graphs without seeing them 🙂

[radiant sentinel]

Does that mean that i can also set it up on my phone

[whole anvil]

yea it can run in your phone

i have no idea how to behaves there tho, on pc if you close the tab, it ends* your current session

[radiant sentinel]

Also since it doesnt use my gpu it means that i can do other stuff too since it doesnt hinder the pcs performance

[whole anvil]

yep

[radiant sentinel]

Alr

Thank you all sm for your help!

What is a ngrok token

[safe cove]

Ayo? @radiant sentinel level 11 !!!

[whole anvil]

is what is needed to generate the rvc links
the GUI and the tensorboard links

https://ngrok.com/

after u made ur account you go here https://dashboard.ngrok.com/get-started/your-authtoken

copy this

[whole anvil]

u paste* the authtoken in this

[radiant sentinel]

Do I need to run the cel everytime I want to train a model

And does continuing training a model work the same as local rvc

[whole anvil]

if u stop your session, yes, you need to repeat the process again
google saves your ngrok token so you don't need to paste it again, just need to run the 3 cells again and you'll be fine

[whole anvil]

is going to continue training until the session stops (google has free daily limits)

around 2~4 hours

the colab is still going to save your model progress, if your training stops because you hit your daily limit, you can continue training later

kaggle has 12 hours worth of daily limit and a 30 hour free weekly
is a bit complicated to use but we have a very good tutorial explaining how to use it

so in case you're not happy with google's daily limit
i'll leave the kaggle tutorial here

-kaggle

[compact stag]

📘 Kaggle Notebooks

• Applio Notebook, by Vidal Kaggle
• Applio Notebook, by Shirou Kaggle
• Music Source Separation, by Shirou Kaggle
• UVR5 NO UI, by Eddy Kaggle
• RVC Mainline, by Hina Kaggle
• Original W-Okada's Voice Changer, Kaggle
• Modified W-Okada's Voice Changer, Kaggle
• 📖 How to use RVC Mainline Kaggle by Cauthess

Note: Kaggle limits GPU usage to 30 hours per week.

[whole anvil]

https://www.kaggle.com/code/hinabl/mainline
https://rentry.co/RVC-Mainline-Kaggle

the tutorial also teaches you how to do the ngrok part as well

[radiant sentinel]

Thank you sm!

[radiant sentinel]

Do I uncheck custop pretrains when running every cell

[whole anvil]

Just don’t run it

U can ignore it

[radiant sentinel]

It gives me an error when i start running the tab with the ngrok_token

Nevermind i got it

When i open the url it says page not available

[whole anvil]

can you post a screenshot of the error here?

[radiant sentinel]

Wait it opened

It looks exactly like the local rvc

[whole anvil]

nice

[radiant sentinel]

So i cant use the file that on in my file explorer on my pc

Does it have to be on google drive?

[whole anvil]

yes, only on google drive

[radiant sentinel]

I'm confused. I used a file that's on my pc and it's training

Oh wait

Its says error

Lol

[whole anvil]

[radiant sentinel]

But how do i get the google drive path file

[whole anvil]

first drag and drop your dataset's folder into google drive
or you can create a new folder there and put your wavs inside, both work

your path is /content/drive/MyDrive/mydataset

with mydataset being the name of your folder of course

so if you named your folder "dataset" the path would be /content/drive/MyDrive/dataset

[radiant sentinel]

For me dataset is in a folder called training

[whole anvil]

/content/drive/MyDrive/training/nameofthedataset

if you put your dataset inside the datasets folder then it would be /content/drive/MyDrive/training/datasets/dataset

or if you didnt create a folder inside the datasets folder
just do /content/drive/MyDrive/training/datasets

[radiant sentinel]

/content/drive/MyDrive/trainingdatasets/nameoffile

but it didnt work

[whole anvil]

you forgot to put one "/"

/training/datasets/

[radiant sentinel]

I changed it and still nothing

Hmmm

[whole anvil]

what if you try training/datasets/nameoffile instead

[radiant sentinel]

It still says no such file directory

[whole anvil]

your dataset is a zip file, folder or only the audios?

zip files doesn't work for example

[radiant sentinel]

No its just the file

I put the file in datasets

[whole anvil]

ohh ok so only do this /content/drive/MyDrive/training/datasets

try if that works

don't put the name of your audio at the end

copy and paste what i sent

[radiant sentinel]

It worked!

[whole anvil]

[radiant sentinel]

Batch size 8 right?

[whole anvil]

yep

[radiant sentinel]

I used to use a custom pre trained model but are the og models better and should i not change it

[whole anvil]

use the og models, they're better

don't change them

[radiant sentinel]

Wait i did everything and pressed train and it gives me an error

[whole anvil]

yea the colab is a bit buggy, go to the colab tab

see if its training there

should be saying epoch 1 or something like that

[radiant sentinel]

It says epoch 41 for some reason

It also says creating converter

[whole anvil]

wat, you started training just now? if u started now it should start from 1

[radiant sentinel]

Wait now it says epoch 1

That was weird

[whole anvil]

you can now open the tensorboard link that was generated when you ran the cell

scroll up in the start running cell until you find it

[radiant sentinel]

Yes I see it

[whole anvil]

nice, now you can let it train without problems

remember to not close the google colab tab

[radiant sentinel]

The rvc tab still says error but it is training right?

[whole anvil]

yeah its training don't worry, the gui is a bit broken but the training is perfectly fine

[radiant sentinel]

I refreshed the tensorboard but its just loading

[whole anvil]

hmm try reloading the tensorboard site once more (not google colab)

is a bit laggy compared to local tensorboard because is running on the internet and not locally

so if you're having some internet issues is going to be a bit slower

[radiant sentinel]

I don't get it already saved 42 epochs to my drive folder

Even tho colab says that it just saved epoch 2

And the epochs saved are folders

[whole anvil]

you have 42 .pth files?

[radiant sentinel]

folders not pth files

[whole anvil]

thats so weird, never happen to me

rvc saves two things
G and D, and .pth

it shouldnt create new folders

[radiant sentinel]

The weights folder right

Thats where the pth files gets saved

[whole anvil]

yes, pth files are saved in the weights folder

[radiant sentinel]

Their all folders

[whole anvil]

ooohhh you're double clicking them
yeah pth files are actually folders
but if you download them, they're going to download as .pth files

[radiant sentinel]

Oh yeah you're right

[whole anvil]

to download them, right click the .pth file and click download

[radiant sentinel]

I did

[whole anvil]

nice

[radiant sentinel]

But i downlpaded the 41st epoch

Even tho colab says its still at epoch 13

[whole anvil]

those epochs are from an older unfinished training of yours

the epoch 13 is actually this training

[radiant sentinel]

But i never trained this dataset

[whole anvil]

probably is from other one

maybe you tried training on colab before?

[radiant sentinel]

I haven't

Thats why i find it really weird haha

[whole anvil]

lol

sure it is

[radiant sentinel]

so i deleted everyhting and also the models in the dataset folders and logs but they keep saving new epochs in the wieghts folder

[whole anvil]

yea its supposed to do that, is saving your epochs at the frequency u set

[radiant sentinel]

Oh no so it's not gonna stop until it reached 1000 epochs!?

[whole anvil]

yup lols, remember to delete the epochs you're not going to use to save space

[radiant sentinel]

Omg lmao

Alr its fine

And how many hours can i use this colab per day

[whole anvil]

2 hours

[radiant sentinel]

Does a higher batch size result into a faster training time

[whole anvil]

yes but too high limits what your model can do
so is less versatile

[radiant sentinel]

I was considering using kaggle too

But it looks really hard to understand

[whole anvil]

its a bit complicated but the guide covers pretty much everything u need to know about kaggle

[safe cove]

Ayo? @whole anvil level 46 !!!

[radiant sentinel]

I got kaggle to work Yay!

[radiant sentinel]

@whole anvil I've used different languages in the dataset is that okay?

[whole anvil]

As long is the same person’s voice, yep

[radiant sentinel]

yeah i was just asking because i saw that some pre trains are more suited for specific languages but then again I don't think i can change the pre trains on kaggle and google collab

[whole anvil]

You can train any language on any pretrain, don’t worry, is going to work fine
For example, the original pretrain is trained in english and you can still train different languages with it

[radiant sentinel]

Alright! Thanks!

[radiant sentinel]

@whole anvil i just have one question. How do I know which target sample rate to choose

[glass oracle]

No

[daring echo]

rvc disconnected is not working for me