if g and d originally mirrored but then begin to trend down together is this still okay?

[lost tree]

Finally using tensorboard to analyze my training and was reading that ideally d and g should be mirrored but mine appear to now trend down together. I think everyhting else looks okay? Thanks in advance!

[mellow geyser]

That is fine

[lost tree]

Awesome! Is there any best practices for mode collapses? like do I just wait for a sudden drop off on g/total and use the model that's a few epochs before it drops? Just thinking out loud from the small number of image models I've trained.

[mellow geyser]

A mode collapse can be identified by 2 main characteristics:

1. Drop to a very low value in g/total and a sudden raise to the normal number (hover over it, and ignore the Smoothed value). E.g: Loss value hovers at around 35-40, suddenly dips to 10, then comes back.

2. Big spike up in d/total

There was a third but my brain did a funny and forgot :)

But those r the main ones

[lost tree]

thanks. so I assume though that its best to stop the training / use the model thats generated before it happens? like doesnt a mode collapse basically ruin the future training post collapse?

[mellow geyser]

That's the thing - a model can still improve after a mode collapse! It's of course not gonna be as good as a model without a mode collapse though, but it ain't that big of a deal if it happens once or twice.

And from your logs, don't see any huge spikes in G nor D. So you should be fine!

[lost tree]

okay awesome. I read online that the model should just be scrapped but 400 epochs is taking me 2 and a half hours. so scrapping it would be such a waste lol

Thanks for the insight. I think this trianing session is my best yet based on what I'm reading regarding metrics

[mellow geyser]

Yeah lol

[mellow geyser]

Good to know 🔥