woeful warren
The model training stopped in the middle of the night with I/O error, apparently it is due to physical disk problem, and i tested it is randomly occur. the consequence is that make my pod idling for at least 6 hours, and i paid for it.
I can provide you the log and my pod number
reef sableBOT
raven widget
The model training stopped in the middle of the night with I/O error, apparently...
hi I am facing the same issue on two different pods accessing the same network storage. Did you found a solution or the reason? was it a physical disk issue? thanks!
elder cosmos
Maybe contact support, they can check if there is an issue with it
yes provide your pod id here
btw does your network-storage has enough free space?
raven widget
btw does your network-storage has enough free space?
it was at 77% with around 70GB free. I terminated both pods and extended the network storage for another 100GB more. For now the error did not happen again. But I don't understand why that helped (if it actually did, they are still running)
elder cosmos
Have you reported this in contact button in website? ( support requests )
raven widget
Have you reported this in contact button in website? ( support requests )
Yes, I just did
elder cosmos
Nice
pine fable
Same thing repeatedly happens to me every few days.
elder cosmos
Make a support ticket providing your pod ID, support can check the server if anything is wrong
pine fable
is there a general reason as to why this happens? I output all logs to an output.log file
which can be a bad idea
?
elder cosmos
is there a general reason as to why this happens? I output all logs to an output...
Not sure what the support said for this