#pod wont start

16 messages · Page 1 of 1 (latest)

chilly topaz
#

I'm trying to open a pod with a rtx 5090 running Ostris AI Toolkit but its been sitting for about 10 minutes now after the last time I restarted it. I've tried clearing the cache, hitting reset pod, stopping and starting the pod, terminating the pod and starting a different one. No luck, the connect tab just shows "waiting for connection data" and the log tab shows "no logs found"

wild solarBOT
tribal ravineBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

chilly topaz
#

I've also tried different browsers now, Brave, Chrome and Edge. same result in each.

chilly topaz
#

changed server location, not sure which it was using before because I didnt specify, but set it to US-IL and it started up.

chilly topaz
#

ok, so I've come back to check the progress expecting it to be around 70% and its not even 1/4 of the way there. At this rate its going to be 13 to 14 hours to finish. I've done quite a few of these and none have taken this long. Watching it's resource usage shows the GPU load will peak around 57% but most of the time its between 25% to 30%. this seems abnormal to me. any of the other ones that I've done have the GPU pegged the entire time. The log shows nothing new since the image paths were decoded and the server started.

chilly topaz
#

nearly 5 hrs in, realizing its going to take forever with that one, so I tried to start a new one, and now I cant even get images to load to the dataset.

chilly topaz
#

I've had nothing but problems tonight. 4 or 5 different pods all with issues. the first very slow one, another one that started up fine, but then wouldnt upload images. another that kept spitting up errors when I tried to upload, and then another that acted like it received the dataset files but never would show an image in the dataset. at least this latest one that I started seems to be working properly so far.

pseudo bronze
#

seems like those problematic pods are slow on the internet

wild solarBOT
pseudo bronze
#

try to report it on a ticket

charred talon
#

you have to be nice to your pod for it to start

silk glacier
#

I’ve noticed several people complaining about pods trying to start AI Toolkit template

#

let me try to boot it up

#

just to confirm - are u using this template? ostris/aitoolkit:latest