I'm trying to open a pod with a rtx 5090 running Ostris AI Toolkit but its been sitting for about 10 minutes now after the last time I restarted it. I've tried clearing the cache, hitting reset pod, stopping and starting the pod, terminating the pod and starting a different one. No luck, the connect tab just shows "waiting for connection data" and the log tab shows "no logs found"
#pod wont start
16 messages · Page 1 of 1 (latest)
I've also tried different browsers now, Brave, Chrome and Edge. same result in each.
changed server location, not sure which it was using before because I didnt specify, but set it to US-IL and it started up.
ok, so I've come back to check the progress expecting it to be around 70% and its not even 1/4 of the way there. At this rate its going to be 13 to 14 hours to finish. I've done quite a few of these and none have taken this long. Watching it's resource usage shows the GPU load will peak around 57% but most of the time its between 25% to 30%. this seems abnormal to me. any of the other ones that I've done have the GPU pegged the entire time. The log shows nothing new since the image paths were decoded and the server started.
nearly 5 hrs in, realizing its going to take forever with that one, so I tried to start a new one, and now I cant even get images to load to the dataset.
I've had nothing but problems tonight. 4 or 5 different pods all with issues. the first very slow one, another one that started up fine, but then wouldnt upload images. another that kept spitting up errors when I tried to upload, and then another that acted like it received the dataset files but never would show an image in the dataset. at least this latest one that I started seems to be working properly so far.
seems like those problematic pods are slow on the internet
@chilly topaz
Zendesk Escalation
A support ticket has been opened for this conversation. Ticket #33041.
try to report it on a ticket
you have to be nice to your pod for it to start
I’ve noticed several people complaining about pods trying to start AI Toolkit template
let me try to boot it up
in the meantime, there’s another template with AI Toolkit - www.lorapilot.com
just to confirm - are u using this template? ostris/aitoolkit:latest