This has been going on pretty much all weekend, sadly, despite the several incidents and them being "resolved". Until I recently tried an empty commit to kick the deployment process and try again, I had deployments sitting in "Waiting" for 5h+. The one that was an attempt to kick the process is currently at 25m and counting (despite the github actions flows being done for at least 10 minutes). Is there anything I can do to help this other than wait and hope railway starts being semi-reliable?
#Deployments (still) stuck in Waiting
outer fulcrum
alpine basaltBOT
dim orbitBOT
outer fulcrum
41f3e5b0-3233-406e-938c-ed0cbe426163
sacred sandal
abort them
outer fulcrum
Can't
sacred sandal
why?
sacred sandal
interesting, and what would happen to a new deployment
outer fulcrum
i've had like 2 succeed over the weekend while others are still sitting waiting
so on one of the services, right now, after the last good deploy, there are 5 waiting, 2 removed, 1 skipped
others are similar
sacred sandal
when was the last time you tried kicking off a new deployment
outer fulcrum
25 minutes ago, with an empty commit
sacred sandal
and whats the current state of that
outer fulcrum
Waiting
alpine basaltBOT
abstract bluff
Hey, I see you have Check Suites enabled. Are they resolving properly?
outer fulcrum
Yes, all the GitHub actions are completing successfully in approximately 10-15 minutes. So I expect the start to be delayed that long
The github actions are actually doing docker builds and pushing the images to the github registry. The railway builds are building essentially empty dockerfiles that just pull and re-push those images, which is why the check suites are on (so the docker images exist before railway tries to pull them)
abstract bluff
Hmm, I suspect there's an issue with the check suites workflow in that case. I can open a ticket for us to look into it this week.
outer fulcrum
I'd appreciate it. I'm so close to getting an alpha version of this out the door, and having to wait days for a config fix to go live has been pretty frustrating
abstract bluff
If it's a blocker, you can technically switch to using GitHub actions to deploy. https://blog.railway.app/p/github-actions
outer fulcrum
Are there any downsides of doing it that way? It seems like it is sending a tarball of the repo contents out from the action instead of railway doing a proper git clone. Is that going to cause issues for me with github over actions bandwidth or anything?
abstract bluff
I am not sure as I haven't extensively used that flow. Just suggested it as an alternative in case you couldn't wait for the fix.
outer fulcrum
After some brief testing today, I think there may be a weird condition that happens if I push to master and my prod deployment branch approximately simultaneously. Even though both the master and prod checks all succeed, I suspect that railway doesn't check or see that the master checks are successful. If i push to master and wait until it is green, and then push to prod, the deployments have been working pretty cleanly.