I've been using a Railway Postgres DB for a few weeks. Tonight, while I was busy using the DB with an app I'm building, I suddenly get "failed to connect" messages. When I logged into the Railway dashboard, the DB shows as existing, but it says I have no tables. Under Settings for the DB, trying to restart the DB also fails with "Error processing request" in the UI. Can you help me figure out what happened?
#Database suddenly unreachable
neon nymph
ionic roseBOT
silver shadowBOT
neon nymph
38b29e3b-f318-4d13-a50c-ad2242257e12
neon nymph
cool. Appreciate the fast response. it's not a production app (yet) so a little downtime won't kill me. Any idea what happened?
exotic herald
Back
neon nymph
yes it seems to be back to normal.that was fast, thank you
exotic herald
cool. Appreciate the fast response. it's not a production app (yet) so a little ...
We are performing a massive amount of VM operations to prepare for a new feature rollout (persistent disk) but we are running into vendor rate limits over in GCP. We have done everything we can to protect Team workloads but small DBs are getting a minute or so of downtime.
We did report this in #🚨｜incidents - but apparently the modification of the script is giving us 3m of downtime not 30s so we will adjust. Sorry about that.
Meanwhile, I am keeping an eye out on reports from ya. Atm we have only 5 affected out of millions of workloads- so unfortunately you are one of the unlucky ones.
(If we had more impact, I would declare a maintainence window)
neon nymph
it's all good. thanks for the info. Just FYI I think I had about 15 min of downtime.
exotic herald
Not good at all, lemme credit you.
neon nymph
don't worry about it i'm a tiny app for now.
exotic herald
No workload too small
neon nymph
it sounds like a team plan would improve the situation though right ?
exotic herald
Gave ya $20
exotic herald
Yes. But, being honest, its not good optics.
So wanna move to a system where we are more forthright about that system and expectation of service tiers.
neon nymph
yeah it's def possible for someone to play with an individual plan, with intenntion of growing into a fullblown account. so reliability matters
BUT, at the same time, responsiveness matters A LOT too. so i appreciate you guys were here, you fixed it
hope the feature rollout goes well
exotic herald
This is why I do it, for builders like you.
Thanks!
I am personally excited.
We just shipped private networking (type /beta)
and we have regions coming after this
exotic herald
Rather these are base primitives that people need to deliver at scale, so a Fly but without upfront config
Don't get me wrong, I love Kurt and team
but they are more below the stack, whereas we try to abstract it all
so our regions impl, is more turnkey rather than you having to copy your config from US-East to EU-West so on so forth
neon nymph
yeah i love how easy it was to set up a DB with you guys
exotic herald
all of app dev should feel that way
neon nymph
if you guys can make it happen that will be a big deal!
exotic herald
for us? more of a when than an if haha
confidence aside, let us know how else we can help
neon nymph
first step to creating the future is believing in it. 🚀 Good luck, will let you know if i need anything else
icy umbra
love this energy
exotic herald
Awesome!