#EU West is down. How do I download my railway data dump or backup if the database is down?

12 messages · Page 1 of 1 (latest)

burnt tartan
#

Hi all,

I have to download my data so I can get service back up for my customers. Is it possible if the DB is down? I have backups but I can't seem to download them.

fading carbonBOT
burnt tartan
#

EU west is down again, so if anyone can clarify ASAP that would be great

#

EU West is down. How do I download my railway data dump or backup if the database is down?

burnt tartan
#

status.railway.com is not reflecting the actual status. Besides the earlier 5+ hour outage, this is another 5+ hour outage. I've now migrated to planetscale, will monitor when railway is back online so I can recover the missing data

tight pike
fading carbonBOT
burnt tartan
#

Figured out what happened. The May 19–20 outage left our MySQL volume with a torn InnoDB page, mysqld crash-loops on startup with MY-013546 and MY-013540 (doublewrite buffer also unrecoverable). This morning's automatic daily backup snapshotted the already-corrupt state and restores into the same crash loop. Yesterday's daily backup restores cleanly, so we're back up, but we've lost ~a day of writes.

Two asks:

  • Do you retain any snapshots of my MySQL volume from between yesterday's daily and ~22:43 UTC May 19 that aren't shown in the dashboard?

  • Can you start the corrupt volume with --innodb-force-recovery=3 (or give me shell access) so I can mysqldump whatever's still readable?

fading carbonBOT
fading carbonBOT
#

New reply sent from Central Station thread:

So much for 72 hour response time. Fortunately, Planetscale is running stable. But unfortunate that you do not assist with data recovery based on your own service issues.

You're seeing this because this thread has been automatically linked to the Central Station thread.

fading carbonBOT
#

New reply sent from Central Station thread:

Discord threads don't get put into the teams queue as they are considered community support, so aren't likely to be seen by the support team. The 72 hour response time is for when you open a private thread here on the station (as outlined here: https://docs.railway.com/platform/support#pro).

Answers to your questions:

  1. They only back up what you tell them to back up, so no
  2. You can change the start command in your service settings (to sleep 999999 if you would like to ssh in without it crashing)

I'm going to guess that this was caused by the service and/or volume being suddenly killed by GCP, causing a corrupt state.

You're seeing this because this thread has been automatically linked to the Central Station thread.