#(ctr) backing up huge amount of data, specifically Minecraft worlds

39 messages · Page 1 of 1 (latest)

sand prairie
#

So… atm we have ~1,5tb in world data on our server, it’s 2 pre-rendered worlds, each ~750gb. In the end it shouldn’t to above 2-3 tb. including nether, end and coreprotect logs.

What would be the best solution to backup things? I have around 8tb of space (excluding the 1,5tb of the server) it’s 2 enterprise hdds, each 4tb. It’s hosted by hetzner and I hate their backup plans and costs. So I’m searching for an alternative.

I thought just using one of the drives for backups, which probably isn’t the best solution, but at
least it’s not on my pc.

Regularly downloading the whole thing is a no go since my pc is just running with 2 1tb m2 ssds and I need my space for myself.

I googled around a bit but it seems like every solution is kind of meh from what I’ve found..

Also I don’t see a solution that wouldn’t take a couple of hours with like stopping the server, putting it into .gz and moving it somewhere else. Ofc I would love to keep the downtime as low as possible..

Server is running debian btw. The harddrives are not running in raid or something for now.

rose berryBOT
#

(ctr) backing up huge amount of data, specifically Minecraft worlds

rose berryBOT
#

Hi I'm AutoThreadBot! Don't mind me, I'll just be adding the helper team to this thread so they can see it. A human will get to you soon.
You can block this bot if you don't want to see these messages, I won't mind.
<@&525394568410038282>

nocturne rover
#

I actually REALLY need a system like that

#

Really really badly

nocturne rover
#

And for backing up

#

Have u tried any cloud service like one drive?

sand prairie
nocturne rover
#

Oh in that way

#

Alright anyway back to backups

#

One drive?

sand prairie
# nocturne rover Have u tried any cloud service like one drive?

Yeah this could be a solution… but finding the space is last prio for now… but the time it takes to put backup everything into .gz etc is enormous. I know there are systems which can backup and store petabytes of data in like seconds. But this stuff is fully out of range for me

#

It’s like 2-3 hour downtime just to pack it into .gz and upload it to somewhere else.

#

But I’m sure there’s faster stuff than just packing it… like doing it constantly and just uploading it in a 5 minute downtime and go back to normal after

#

Also as a probably bad example… but servers like 2b2t got backups aswell, those have worlddata in the petabyte range but the server is online pretty much 100% of the time as far as I know.

I just can’t find anything to do this except for enterprise 5000$+/month solutions with multiple servers working together and whatnot… but I’m sure there is something cheap and useful for not amazon sized companies lol

nocturne rover
#

Hmm

#

I dont know then this is a LOT of data

pallid pawn
#

@sand prairie I recommend borg backup. The data is compressed and encrypted, and borg also only saves the changes since its last backup so it saves a ton of space. For example, my server with ~200GB of data has dozens of backups with data retention policies set in borg. I do one every 3 hours, and the full backup takes under 2 minutes. Total used space for all backups right now is about 240GB. I rented a dedicated storage machine from hetzner and stream the backups into it (borg supports backups over ssh), but you can also do it on a separate mounted drive. There are even some solutions to sync borg backups to cloud providers like AWS S3 so you have the redundancy, and S3 is super low cost.

#

Oh, and I do my backups live. No server downtime.

#

I have a shell script that runs the /save-off command on the servers, waits 30s for the chunk queues to drain in paper, do the live backup, and then /save-on

#

Let me know if you have any questions

#

Oh, and one more thing - because it's a live backup, and I am lazy and use CoreProtect's flat sqllite system, I don't backup the CoreProtect data. The plugin will write data to the flat file and corrupt the backup. Obviously if you do the backup as a part of a nightly restart or something, you could back it up. But I don't see CP logs as super critical.

#

borg is free and open source

nocturne rover
#

Oh yeh that seems like an absolutely solid solution

#

Imma actually need to look into that myself too

sand prairie
#

Thank you I will look into this in the next few days :) exactly what I was looking for from what it sounds like

worldly cragBOT
#
Thread Closing Reminder

Has your issue been resolved, or your question been answered?
If so, please use the </resolved:1028673926114594866> command to close your thread.
Or </invalid:1028673926898909185> if it's not possible to resolve.

If not yet resolved, please reply below to tell us what you still need.

(Note that if there is no reply for a few days, this thread will eventually close itself.)

#

@sand prairie

worldly cragBOT
#
Thread Closing Reminder

Has your issue been resolved, or your question been answered?
If so, please use the </resolved:1028673926114594866> command to close your thread.
Or </invalid:1028673926898909185> if it's not possible to resolve.

If not yet resolved, please reply below to tell us what you still need.

(Note that if there is no reply for a few days, this thread will eventually close itself.)

#

@sand prairie