Ubutu server shutting down when schedule backup cron

[wild laurel]

I schedule my backup at 3AM daily. when I see the log the server is shutting down. i dont know why. then the next day I check the server it is up. but the appwrite is not accessible anymore. it usually works back when I manually restart the server. but sometiem not

[wild laurel]

@wild laurel

[swift palm]

weeeird....i wonder if it's crashing or something

[wild laurel]

I have no idea too. the script should be fine as I copied and I tested manually backup. it is okay

[swift palm]

Maybe there's extra load going on at that time

[wild laurel]

do you mean this is the server issue ? when backing up, the server crush and forced to restart?

if that so, why when it restarted, the service is no longer up.

[swift palm]

Are containers not running after server restart?

[wild laurel]

it is running but the appwrite is no longer accessible

[jovial mango]

What are the logs for InfluxDB?

I think fluxdb could be shutting down or something so it's making Appwrite not being accessible

If you can get logs just after replicating the issue it will be very useful

Also what's the error you get when you can't access the appwrite instance?

500 error, timeout...?

[wild laurel]

How to get that logs

[jovial mango]

Run
docker compose logs influxdb in the folder where the docker compose file is located at

[wild laurel]

shown someting like this

[jovial mango]

And Traefik?
docker compose logs traefik

[jovial mango]

Interesting. Looks like that's the issue

You will need to stop Traefik and start after backup

But I think it should not be needed 🤔

@swift palm In the video, you stop Traefik and after backup you turn it up again. What was the reason for that?

[wild laurel]

Awww then it should have script to restart

[jovial mango]

Not everything, just run
docker compose stop traefik
and then
docker compose up traefik
If stop doesn't work, then use down

[wild laurel]

It is auto back up then i need to restart every morning

[jovial mango]

In the backup.sh you can add after everything
docker compose down traefik && docker compose up traefik and it will be restarted automatically

[wild laurel]

Is it necessary

[wild laurel]

i restarted but still the same

[jovial mango]

What? 🤯

[wild laurel]

i did thiis. yet, it is still can't connect to appwrite console

[jovial mango]

What's the error you get while trying to access?

[wild laurel]

This site can’t be reached

[jovial mango]

And network logs?

What's the error you get?

[tepid swan]

After an unexpected reboot, i would run docker stats --no-stream to check to see if there's any suspiciously low resource usage for a container and dive into the logs on it and restart it if it's stale. Depending on the specs of your machine, you could be running out of memory potentially and the process picked up by the OOM Killer, somethin like dmesg -T | grep -i 'killed process' might confirm.

[jovial mango]

Well, the machine itself is not being restarted, only the docker instance supposedly, and what's even more weird is that the containers seems to still be running when the backup ends but although that, it's impossible to enter appwrite

Anyways, @wild laurel try checking that after it restarts

[wild laurel]

it works back after restarting somehow, restart machine manually

it will eventually happen again

[wild laurel]

How do i check the network log

[wild laurel]

after restarting manually the server, it works !

[jovial mango]

That's weird. Do you have done what Evan said?

[wild laurel]

yes @jovial mango

[tepid swan]

Let's filter out the noise and look at the logs for mariadb and influx db. @wild laurel - try docker logs appwrite-mariadb 2>&1 | grep "error" and docker logs appwrite-influxdb 2>&1 | grep "error". (if those don't show anything, replace error with warn or exception) The suspicion im leaning towards is mariadb completely locking up after the sqldump and it not recovering appropriately.

[wild laurel]

show nothing

[tepid swan]

Let's try one without a filter on mariadb: docker logs appwrite-mariadb .

[tepid swan]

Thanks, I don't see anything alarming there from those logs as I get those warns when running an idle instance.

I'm curious about the following:

• what is your _APP_MAINTENANCE_INTERVAL environment variable set to? might be possible that it's conflicting with your scheduled backup
• specs of the machine - gb/ram/disksize - it's important you have at least 2gb set for swap usage

I think as a band aid, you can add a compose down/up that occurs after the backup completes.

[jovial mango]

The server has 3gb of ram. I think that's not the issue, maybe I'm wrong. Anyways @wild laurel could you please send some screenshots of htop?

[wild laurel]

you are right, I just increased to 4GB. it should be okay. storage is fine.

[wild laurel]

I schedule backup everyday at 3:00AM

[tepid swan]

Do you have any system monitoring? For my home lab I use Prometheus+Grafana w Node Exporter for system metrics and cAdvisor for container metrics. It might be helpful to see what the load is like during the backup time and see what might be causing additional load

[wild laurel]

No I dont have.

so your thinking is the peformance of machine is the factor?

[jovial mango]

Could be

[wild laurel]

what could it be . less RAM or what?

[jovial mango]

Not enough ram

[wild laurel]

4GB? backup costs that much

[jovial mango]

4gb should be enough depending on the load that the server is having at such moment

If you run the backup manually, does the same happens?

And also, when it's run automatically, does the backup complete successfully? Or it fails on being created?

[wild laurel]

no it doesnt if i do manually

[wild laurel]

how do I verify

[jovial mango]

Basically, rename the backup folder/file that is being created to something else, wait for the auto backup to complete and see if a new file/folder is being created with all the corresponding files inside it (like mariadb export, storage files...)

[wild laurel]

yes. will have a try

[wild laurel]

it should be done backup.

the appwrite is no longer access again

[jovial mango]

Interesting. Does the upload folder contain something?

I searched and some people with this issue was due to docker not being updated. You have the latest version, right?

[wild laurel]

there is

[wild laurel]

it should be the lastest

but when i open it , it failed

[jovial mango]

So the file is like corrupted? 🤔

[wild laurel]

i think so.

[jovial mango]

Do you have encryption enabled?

[wild laurel]

how do I check, it should be no, i didn't set

[jovial mango]

In the console, go to the bucket settings and there you should have an option to enable/disable. Check if it's enabled

[wild laurel]

FYI, this is right now.

[jovial mango]

This looks fine, I think?

[wild laurel]

i think so. RAM should be ok

[jovial mango]

@tepid swan what do you think

[wild laurel]

after manual restarting is okay again.

[wild laurel]

yes right "eanbled"

[jovial mango]

Then backup is correct. You can't open because it's encrypted

[wild laurel]

yep. right .

[tepid swan]

Is this a snippet of how it looks when the service is unreachable?

[wild laurel]

yes right.

[tepid swan]

This will be a tough one to diagnose. So to just to clarify - is the server getting restarted after the backup? Or are the containers dying and not recovering? I would be curious to see what your docker compose file looks like if it deviates from the template. (you can dm me if you don't feel comfortable posting here)

[jovial mango]

@wild laurel

[wild laurel]

I have no idea if is restarted after the backup. it should be NO. because when I restart the server the service will be back normal.

[wild laurel]

this happens again.

[jovial mango]

So it stopped happening and now it happens again? That's weird

What's your backup procedure?

[wild laurel]

it stopped for quite sometime.