[SOLVED] Addressing Appwrite Freezing Issues: Seeking Guidance on Cron?

[acoustic agate]

I've noticed that Appwrite stops responding to requests when CPU usage exceeds 50%, occurring 2-3 times daily, necessitating a server restart. This happens under stress and randomly with low load. To mitigate this, consider scheduling cron jobs for regular cache and memory cleanup. Despite checking error logs, the freeze events aren't captured. Any suggestions on where to find relevant logs for a more thorough investigation?"

[winged geode]

What are the server specs?

And what things are being performed when the spikes occur?

[acoustic agate]

2 GB Memory / 50 GB Disk / AMS3 - Ubuntu 22.04 (LTS) x64
I use external S3 for storage files.

[Disk Space]
🟢 Uploads Volume has 39.53GB free space (23.74% used)
🟢 Cache Volume has 39.53GB free space (23.74% used)
🟢 Config Volume has 39.53GB free space (23.74% used)
🟢 Certs Volume has 39.53GB free space (23.74% used)

[winged geode]

Could be due to it running out of free memory 🤔

Do you have set any swap?

[acoustic agate]

it occurs suddenly without any warnings, after restart it works well, is appwrite cleans its cache regularly or not ?

[winged geode]

You can check it with htop or swapon -s

[winged geode]

Interesting, I think yes, but no idea what's the interval

How many RPS are you handling? Are they mainly related to storage?

[acoustic agate]

  CPU[|||||||                                          9.7%]   Tasks: 130, 536 thr; 1 running
  Mem[||||||||||||||||||||||||||||||||||||||||||1.69G/1.92G]   Load average: 0.26 0.34 0.30
  Swp[                                                0K/0K]   Uptime: 10:33:14

i set an s3 do server in .env file, so it use 5TB of digital ocean spaces.

appwrite buckets store only metadata.

[winged geode]

I suppose that's the issue, you're running out of memory. I recommend setting at least 2gb or 4gb of SWAP as Ubuntu recommends. In my case I have 2gb and seems enough.
I recommend this tutorial, but be careful anyways 😅
https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

So that way appwrite will run a bit slower instead of getting killed and freezed when it doesn't have enough memory

In my case it froze when using/compiling dart runtime and that solved the issue

[acoustic agate]

let me try that one yes and i can increase memory as well as i understand.

[acoustic agate]

actually i decided not to use cloud functions because of the same issue. only push notifications and same basic things, deploying and executing may cause freeze as well.

So thank you for your help, i will try to increase memory and set swap. 😉

[winged geode]

I'm using function in a pretty similar server and as said, with swap it works very well and stable

Never faced any issue after setting swap

Not even slowness

[acoustic agate]

let me try to set swap first! Yea you left the good idea here

[winged geode]

Perfect, tell me how everything goes after setting it 😁

[acoustic agate]

root@appwrite:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       984Mi        65Mi       396Mi       913Mi       478Mi
Swap:          4.0Gi        63Mi       3.9Gi

let see w/ these configs 😄

vm.swappiness=10
vm.vfs_cache_pressure=50

[winged geode]

That's perfect

I didn't have modified the cache pressure, but if it's recommended, probably it's better

[acoustic agate]

lets see , will update this thread tomorrow 😉

[winged geode]

Thanks! 💯

[acoustic agate]

it worked, no freeze, no restart it is stable now if i do stress test as well.

[winged geode]

Magic heheh

Seems it was due to memory being filled up

[winged geode]

@acoustic agate Since it worked well, I'm closing this post, but if you face the same issue again, please tell me and reopen the post