#[SOLVED] Cloud functions fail under little to moderate load

80 messages · Page 1 of 1 (latest)

orchid flax
#

I have a 4GB memory, 2 Intel vCPUs droplet in DigitalOcean. The server works fine when there's only one or two person are browsing. If more people(<20) use it concurrently, it becomes really slow and all the cloud functions begin to fail to execute.
I get this error:
An internal curl error has occurred within the executor! Error Msg: Operation timed out
I tried wrapping the whole function with try catch. But the same error remains. I checked the CPU and memory usages, they aren't that high.
This is a very big problem for us, as we begin to go to production. Can anyone help? Thank you!
PS: This error might happen randomly too, in no load, but it is certainly more apparent when there is a number of concurrent users.
And I have set the timeout to 5 minutes. The same error happens after waiting.

prisma pulsar
#

How are your triggering functions?

#

By calling them from client side?

orchid flax
prisma pulsar
#

Also an user is triggering multiple functions?

#

Or just one at once

orchid flax
#

A random fail like this:

image.png
orchid flax
# prisma pulsar Or just one at once

It should be only one but, as my framework(SvelteKit) likes to preload things on mouse hover, I can see a situation where there might be two function calls in a very short time.

prisma pulsar
#

Then try running them asynchronously

#

When calling function execution, define the async parameter as true

orchid flax
#

And can you please explain what it actually does? If it runs the functions in the server asynchronously, shouldn't that have been the default?

prisma pulsar
prisma pulsar
orchid flax
#

I see, thanks!

prisma pulsar
orchid flax
orchid flax
# prisma pulsar Another workaround is probably increasing timeout, that could help, but probably...

I will probably need to try this, but I would like a suggestion.
I created a lot of (around 15) functions for fetching data alone. I thought it was a good idea because, let's say, I'm fetching a post. The post document would have an attribute called user, which will just hold the user ID. So using that ID, I would need to get the name, DP etc from the users collection. I have to do this for each post I wanted to display, right? So I thought, if I move all these document reading to a function it would execute really fast, because the function could access the DB really fast as they're on the same machine. But it looks like the function again goes through internet -> trafeik -> appwrite -> database etc.
So now, I could do the following:

  1. Either go with the tunnelling method like the issue you linked
  2. Or, I could stop using the functions and move all my data fetching codes to the framework(SvelteKit, in this case)
    Which one do you think would be better? Thanks.
limpid bobcat
#

From what I can see your problem is indeed the one in my issue D5 mentioned before.

As you have 2 vCPUs, that means you have 12 workers, so if you get 20 concurrently execution of your function, then it will always lead to an Operation timed out

From your project planning this is indeed the right way to go. As this what Appwrite functions meant to be, to be able to expand your backend without having the need to use any other backends, after all Appwrite is a BaaS 😉

And even if you'll add host routing to your functions you'll still gonna face the recursive workers calls.

I'll suggest as in that issue, add another container to your docker-compose file, set your function to request that URL, and, from what I've expreince - with this method - Appwrite was able to serve all function execution calls.

prisma pulsar
orchid flax
limpid bobcat
prisma pulsar
#

Probably that will be more efficient in the long term

#

And getting client-sided the user name corresponding to such ID

#

Note that with queries you can select and get only the needed attributes, so if you only need name, you don't need to get all details from the user document

orchid flax
limpid bobcat
# limpid bobcat You protect your self also from this

Like this
Create another file call it docker-compose-functions.yml
In the same folder.

This could be the content:

version: '3'

services:
  appwrite-functions:
    image: appwrite/appwrite:1.3.7
    container_name: appwrite-functions
    restart: unless-stopped
    networks:
      - appwrite
      - runtimes
    volumes:
      - appwrite-uploads:/storage/uploads:rw
      - appwrite-cache:/storage/cache:rw
      - appwrite-config:/storage/config:rw
      - appwrite-certificates:/storage/certificates:rw
      - appwrite-functions:/storage/functions:rw
    depends_on:
      - mariadb
      - redis
      - influxdb
    environment:
      - _APP_ENV

volumes:
  appwrite-cache:
    external: true
    name: appwrite-cache
  appwrite-uploads:
    external: true
    name: appwrite-uploads
  appwrite-certificates:
    external: true
    name: appwrite-certificates
  appwrite-functions:
    external: true
    name: appwrite-functions
  appwrite-config:
    external: true
    name: appwrite-config
  appwrite-executor:
    external: true
    name: appwrite-executor 

networks:
  appwrite:
    name: appwrite_appwrite
    external: true
  runtimes:
    name: appwrite_runtimes
    external: true

Make sure to add all the variables.

Now you can start and stop this compose file individually

docker compose -f docker-compose-functions.yml up -d
docker compose -f docker-compose-functions.yml down

Using this method will separate the changes you've made from Appwrite base file.

prisma pulsar
orchid flax
orchid flax
prisma pulsar
#

Note that this will make all data in the user document visible, so for example if you have details, etc. In such case probably it could be better using a function

orchid flax
#

Anyhow, thank you so much guys! ❤️

prisma pulsar
#

Another alternative is to hide sensitive data by storing it in user preferences (account API)

#

Sorry to make so many proposals, stopping now 😅

orchid flax
limpid bobcat
#

Nope, as you'll need to change the env for each one.

orchid flax
# limpid bobcat Nope, as you'll need to change the env for each one.

Hi! I hope you're still there.
I believe the new appwrite-functions container is running. But the functions still fail with this: {"message":"connect ECONNREFUSED 172.18.0.2:443","status":"failed"}, as if there's no endpoint called http://appwrite-functions/v1.
Can you help?

limpid bobcat
#

Sure,

#

Can you show the output of

docker ps -a
orchid flax
limpid bobcat
#

Maybe you've put https with s in APPWRITE_FUNCTION_ENDPOINT?

orchid flax
#

I didn't create a separate file for the container. I appended the codes at the end.

limpid bobcat
#

It looks like in the logs that it tried to access https 443 resource

limpid bobcat
#

Mmm

#

I know

orchid flax
limpid bobcat
#

You have _APP_OPTIONS_FORCE_HTTPS as enabled?

orchid flax
#

Let me check. I might.

limpid bobcat
#

In the new container only
set

 - _APP_OPTIONS_FORCE_HTTPS=disabled
#

Then,

docker compose down && docker compose up -d
#

And retry.

orchid flax
#

Yes I had it enabled!

orchid flax
limpid bobcat
# 
 if (App::getEnv('_APP_OPTIONS_FORCE_HTTPS', 'disabled') === 'enabled') { // Force HTTPS

The default is disabled so you can remove it altogether - in the new appwrite-functions container

orchid flax
limpid bobcat
#

👍

orchid flax
#

I can't thank you enough. You'll be remembered.

#

So cool

limpid bobcat
#

Glad it worked
ablobattention

orchid flax
limpid bobcat
#

How complex is the function?
If you don't cross the max workers in a given time then it should take the same, as this is just Appwrite.

The benchmarks that you can see in the issue are showing faster times when Appwrite is out of workers.

So if you have 10 concurrent connection, then you won't notice any different.
But, if you more connection then your available workers - 12 in your case - then you'll see a significant changes - some time even crucial ones as you can see in the Benchmark table.

#

What this semi-solution guarantee is that you'll never ran out of workers, and all of your request will be respected - maybe no in the same second but eventually - all of them will get execute.

orchid flax
orchid flax
limpid bobcat
orchid flax
#

Again, many many thanks to you. It's 5 AM in the morning and I feel so relieved.

limpid bobcat
leaden glacier
limpid bobcat
leaden glacier
orchid flax
#

[SOLVED] Cloud functions fail under little to moderate load

orchid flax
#

Hello! Sorry to disturb you, @limpid bobcat
I was using your tunneling method with great success. But I just updated Appwrite to the latest version, 1.4.4. Now I get an error that saysThis domain is not connected to any Appwrite resource yet. Please configure custom domain or function domain to allow this request., which is obviously implying http://appwrite-functions that I have in my functions. How do I fix it?
I tried adding this domain in the Appwrite console but it can't seem to pass the certificate validations for obvious reasons.

#

If anyone could help it'd be very awesome.

prisma pulsar
#

I think that's because the new changes in the new Appwrite functions

#

Binyaming will guide you better than me 😅