Function Timed Out During Cold Start?

[spring widget]

Function timed out during cold start.\nError Code: 400

[spring widget]

randomly super simple function gets Function timed out during cold start.

It just counts documents in the collection using an offset, but if it's gonna error it should error like a man

I have 50 workers enabled

[spring widget]

upgraded executor to 0.5.7, will report back

[spring widget]

@radiant sun ```
Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66b78e23d04a0\nError Code: 500

now this instead of cold start

[spring widget]

@radiant sun

Successfully removed exc1-65e61e3a6f3bfee5dac4-669a90b3df369f01ffae
Skipping runtimes stats loop due to error: Docker Error: EOF
EOF
EOF
EOF
EOF
EOF
EOF
EOF
EOF
EOF
Successfully removed exc1-65e61e3a6f3bfee5dac4-669a95c4e6dfc83d91ac

causing my executor to die

or something

[radiant sun]

These are logs for removing inactive runtimes

[spring widget]

hm, I'll find the log then

my functions are still failing when being used a lot, but

also, the upgrade to 1.6 could fix it

[spring widget]

Successfully removed exc1-65e61e3a6f3bfee5dac4-669a9427dc6fa7bfcafa
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 104. Error Msg: Connection reset by peer
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66ccf403df0ac
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66ccf403df0ac
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66ccf403df0ac
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
Skipping runtimes stats loop due to error: Docker Error: EOF
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 104. Error Msg: Connection reset by peer
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
Skipping runtimes stats loop due to error: Docker Error: EOF
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 104. Error Msg: Connection reset by peer
[Error] File: /usr/local/app/http.php
[Error] Line: 1073

@radiant sun so it looks like some of the extra workers failed or whatever these are

Preparing for start ...


Starting ...
Segmentation fault (core dumped)

I wonder if it's because I am running WSL2

so maybe the internal proxies bounce around the request

[radiant sun]

ooooh this tracks! sooo the containers crashed/exited due to memory problems (you can search online for that 139 exit code). This is problematic for Appwrite because Appwrite doesn't know the container doesn't exist anymore and still tries to send requests to the container, resulting in the "could not resolve host" error

[spring widget]

Interesting, so it’s a memory overload error? Is it a setting or the host machine running out of memory?

[radiant sun]

Could be either

[spring widget]

Any way to up appwrite’s memory limit?

[radiant sun]

There are some memory environment variables. Besides that, it would be system memory or docker settings or wsl settings

[spring widget]

copy, oh the wsl was an accidental mention, that’s in the duplicate function deployment idea thread but put it in here accidentally

[spring widget]

@radiant sun

Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
Running maintenance task ...
Maintanance task finished.
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 104. Error Msg: Connection reset by peer
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 110. Error Msg: Operation timed out
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66d201e16937b
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
Running maintenance task ...
Maintanance task finished.
[Error] Type: Exception
[Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66d201e16937b
[Error] File: /usr/local/app/http.php
[Error] Line: 1073
Skipping runtimes stats loop due to error: Docker Error: EOF

got some new ones

[radiant sun]

Can you get this with timestamps.

That said, connection reset by peer typically happens when a function is killed while it's processing. This can happen if it hasn't completed successfully recently. Then, maybe it wasn't removed properly and Appwrite still thought it was around

[spring widget]

hm, then how do I stop that from happening?

2024-08-30T16:56:57.583946800Z Running maintenance task ...
2024-08-30T16:56:57.585047860Z Maintanance task finished.
2024-08-30T17:55:40.016740023Z [Error] Type: Exception
2024-08-30T17:55:40.016770481Z [Error] Message: Internal curl errors has occurred within the executor! Error Number: 104. Error Msg: Connection reset by peer
2024-08-30T17:55:40.016823370Z [Error] File: /usr/local/app/http.php
2024-08-30T17:55:40.016888092Z [Error] Line: 1073
2024-08-30T17:56:25.004575223Z [Error] Type: Exception
2024-08-30T17:56:25.004612623Z [Error] Message: Internal curl errors has occurred within the executor! Error Number: 110. Error Msg: Operation timed out
2024-08-30T17:56:25.004619996Z [Error] File: /usr/local/app/http.php
2024-08-30T17:56:25.004679679Z [Error] Line: 1073
2024-08-30T17:56:46.981708717Z [Error] Type: Exception
2024-08-30T17:56:46.981751697Z [Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66d201e16937b
2024-08-30T17:56:46.981781944Z [Error] File: /usr/local/app/http.php
2024-08-30T17:56:46.981901470Z [Error] Line: 1073
2024-08-30T17:56:57.583151560Z Running maintenance task ...
2024-08-30T17:56:57.584279813Z Maintanance task finished.
2024-08-30T17:58:28.763531273Z [Error] Type: Exception
2024-08-30T17:58:28.763568834Z [Error] Message: Internal curl errors has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 66d201e16937b
2024-08-30T17:58:28.763580817Z [Error] File: /usr/local/app/http.php
2024-08-30T17:58:28.763687156Z [Error] Line: 1073
2024-08-30T18:00:22.340294083Z Skipping runtimes stats loop due to error: Docker Error: EOF
2024-08-30T18:00:22.340330743Z 
2024-08-30T18:56:57.583738362Z Running maintenance task ...
2024-08-30T18:56:57.589786967Z Maintanance task finished.

more complete

[spring widget]

that's 4 times today I've had to restart it

[radiant sun]

How much time between executions?

[radiant sun]

You can try to increase _APP_FUNCTIONS_INACTIVE_THRESHOLD

[spring widget]

a good amount

it's not launched yet

[radiant sun]

Ya so probably killed while executing because it hadn't completed successfully.

[spring widget]

But it completes successfully almost every time, and I was using an image that’s completed successfully every time

It only fails on small images

[radiant sun]

How long was it running before it was killed?

[spring widget]

117 ms

It normally runs for 0.5-5s, depending on image size

[radiant sun]

And this is Appwrite 1.5.10?

What version executor?

[spring widget]

1.5.8 and 0.5.7 I believe

[spring widget]

should I upgrade to 1.5.10?

you had me upgrade my executor to try and fix it, and it did fix the cold start stuff

but this one keeps popping up

[radiant sun]

Hmm timing looks odd...the maintenance task started at 17:56 but the connection reset error was at 17:55.

Did you check the status of the container using docker ps -a?

[spring widget]

I don't need to, I can see it's running in portainer

it has the status right next to it

[radiant sun]

Of the runtime container that exited

[spring widget]

ohhhhhhh

one sec

I don't see it anywhere

[radiant sun]

It may be purged already. Check it next time

[spring widget]

why does it error instead of trying to start another one?

it's 100% not memory related

[radiant sun]

If it's exit code 137 and the status OOMkilled is true, then it is.

[spring widget]

that doesn't make sense though

I have 32 gigs of ram on this thing, and only 3 appwrite functions, and 20 users total, all of whom are in the company and are not actively testing it, && we have 0 memory limit set for functions

it's dying on a cold start and just reporting errors, I don't think RAM actually has anything to do with it

unless there's an artificial cap

[radiant sun]

If OOMKilled is true, docker is reporting it killed the container because of memory

[spring widget]

but that just means, to me, that there's a bug. I've monitored the system, so either there's an artificial limit, a bug, or

[radiant sun]

The only thing we have is that memory environment variable.

If there's any other problem it's docker and your system

[spring widget]

hm, well it's happening on multiple systems, not just this one, but both are on 1.5.8

[spring widget]

so basically @radiant sun what I've figured out

is if I have the function open in the console, it happens

[spring widget]

So currently deploying a 10 MB ( I know it's big ) function in typescript, and getting internal curl error 104

So my worker-usage-dumps is crashing again, redis this time

[radiant sun]

weird....how are your redis containers and logs?

[spring widget]

normal

[radiant sun]

hmm it exited. do you know why?

[radiant sun]

timestamp?

[spring widget]

lemme restart appwrite and see if I can get it to go down again, I think that was a docker update auto-initiated

[spring widget]

I still am getting this too from the CLI

is it possible because I have the _APP_WORKERS_NUM set to 50 that's doing something?

maybe 3 are picking it up at the same time lol I'm really not sure

[radiant sun]

i don't think so.

I'd try to enable traefik access logs or add some logging to the create deployment api endpoint to see if the endpoint is being triggere more than it should

[spring widget]

@radiant sun got another

Preparing for start ...


Starting ...
Segmentation fault (core dumped)

this was in the cold start

DM'd you inspect of the dead container

[radiant sun]

Seg fault lol.

[radiant sun]

The exit code was 139.

Googling that shows stuff like https://stackoverflow.com/questions/55508604/docker-is-exited-immediately-when-runs-with-error-code-139

[radiant sun]

What logs were these?

I wonder if the seg fault was from docker or bun 🧐

[spring widget]

do you use Bun a lot?

Has the runtime been updated past 1.0?

It's on like 1.22 now

[spring widget]

@radiant sun is it possible at all that it's because the Appwrite service is started by Portainer? Like would it possibly be caused by some sort of permission issue with it not being able to start the container with bun specifically?

I've had other functions like Python and others deployed for months now and they have never once failed, but Bun fails almost every time on cold boot

[subtle tundra]

Have you tried the same code with node? I had some troubles with bun in the past, that wasn't a problem with node

[spring widget]

No, and that's sort of my point

[spring widget]

@radiant sun so I just deployed a Nodejs function... no duplicate functions, no executor errors

@void flax I don't know exactly how to replicate it, but I can give you a function that always ends up failing for me if you want, either way I can confirm the bun runtime is effed

[rain canopy]

Are you logging many things?

Hmmm the latest time I've seen a similar error was when trying to run an ubuntu server virtualbox VM with Dart. It was just not running, probably due to docker issues with the machine 🤔

[spring widget]

yes I am, this is not a fixable thing on my end

[rain canopy]

What's your environment?

Are you using some kind of virtualisation, etc?

[spring widget]

32 gb ram, 4 cpu's, etc.

portainer, same env as running it on its own, just my own traefik proxy instead

I am currently trying to get my function to build in an Appwrite function container, but it's unfortunately 17 MB lmao