Unable to execute functions in self-hosted appwrite deployment.

[vital steppe]

Hello there! I am running a custom deployment of appwrite on premise using the docker container. When attempting to deploy a function, the deployment succeeds without any errors or warnings. But running the function / testing it through the UI does not work and gives me a very non-descriptive server error message with no additional details. The execution tab does not show any executions of the function which means it is probably never ran.

one thing that I have noticed is that the automatically generated domains are malformed and look like this:
${function_id).https://...

could this be the reason executions through the UI are also failing?

Do you have some advice on how to start debugging this? Which environment variables to modify?

[gaunt pine]

Hello 👋 You can learn more about "Server error" by looking at docker compose logs appwrite.
If you executed function by clicking "Execute" button in Console, you it should always create execution document. If anythong goes wrong there, execution itself should show error, and if not, you can find more in docker compose logs appwrite-worker-functions

[vital steppe]

2025-01-08T11:46:56.228446795Z [Error] Timestamp: 2025-01-08T11:46:56+00:00
2025-01-08T11:46:56.228491395Z [Error] Method: POST
2025-01-08T11:46:56.228496835Z [Error] URL: /v1/functions/:functionId/executions
2025-01-08T11:46:56.228501035Z [Error] Type: Ahc\Jwt\JWTException
2025-01-08T11:46:56.228505115Z [Error] Message: Signing key cannot be empty
2025-01-08T11:46:56.228508955Z [Error] File: /usr/src/code/vendor/adhocore/jwt/src/ValidatesJWT.php
2025-01-08T11:46:56.228512915Z [Error] Line: 34

@gaunt pine thanks for your answer! could this be the issue?

do I maybe have to send the API key in the request headers when using the console / UI to execute my function?

on the web panel i cannot see any executions being logged

[gaunt pine]

Yes yes, could easily be the source of the issue.
May I ask, how did you install Appwrite on your machine? Did you follow some article, video, docs..?

[vital steppe]

I used this platform here:

• https://coolify.io/
• https://coolify.io/docs/services/appwrite

they offer a one click installer for appwrite, i do have experience using docker on it's own though

[gaunt pine]

Got it. Probably best to cc @delicate dirge or @earnest briar here, they might be familiar with Coolify integration better than me.

The error indictes that env variable _APP_OPENSSL_KEY_V1 is empty. It needs to be set to sone random string, and it serves as encryption key in many flows.

You can try to add this manually, and go from there, but Im a bit worried more variables wont be set properly

[vital steppe]

okay let me check!

[gaunt pine]

^ If this reminds you of any coolify flow you went thorugh, maybe it asked for values for those vars and you left it empty just to begin with. If so, you might recall what all variables might be missing

[vital steppe]

thanks for telling me the correct environment variable to modify

do you happen to know anything about the domain issue described above?

[gaunt pine]

Never heard of it before, which might mean it's also just env varible setup issues. Let me grab .env file from our QA server, remove any secrets, and share it with you so you can understand what variables there are, and what are example values

[vital steppe]

i managed to fix the jwt related issue thanks to your help

but now An internal curl error has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 677e6b509aff2

it seems like the function cannot be correctly resolved, probably because of the domain

[gaunt pine]

_APP_ENV=production # I recommend, so LetsEncrypt works fine (HTTPS)
_APP_EDITION=self-hosted
_APP_LOCALE=en
_APP_WORKER_PER_CORE=6
_APP_COMPRESSION_MIN_SIZE_BYTES=1024
_APP_CONSOLE_WHITELIST_ROOT=disabled
_APP_CONSOLE_WHITELIST_EMAILS=
_APP_CONSOLE_SESSION_ALERTS=enabled
_APP_CONSOLE_WHITELIST_IPS=
_APP_CONSOLE_COUNTRIES_DENYLIST=AQ
_APP_CONSOLE_HOSTNAMES=localhost,appwrite.io,*.appwrite.io,* # Use your hostname (domain) here
_APP_SYSTEM_EMAIL_NAME=Appwrite
[email protected]
[email protected]
[email protected]
[email protected]
_APP_SYSTEM_RESPONSE_FORMAT=
_APP_OPTIONS_ABUSE=enabled
_APP_OPTIONS_ROUTER_PROTECTION=disabled
_APP_OPTIONS_FORCE_HTTPS=enabled
_APP_OPTIONS_COMPUTE_FORCE_HTTPS=enabled
_APP_OPENSSL_KEY_V1=your-secret-key # Change to random string
_APP_DOMAIN=qaserver.something.com # Change to your domain
_APP_DOMAIN_FUNCTIONS=functions.qaserver.something.com # Change to your domain, keep functions. at beginning
_APP_DOMAIN_TARGET=qaserver.something.com # Same as _APP_DOMAIN
_APP_REDIS_HOST=redis
_APP_REDIS_PORT=6379
_APP_REDIS_PASS=
_APP_REDIS_USER=
_APP_DB_HOST=mariadb
_APP_DB_PORT=3306
_APP_DB_SCHEMA=appwrite
_APP_DB_USER=user
_APP_DB_PASS=password
_APP_DB_ROOT_PASS=rootsecretpassword
_APP_STORAGE_DEVICE=Local
_APP_STORAGE_S3_ACCESS_KEY=
_APP_STORAGE_S3_SECRET=
_APP_STORAGE_S3_REGION=us-east-1
_APP_STORAGE_S3_BUCKET=
_APP_STORAGE_DO_SPACES_ACCESS_KEY=
_APP_STORAGE_DO_SPACES_SECRET=
_APP_STORAGE_DO_SPACES_REGION=us-east-1
_APP_STORAGE_DO_SPACES_BUCKET=
_APP_STORAGE_BACKBLAZE_ACCESS_KEY=
_APP_STORAGE_BACKBLAZE_SECRET=
_APP_STORAGE_BACKBLAZE_REGION=us-west-004
_APP_STORAGE_BACKBLAZE_BUCKET=
_APP_STORAGE_LINODE_ACCESS_KEY=
_APP_STORAGE_LINODE_SECRET=
_APP_STORAGE_LINODE_REGION=eu-central-1
_APP_STORAGE_LINODE_BUCKET=
_APP_STORAGE_WASABI_ACCESS_KEY=
_APP_STORAGE_WASABI_SECRET=
_APP_STORAGE_WASABI_REGION=eu-central-1
_APP_STORAGE_WASABI_BUCKET=
_APP_STORAGE_ANTIVIRUS=disabled
_APP_STORAGE_ANTIVIRUS_HOST=clamav
_APP_STORAGE_ANTIVIRUS_PORT=3310
_APP_SMTP_HOST=smtp.mailgun.org # Change all SMTP stuff here if you want emails to work
_APP_SMTP_PASSWORD="..."
_APP_SMTP_PORT=587
_APP_SMTP_SECURE=tls
[email protected]
_APP_SMS_PROVIDER=sms://username:password@mock # Change SMS stuff here if you want SMS to work
_APP_SMS_FROM=[phone]
_APP_SMS_PROJECTS_DENY_LIST=
_APP_STORAGE_LIMIT=30000000
_APP_STORAGE_PREVIEW_LIMIT=20000000
_APP_COMPUTE_SIZE_LIMIT=30000000
_APP_COMPUTE_TIMEOUT=900
_APP_COMPUTE_BUILD_TIMEOUT=900
_APP_COMPUTE_CPUS=8
_APP_COMPUTE_MEMORY=8192
_APP_COMPUTE_INACTIVE_THRESHOLD=600
_APP_COMPUTE_MAINTENANCE_INTERVAL=600
_APP_COMPUTE_RUNTIMES_NETWORK=runtimes
_APP_EXECUTOR_SECRET=your-secret-key
_APP_EXECUTOR_HOST=http://exc1/v1
_APP_FUNCTIONS_RUNTIMES=php-8.0,node-18.0,python-3.9,ruby-3.1,node-22
_APP_MAINTENANCE_INTERVAL=86400
_APP_MAINTENANCE_DELAY=
_APP_MAINTENANCE_RETENTION_CACHE=2592000
_APP_MAINTENANCE_RETENTION_EXECUTION=1209600
_APP_MAINTENANCE_RETENTION_ABUSE=86400
_APP_MAINTENANCE_RETENTION_AUDIT=1209600
_APP_USAGE_AGGREGATION_INTERVAL=30
_APP_MAINTENANCE_RETENTION_USAGE_HOURLY=8640000
_APP_MAINTENANCE_RETENTION_SCHEDULES=86400
_APP_USAGE_STATS=enabled
_APP_LOGGING_CONFIG=
_APP_GRAPHQL_MAX_BATCH_SIZE=10
_APP_GRAPHQL_MAX_COMPLEXITY=250
_APP_GRAPHQL_MAX_DEPTH=4
_APP_DOCKER_HUB_USERNAME=
_APP_DOCKER_HUB_PASSWORD=
_APP_VCS_GITHUB_APP_NAME=xxx # Set all VCS stuff if you want automated Function deployments
_APP_VCS_GITHUB_PRIVATE_KEY="..."
_APP_VCS_GITHUB_APP_ID=xxx
_APP_VCS_GITHUB_CLIENT_ID=xxx
_APP_VCS_GITHUB_CLIENT_SECRET=xxx
_APP_VCS_GITHUB_WEBHOOK_SECRET=xxx
_APP_MIGRATIONS_FIREBASE_CLIENT_ID=
_APP_MIGRATIONS_FIREBASE_CLIENT_SECRET=
_APP_ASSISTANT_OPENAI_API_KEY=
_APP_MESSAGE_SMS_TEST_DSN=
_APP_MESSAGE_EMAIL_TEST_DSN=
_APP_MESSAGE_PUSH_TEST_DSN=
_APP_WEBHOOK_MAX_FAILED_ATTEMPTS=10
_APP_PROJECT_REGIONS=default

You dont need to set all of them, just use it for inspiration when you find out some of them is missing and causing problem

[gaunt pine]

Hmm this one is annoying :/ We usually only face those during development. It usually means the function crashed.

Can you share your function's source code with me, to check it quickly?

[vital steppe]

const dotenv = require("dotenv");
const Groq = require("groq-sdk");

const { Client } = require("node-appwrite");

dotenv.config();

const client = new Client()
    .setEndpoint(process.env.APPWRITE_ENDPOINT)
    .setProject(process.env.APPWRITE_PROJECT_ID)
    .setKey(process.env.APPWRITE_API_KEY);


const groq = new Groq({
    apiKey: process.env.GROQ_API_KEY
});

/**
 * Throws an error if any of the keys are missing from the object
 * @param {*} obj
 * @param {string[]} keys
 * @throws {Error}
 */
export function throwIfMissing(obj, keys) {
    const missing = [];
    for (let key of keys) {
      if (!(key in obj) || !obj[key]) {
        missing.push(key);
      }
    }
    if (missing.length > 0) {
      throw new Error(`Missing required fields: ${missing.join(', ')}`);
    }
}

export default async ({req, res, error, log}) => {
    try {
        throwIfMissing(req.body, ["model", "messages"]);
    } catch (err) {
        error("Error: ", err.message);
        return res.json({ ok: false, error: err.message }, 400);
    }

    try {
        const response = await groq.chat.completions.create({
            model: req.body.model,
            messages: req.body.messages
        });
        log("Response: ", response.choices[0].message.content);
        return res.json({ ok: true, response }, 200);
    } catch (err) {
        error("Error: ", err.message);
        return res.json({ ok: false, error: err.message }, 500);
    }
    
}

sorry for wasting your time.. it's my first time doing this..

this is basically an adapted version of the promptChatGPT starter template

[gaunt pine]

No worries 😇

[gaunt pine]

Got it. Seeoms okay on first sight, but I can imagine few things what could go wrong.

Let's try one thing. Let's take a starter, and make it into a very simple hello world fnction that just returns dummy json. And let's try to deploy that

Knowing that works can ensure us your server setup is now perfectly fine

[vital steppe]

I have a feeling it might be unrelated to the functions code and most likely related to how functions are created on appwrite

every function gets it's own subdomain right?

[gaunt pine]

Correct.

[vital steppe]

looking at the domain it looks like this 677e6b50957484e7a645.https://appwrite.syntaxbullet.com/...

the https:// scheme probably does not belong there

but it gets autogenerated as that

[gaunt pine]

I think some env variables might be set wrong such as _APP_DOMAIN, _APP_DOMAIN_TARGET, _APP_DOMAIN_FUNCTIONS. All of those should be hostnames, not URLs. So myapp.com, not http://myapp.com/

^ After fixing those and restariting appwrite, you might need to delete function and create it again - to fix the domain assigned to it

[vital steppe]

okay, let me test it, one second

all three of those are set to the hostname

the container is restarted

but the issue is that the url generated still includes the https scheme

[gaunt pine]

One small test, let's make a new function. Does new function also got bad domain?

[vital steppe]

yes

i also somehow ended up with a function being listed in the UI that gives me a 404 when i click on it

[gaunt pine]

Hmm. Does Coolify let you go into terminal of Appwrite (main api container, called just appwrite), and run commands? I would love to know if he sees new variables. Doing echo $_APP_DOMAIN_FUNCTIONS could show the value it sees

[vital steppe]

yes one second

very interesting

let me make sure i have it saved

[gaunt pine]

When Docker images gets recreated, it will fix itself. Maybe restart on Coolify only restarts the containers. You might need to look around Coolify and try to find a way to redeploy it properly

[vital steppe]

there is some weird behavior when deleting functions

the deployments get deleted but the functions still get listed

okay, i created a new function and the domain issue seems to be fixed now but i still get this:

An internal curl error has occurred within the executor! Error Number: 6. Error Msg: Could not resolve host: 677e74f8c9276\nError Code: 500

i think we are getting closer to it working

I will try a hello world function

it still cannot resolve the host..