Transmit large array to function | Appwrite | Page 1

arctic surge Jun 22, 2023, 5:49 PM

#

I try to transmit a large array via function. I get this error:

function:  AppwriteException: Invalid data: Value must be a valid string and at least 1 chars and no longer than 8192 chars

How I can solve this problem?

#

array length is 5054 and the chars are greater then 8192. How I can sent more chars?

dim epoch Jun 22, 2023, 5:58 PM

#

You'll probably need to use some zip solutions, check if this 👇 is helping you.

Pako.js would be very useful if you're using js-functions
https://github.com/nodeca/pako

This is a demonstration of how it should go.
In your **client **side

const pako = require('pako');
const {toBase64, fromUint8Array} = require('js-base64');

const data = { my: 'function', data: [1, 2], so: 'big!!!!' };
const compressed = pako.deflate(JSON.stringify(data));
const string = toBase64(fromUint8Array(compressed));

await functions.createExecution('F_ID', string);

In your **server **JS function

const pako = require('pako');
const {fromBase64, toUint8Array} = require('js-base64');

const bytes = toUint8Array(fromBase64(req.payload ?? '');

const restored = JSON.parse(pako.inflate(bytes, { to: 'string' }));

arctic surge Jun 22, 2023, 6:05 PM

#

server-side im using .js
client-side .ts

#

I guess I can use ths: https://www.npmjs.com/package/@types/pako

npm

@types/pako

TypeScript definitions for pako. Latest version: 2.0.0, last published: a year ago. Start using @types/pako in your project by running npm i @types/pako. There are 119 other projects in the npm registry using @types/pako.

dim epoch Jun 22, 2023, 6:06 PM

#

Yep.

#

Give a try
And let us know.

arctic surge Jun 22, 2023, 6:06 PM

#

okay, thanks

#

const compressed = pako.deflate(JSON.stringify(data));

return await this.appwrite.functions.createExecution(functionId, compressed);

The argument of type "Uint8Array" cannot be assigned to the parameter of type "string".

#

this error cames for compressed in functions.createExecution(functionId, compressed)

dim epoch Jun 22, 2023, 6:27 PM

#

I've update the snippet, try now.

arctic surge Jun 22, 2023, 6:38 PM

#

stringData looks like that:

#

that is the latest code:

        const compressed = pako.deflate(JSON.stringify(data));
        const stringData = new TextDecoder().decode(compressed);
        console.log(stringData)
        return await this.appwrite.functions.createExecution(functionId, stringData, async);

#

same error

dim epoch Jun 22, 2023, 6:39 PM

#

Yep, sorry
Let me run it once on my function.

arctic surge Jun 22, 2023, 6:39 PM

#

AppwriteException: Invalid data: Value must be a valid string and at least 1 chars and no longer than 8192 chars
AppwriteException: Invalid data: Value must be a valid string and at least 1 chars and no longer than 8192 chars

#

That is my data what i want to try to submit to my appwrite function

Bildschirmfoto_2023-06-22_um_20.43.19.png

dim epoch Jun 22, 2023, 6:44 PM

#

Okay

#

We also need to use js-base64
I'll update it the snippet

dim epoch Jun 22, 2023, 6:49 PM

#

dim epoch You'll probably need to use some zip solutions, check if this 👇 is helping you....

Done

arctic surge Jun 22, 2023, 6:50 PM

#

ckecking

#

unfortunately still the same error.

const compressed = pako.deflate(JSON.stringify(data));
const stringData = toBase64(fromUint8Array(compressed));
return await this.appwrite.functions.createExecution(functionId, stringData, async);

#

dim epoch Jun 22, 2023, 6:59 PM

#

Using this shrinked a 20K characters to just 4K nad was able to go the function

#

Can you share the payload data?

arctic surge Jun 22, 2023, 6:59 PM

#

of corse

dim epoch Jun 22, 2023, 6:59 PM

#

In network tab

arctic surge Jun 22, 2023, 7:01 PM

#

📎 message.txt

dim epoch Jun 22, 2023, 7:02 PM

#

Mmm, this one is 20K long
Meaning is more then the 8192 chars limit.

arctic surge Jun 22, 2023, 7:02 PM

#

yes chatgpt says 10K 🙂

dim epoch Jun 22, 2023, 7:03 PM

#

arctic surge yes chatgpt says 10K 🙂

#

With this amount of data you'll need to choose a different approach

#

You'll need to use the Appwrite Storage module.

Upload the file to storage using createFile https://appwrite.io/docs/server/storage?sdk=web-default#storageCreateFile
In the function fetch the file using getFile https://appwrite.io/docs/server/storage?sdk=web-default#storageGetFile
Then, just delete it https://appwrite.io/docs/server/storage?sdk=web-default#storageDeleteFile

#

Or

arctic surge Jun 22, 2023, 7:06 PM

#

Okay, I feared that 🙂
I tried to find a solution without storage.

dim epoch Jun 22, 2023, 7:07 PM

#

dim epoch Or

You can dived the Base64 string into 8000 chars chunks, then in the function you can try do maybesomthing like this

for(let chunk in chunks){
   await this.appwrite.functions.createExecution(functionId, JSON.stringify({lastChunk: false, chunk: chunk }), async);
}

#

It's possible but will require some extra work from your side.

#

And its actually good option if you want to avoid storage.

arctic surge Jun 22, 2023, 7:09 PM

#

but thanks for your help. I will try to split the array into chunks less hen 8000 chars. If i find something. I can share it with you

dim epoch Jun 22, 2023, 7:09 PM

#

👍

arctic surge Jun 22, 2023, 7:09 PM

#

But why is there a limitation from appwrrite. Do you know it?

dim epoch Jun 22, 2023, 7:10 PM

#

Probably against abuse

arctic surge Jun 22, 2023, 7:11 PM

#

hmm, okay. but for self host there should be a option to change the limitations. but its okay. We can find something

dim epoch Jun 22, 2023, 7:12 PM

#

You are on self host?

arctic surge Jun 22, 2023, 7:13 PM

#

yes

#

im not using the cloud

dim epoch Jun 22, 2023, 7:17 PM

#

If you want to change it in a self-hosted one, you'll need to edit these two lines
https://github.com/appwrite/appwrite/blob/master/app/executor.php#L462
https://github.com/appwrite/appwrite/blob/master/app/controllers/api/functions.php#L1025

In order to edit and leave your Appwrite up to date, like you can see here
#1120778518360498276 message

Also check this
#1115191629733703730 message
#1115191629733703730 message

#

Change it to let's say 100000
You can still use the zip choice to get shorter upload times.

arctic surge Jun 22, 2023, 7:20 PM

#

nice, I will try that definitly. Big thanks. That would help. I will keep you up-to-date

arctic surge Jun 22, 2023, 9:22 PM

#

the response payload is also limited. It trims in the and that affacting the large object. That means the response payload cuts after some limit. You know in the code where to change it?

dim epoch Jun 23, 2023, 1:10 AM

#

What error do you get?

#

Any time you see this line

  \mb_strcut($res, 0, 1000000), // Limit to 1MB

Like in here https://github.com/appwrite/appwrite/blob/master/app/executor.php#L578
It's when the limit of the response is set 1MB, just change it to whatever number you like.

#Transmit large array to function