#[CLOSED] list limit

67 messages · Page 1 of 1 (latest)

lusty vigil
#

Can I increase the limit of the number of documents returned.

Query.limit(500)

Will it be possible? Currently, the limit is 100 I think

pine abyss
#

You can up to 5000

lusty vigil
#

Got this error

Screenshot_2023-05-21_at_11.40.43_PM.png
pine abyss
#

Oh this is the cloud

#

?

lusty vigil
#

Yes

pine abyss
#

So the limit was removed but only in version 1.3.0+

#

The cloud is still on 1.1.x

lusty vigil
#

What should I do then?

pine abyss
#

You can offset the list

lusty vigil
#

What do you mean?

pine abyss
#

This is the right link

lusty vigil
#

Okay got it

#

Will check it out

pine abyss
#

👍

lusty vigil
#

Thank you

lean harness
#

[CLOSED] list limit

icy pecan
# pine abyss You can up to 5000

Hi @pine abyss if I call listDocuments without Query.limit, what is the default limit used? 100 or 5000. Can you share please the link to source code where we can find the process of applying queries

icy pecan
#

I just made some tests, the limit default is 25, and there is no maximun number to be used in limit() I can query 100K if I want, that IS GREAT cause I have some tables with a lot of records.

#

The only problem is "total" return a maximum of 5000 (so bad) cause it would be great if return r}eal total

pine abyss
icy pecan
#

Thanks, this is helpful.
Just to be clear, how could I query 10K rows if the limit is 5000? (I used postman)

pine abyss
#

You'll need to use cursor pagination
https://appwrite.io/docs/pagination#cursor-pagination
In which your getting value after the last ID Query.cursorAfter(lastId),
So you can do that

let docs = [];

let data = await databases.listDocuments(
    '[DATABASE_ID]',
    '[COLLECTION_ID]',
    [
        Query.limit(5000),
    ]
  );

docs = [...data.documents];
while(true)
{
  const lastId = data.documents[data.documents.length - 1].$id;


  data = await databases.listDocuments(
    '[DATABASE_ID]',
    '[COLLECTION_ID]',
    [
        Query.limit(500),
        Query.cursorAfter(lastId),
    ]
  );

  if(docs.total === 0) {
    // No more.
    break;
  }

  docs = [...data.documents,...docs];
}

console.log(docs);
#

This way you'll have all of the documents.

icy pecan
#

Thanks that is right, but I mean that i use postman with query[] = limit(10000) and it return that ammount

#

it returns 10K rows, I count the json response objects. Also i put limit() (without any number) to provoke an intentional error, this message I received:
{
"message": "Query not valid: Invalid limit: Value must be a valid range between 0 and 9,223,372,036,854,775,808",
"code": 400,
"type": "general_argument_invalid",
"version": "1.3.4"
}

pine abyss
#

Limit must have a number in it ad a parameter

icy pecan
# pine abyss You can up to 5000

Yes I undestand that, but you said limit can have up to 5000, but I used more, and the error message says 9,223,372,036,854,775,808

#

I'm confused, I just want to be sure what is the limit of limit

pine abyss
#

It's okay
Can you share the request in which you got more than 5

icy pecan
#

The result is:

{
    "total": 5000,
    "documents": [
          ... 6000 objects
     ]

I dont think appwrite should limit developers to get a max number of records, 9,223,372,036,854,775,808 is good xD
the bug is in total it should be 6000 in this request

dusk jungle
#

It should always have a limit, although it should be customizable, but not having a limit server-sided supposes that someone with bad intentions could abuse of your appwrite instance (slowing it down and increasing costs)

If that's the limit and you managed to bypass it, that's a problem

#

@icy pecan Do you have rate limit enabled in your app

icy pecan
dusk jungle
#

By default rate limiting seems to be enabled

#

Also are you doing the queries client sided? Did you have set any API keys apart from project ID?

icy pecan
#

I have the deault docker-compose.yml

#

and .env

dusk jungle
#

I see

icy pecan
#

I'm calling the appwrite database api (Get Document List) with postman

icy pecan
dusk jungle
#

Are you specifying an API key in the request?

#

I mean in the headers

icy pecan
#

But when trying to get a list from a function (to make business logic or integration, like sync tables) sometimes fetch all table records is needed, for example, I will have a table with almost 10K records, so I will need to fetch them in the function, but for client sdk It is good the limit of 5000 (or even less)

icy pecan
#

both responses are the same

dusk jungle
#

And in both cases without an API key you're able to get unlimited documents?

icy pecan
#

yes

dusk jungle
#

🤔 @elder depot Could you take a look into this? Shouldn't that be limited client sided without an API key?

icy pecan
#

I mean when using apikey

elder depot
dusk jungle
#

Whaaat 😅

elder depot
#

but total is still capped at 5k

dusk jungle
#

But isn't that going to allow abuse?

elder depot
icy pecan
elder depot
dusk jungle
#

I think instead of removing, it should be customizable, like all other abuse limits, in my case just to reduce them. Meanwhile I will make a fork, since I don't want anyone getting 5k documents with a single request 😅

icy pecan
elder depot
icy pecan
#

oh, ok thanks

#

Are you agree that when calling "list documents " from functions there should not exist a limit or kills for slow queries? cause that is made by the developer so he must have care of what is he coding. On the contrary, the "client rest/sdk" requests should have the limit/kill

elder depot
icy pecan