Local API Pagination Issue with Array Fields

[fallen field]

Hello guys !

Did anyone encounter an issue while using the Local API with pagination and array fields ?

I'm trying to query a paginated list of docs with a disponibilities array field and payload does not return all the docs of each pages.

When the pagination is set to true, payload retrieve only a part of the paginated docs.
Here is the field config :

{
  type: "array",
  name: "disponibilities",
  label: "Disponibilités",
  index: true,
  fields: [
    {
      type: "select",
      name: "status",
      label: "Statut",
      required: true,
      index: true,
      options: [
        {
          label: "Disponible",
          value: "available",
        },
        {
          label: "Réservation en cours",
          value: "booking_in_progress",
        },
        {
          label: "Non disponible",
          value: "not_available",
    },
    {
          label: "Nous consulter",
          value: "consult",
        },
      ],
    },
    {
      type: "date",
      name: "start_date",
      label: "Date de début",
      required: true,
      index: true,
    },
    {
      type: "date",
      name: "end_date",
      label: "Date de fin",
      required: true,
      index: true,
    },
  ],
}

Here's an exemple of LocalAPI query :

const result = await payload.find({
  collection: ITEMS_SLUG,
  where: {
    "destinations.slug": { contains: "espagne" },
    "disponibilities.end_date": { greater_than: new Date() },
    "disponibilities.status": { equals: "available" },
  },
  pagination: true,
  page: 1,
  limit: 18,
});

Here's the query result:

{
  /**
   * only some results of the 18 of the page
   */
  docs: [{}, {}, {}]
  limit: 18,
  page: 1,
  totalDocs: 140,
  totalPages: 8,
  ...
}

[hushed schooner]

Original message from @keen pilot - Moved from #general message

New Community-Help Thread Created!

Help is on the way! To mark it as solved, use the /solve command. In the meantime, here are some existing threads that may help you:

Documentation:

• Join Field - Query Options - Local API
• Sort - Local API
• Sort
• Pagination - Pagination controls - Disabling pagination within Local API
• Local API - Local options available

Community-Help:

• payload.find() - Retrieve only certain fields?

[dry stag]

Hey @keen pilot

So, you're saying only some docs are returned right?

But you are still getting proper docs back

[keen pilot]

Hey,

yes i get some proper docs of the query, the result is fine, but not all docs are here

and if i disable the pagination, i get all the docs

[dry stag]

All the docs, as in all of the docs in the collection or correctly all the docs that fit your query?

[keen pilot]

i mean the filtered docs that fits my query

if i take my exemple above, i should have 140 docs. Payload find 8 total pages and 140 total docs, that is correct. The difference is when i paginate the query, some docs are not retrieved on each pages

And important thing, this issue appears only when i filter on date field in my disponibilities array field

[dry stag]

Hmm, yeah that new Date() call in your query is suspicious to be honest

Can you try doing a new Date().toISOString() instead?

This will make it so we are not using your local time, but UTC instead

[keen pilot]

I tried with a .toISOString() call too, same issue

i can't find a solution, or anybody talking about this

[dry stag]

The thing is internally I think Payload stores the date in UTC anyway, so best not use your local time either way

What if you do it like this:

const now = new Date()
const todayUTC = new Date(Date.UTC(
  now.getUTCFullYear(),
  now.getUTCMonth(),
  now.getUTCDate()
))

[keen pilot]

Same issue.

I'm not sure the UTC / local date difference is a problem here.

the issue happens when i activate the pagination on the query and filter on a date field inside an array field

[dry stag]

If you keep pagination but remove the date field query, does it work? And does it work by keeping the date field but removing pagination true?

Also try this:

const result = await payload.find({
  collection: ITEMS_SLUG,
  where: {
    and: [
      "destinations.slug": { contains: "espagne" },
      "disponibilities.end_date": { greater_than: new Date() },
      "disponibilities.status": { equals: "available" },
    ]
  },
  pagination: true,
  page: 1,
  limit: 18,
});

Should be the same but just want to cover the bases here

[keen pilot]

I added the and: [].
With pagination: false i've got the 140 results.
With pagination: true and without the date field filter, i've got 141 total docs and all the docs per page

[dry stag]

So adding the and with pagination - is this the result you needed or still no?

[keen pilot]

i need to paginate my query and filter it by this date field

that's where my probleme is

Here's the result: { docs: 4, totalDocs: 140, totalPages: 8 }

[dry stag]

This is bizarre

[keen pilot]

i agree 😂

[dry stag]

Do you have access controls on the field/collection that might interfere?

Although you should still be getting ID's in that case

[keen pilot]

did you mean ID of docs ?

[dry stag]

Right

[keen pilot]

i can get them

but not all of them in the page

{ docs: [ 697, 692, 691, 690 ], totalDocs: 140, totalPages: 8 }

[dry stag]

Yeah, this is strange because you have a limit of 18 - you should be getting 18 docs (ID's or full docs) anyway

[keen pilot]

and this is page 2: { docs: [ 688, 686, 684, 683, 682 ], totalDocs: 140, totalPages: 8 }

[dry stag]

Can you try reproducing this in a minimal blank template and opening an issue with it as a reproduction?

Nothing really stands out to me as user-error to be honest

[keen pilot]

i'll try to do it, first time for me

Thanks the help anyway !

[dry stag]

Awesome, yeah go for it. My pleasure, hopefully it can be resolved for you here asap!