Paginated query missing values when filtered | Convex Community | Page 1

pale knot Nov 20, 2025, 12:51 AM

#

I have a contacts table which is infinitely scrolled via .paginate. There is a search bar users can search by name for.

Query:

export async function paginate(
  ctx: QueryCtx,
  args: {
    firmId: Id<"firms">;
    paginationOpts: PaginationOpts;
    searchTerm?: string;
  },
) {
  const { firmId, searchTerm } = args;
  const term = searchTerm?.toLowerCase().trim();

  return filter(
    ctx.db
      .query("contacts")
      .withIndex("firmId_deletedAt", (q) =>
        q.eq("firmId", firmId).eq("deletedAt", undefined),
      ),
    (contact) => {
      if (!term) return true;
      return contact.name.toLowerCase().includes(term);
    },
  )
    .order("desc")
    .paginate(args.paginationOpts);


// calling
export const paginate = queryWithAuth({
  args: {
    paginationOpts: paginationOptsValidator,
    searchTerm: v.optional(v.string()),
  },
  handler: async (ctx, args) => {
    return Contacts.paginate(ctx, { ...args, firmId: ctx.user.firmId });
  },
});

// Call in UI:
  const {
    results: contacts,
    status,
    loadMore,
  } = usePaginatedQuery(
    api.queries.contacts.paginate,
    { searchTerm: filter.searchTerm ?? undefined },
    { initialNumItems: 50 },
  );
}```

When i filter for names that are on the first page, theres 150 total entries, and i display 50 per page, it will successfully return a match. But if I search for the name of the last record in the table "Emma Thompson", the query returns no results. Why is this?

weak falconBOT Nov 20, 2025, 12:51 AM

#

Hi @pale knot. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

This happens because you’re doing the filtering in TypeScript after pagination has already limited which rows are read.

Your pipeline is:

ctx.db
  .query("contacts")
  .withIndex("firmId_deletedAt", q =>
    q.eq("firmId", firmId).eq("deletedAt", undefined),
  )
  // ↓ filter helper from convex-helpers
  |> filter((contact) => {
    if (!term) return true;
    return contact.name.toLowerCase().includes(term);
  })
  .order("desc")
  .paginate(paginationOpts);

paginate only ever sees the documents in the current index range; filter then excludes many of them. So if “Emma Thompson” is beyond the range you’ve read for this page, she will never be seen by the filter, and you get an empty page even though a matching document exists later in the table. This is exactly the pagination pitfall described for TypeScript filters: they can “cause small or empty pages if the filter excludes most documents,” because pagination happens before filtering. [Merging streams]

To get correct behavior you have three options (tradeoffs):

Filter in the database query (with .filter) before .paginate

return ctx.db
  .query("contacts")
  .withIndex("firmId_deletedAt", q =>
    q.eq("firmId", firmId).eq("deletedAt", undefined),
  )
  .filter(q =>
    term
      ? q.contains(q.lower(q.field("name")), term) // schematic example
      : true
  )
  .order("desc")
  .paginate(args.paginationOpts);

Using .filter on a paginated query ensures pages are full according to the filter, unlike filtering in code afterward. [Best practices; Paginated queries]

#

Read more rows and filter in TypeScript

Keep your current filter helper but increase how many underlying rows you scan, so that filtered results still fill the page. With stream(...).filterWith you can also set maximumRowsRead to avoid reading too much, though that’s not in your snippet. [Merging streams]
Use a search index

If the name search is important and data is large, define a searchIndex on name filtered by firmId (and maybe deletedAt) and use withSearchIndex(...).paginate(...) to have the filtering done by the index before pagination. [Discord: multiple-field search; Search empty-string behavior]

In short: your code only searches within whatever slice of the table the current page happens to read; documents past that slice (like the last record) are never considered. Move the filter into the Convex query (`.filter` or `withSearchIndex`) before `.paginate`, or change your approach to read more rows before filtering.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale knot Kapa.ai is still learning and improving, please let me know how I did by reacting below.

pale knot Nov 20, 2025, 12:56 AM

#

@weak falcon I'l try option one, but q.containts q.lower is not valid methods..

weak falconBOT Nov 20, 2025, 12:56 AM

#

Let me check the knowledge base for you again. One moment! :books:

#

You’re right ‒ q.contains / q.lower don’t exist. I used them as schematic examples; Convex’s built‑in .filter doesn’t support substring / case‑insensitive text search. [Filtering docs]

Given the actual APIs, here are the concrete options that do exist:

1. Keep using the `filter` helper, but accept the pagination behavior

The filter helper from convex-helpers is the intended way to do arbitrary TypeScript filtering that still works with .paginate. [Complex filters]

Your code is already using it correctly:

return filter(
  ctx.db
    .query("contacts")
    .withIndex("firmId_deletedAt", (q) =>
      q.eq("firmId", firmId).eq("deletedAt", undefined),
    ),
  (contact) => {
    if (!term) return true;
    return contact.name.toLowerCase().includes(term);
  },
)
  .order("desc")
  .paginate(args.paginationOpts);

The limitation is conceptual: pagination happens over the underlying index order, then filter drops non‑matching items, so pages can be sparse or empty if matches are rare. [Merging streams / filterWith]

This is expected behavior; there’s no way with current primitives to have “LIKE %term%” semantics and perfectly dense pages without reading ahead.

2. Use full‑text search for name instead of substring includes

If you can accept “word‑based” full text search semantics rather than raw substring .includes, you can:

Add a search index on contacts.name with firmId (and perhaps deletedAt) as filter fields:

#

contacts: defineTable({
  name: v.string(),
  firmId: v.id("firms"),
  deletedAt: v.optional(v.number()),
  // ...
}).searchIndex("search_name", {
  searchField: "name",
  filterFields: ["firmId", "deletedAt"],
})

Query it with pagination:

export const paginate = query({
  args: {
    paginationOpts: paginationOptsValidator,
    searchTerm: v.optional(v.string()),
    firmId: v.id("firms"),
  },
  handler: async (ctx, args) => {
    const { firmId, searchTerm, paginationOpts } = args;

    if (!searchTerm) {
      // no search: just index + paginate
      return ctx.db
        .query("contacts")
        .withIndex("firmId_deletedAt", (q) =>
          q.eq("firmId", firmId).eq("deletedAt", undefined),
        )
        .order("desc")
        .paginate(paginationOpts);
    }

    return ctx.db
      .query("contacts")
      .withSearchIndex("search_name", (q) =>
        q
          .search("name", searchTerm)
          .eq("firmId", firmId)
          .eq("deletedAt", undefined),
      )
      .order("desc")
      .paginate(paginationOpts);
  },
});

Here the filtering is done by the search index before pagination, so pages are “correct” under the full‑text semantics. [Search indexes; Running search queries]

Note: Full‑text search doesn’t do arbitrary mid‑word substring LIKE "%erm%" style matching; it tokenizes into words and supports prefix matches. [Search behavior discussion]

3. Trade correctness for performance by reading more rows

#

If you must keep substring .includes, your only lever is how many underlying rows you’re willing to scan per page. With stream(...).filterWith(...) you can control maximumRowsRead in paginate to avoid scanning the whole table, at the cost of sometimes returning fewer results than theoretically exist. [Merging streams / filterWith]

Because the docs don’t describe any built‑in “contains” operator for .filter, there’s no “correct” .filter‑only rewrite of your .includes search. The two supported patterns are:

Complex text logic in TypeScript via filter / filterWith (your current approach), accepting sparse/empty pages.
Index‑based text search via withSearchIndex when its semantics are acceptable.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale knot Kapa.ai is still learning and improving, please let me know how I did by reacting below.

pale knot Nov 20, 2025, 12:58 AM

#

@weak falcon i also would've thought with filter convex helper that it uses the index, filters the row, then applies pagination to that filtered set, rather than paginating then filtering?

weak falconBOT Nov 20, 2025, 12:58 AM

#

Let me check the knowledge base for you again. One moment! :books:

#

You’re correct to focus on that ordering; it’s subtle.

For the Convex DB .filter method, the behavior is:

“Using .filter on a paginated query (.paginate) has advantages over filtering in code. The paginated query will return the number of documents requested, including the .filter condition, so filtering in code afterwards can result in a smaller page or even an empty page.” [Best practices]

So for:

ctx.db.query("contacts")
  .withIndex("by_something")
  .filter(...)
  .paginate(paginationOpts);

Convex’s own .filter is applied before pagination: it keeps scanning documents until it has enough that pass the filter, then forms the page. That’s why it avoids the “empty pages” problem.

However, the filter helper from convex-helpers is different. It’s a TypeScript-level wrapper over an OrderedQuery:

filter(ctx.db.query("posts"),
  (post) => ...complex logic...
).paginate(opts);

The Stack article explicitly warns that TypeScript filters (including this helper) can cause “small or empty pages” because they’re not integrated into the underlying pagination planner:

“These filters can cause unwanted behavior with pagination: they can cause small or empty pages if the filter excludes most documents.” [Merging streams – Filtering]
“TypeScript filters can work with pagination. This can return small or empty pages to the client, which can slow down load times and cost database bandwidth as it traverses the entire table across many pages. But it always works!” [Complex filters – Escape hatch: pagination]

#

The docs don’t spell out the exact internal implementation of the helper, but they do make clear its pagination behavior matches “TypeScript filters with paginate”: you should expect pages that may be sparse/empty when matches are rare.

So:

DB .filter + .paginate: scans until it has numItems that pass the filter, then pages. ✅ “Filter then paginate.”
convex-helpers filter(...).paginate: conceptually paginates over the underlying index, then applies your TypeScript predicate, so pages can be missing items you “know” exist later, or be partially/fully empty. ⚠️ “Paginate then filter” in effect. [Merging streams – Filtering]

Because of that, your intuition about how it should work is reasonable, but the current helper doesn’t integrate with the internal pagination logic the way core `.filter` does—hence the behavior you’re seeing with “Emma Thompson”.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale knot Kapa.ai is still learning and improving, please let me know how I did by reacting below.

pale knot Nov 20, 2025, 1:00 AM

#

@weak falcon zzzzz, so if i want to do the sane query.filter().paginate() for the search term i cant because the filter callbacks (q) doesnt have contains. If i use a search index im limiting myself to only being able to search against a single field (name), but what if I want to also search on email in the future

#

@weak falcon zzzzz, so if i want to do the sane query.filter().paginate() for the search term i cant because the filter callbacks (q) doesnt have contains. If i use a search index im limiting myself to only being able to search against a single field (name), but what if I want to also search on email in the future

weak falconBOT Nov 20, 2025, 1:00 AM

#

Let me check the knowledge base for you again. One moment! :books:

#

You’ve hit the current edges of Convex’s built‑ins:

Database .filter does not support substring / contains style operators, only comparisons (eq, lt, etc.). [Complex filters]
A searchIndex can only have one searchField, plus any number of filterFields. [Search indexes]

So there isn’t a way to do:

.query("contacts")
.withIndex(...)
.filter(q => q.contains(q.field("name"), term) || q.contains(q.field("email"), term))
.paginate(...)

because q.contains simply doesn’t exist.

Given that, the patterns Convex folks recommend in situations like yours are:

1. Denormalize into a single searchable field, then use a search index

This is the official way to “search multiple columns” efficiently with indexes.

Add a computed / denormalized field like searchNameEmail that concatenates what you want to search:

contacts: defineTable({
  name: v.string(),
  email: v.string(),
  firmId: v.id("firms"),
  deletedAt: v.optional(v.number()),
  searchNameEmail: v.string(), // "name email" lowercased, etc.
}).searchIndex("search_contact", {
  searchField: "searchNameEmail",
  filterFields: ["firmId", "deletedAt"],
});

Maintain searchNameEmail via triggers or in your mutations (the “denormalizing a field” pattern). [Searching two columns]

Then query:

if (!term) {
  return ctx.db
    .query("contacts")
    .withIndex("firmId_deletedAt", q =>
      q.eq("firmId", firmId).eq("deletedAt", undefined),
    )
    .order("desc")
    .paginate(paginationOpts);
}

#

return ctx.db
.query("contacts")
.withSearchIndex("search_contact", q =>
q
.search("searchNameEmail", term)
.eq("firmId", firmId)
.eq("deletedAt", undefined),
)
.order("desc")
.paginate(paginationOpts);


This gives you:

- Index‑backed search
- Pagination that counts *after* filtering (because the search index is part of the query planner)
- Ability to include name, email, and anything else in that one field

This is exactly what staff suggest when people ask for “search two columns with pagination” or “multi‑field search”: concat into one field, index that field. [[Searching two columns](<https://discord.com/channels/1019350475847499849/1356037921332924436>); [Multi‑field search thread](<https://discord.com/channels/1019350475847499849/1385358143437017178>)]

---

### 2. Accept TypeScript filtering + sparse pages

If you want real substring `.includes` semantics and don’t want to maintain a denormalized field, you’re back to:

```ts
filter(
ctx.db
  .query("contacts")
  .withIndex("firmId_deletedAt", q =>
    q.eq("firmId", firmId).eq("deletedAt", undefined),
  ),
(c) => {
  if (!term) return true;
  const t = term.toLowerCase();
  return (
    c.name.toLowerCase().includes(t) ||
    c.email.toLowerCase().includes(t)
  );
},
).order("desc").paginate(paginationOpts);

This:

Uses the index to scope by firmId / deletedAt.
Does arbitrary multi‑field matching in TypeScript.
But pagination is over the underlying index, so pages may be small/empty if matches are rare. [Merging streams – Filtering; [Complex filters – pagination]

#

This is what staff recommend when you “need advanced filtering like multiple field searches, string array filtering” and can tolerate JS filtering overhead. [Advanced filtering thread]

3. Hybrid: search index for coarse match, TS filter for extras

You can also:

Use a search index on searchNameEmail to cheaply narrow to “likely matches”.
Then run a TypeScript predicate (e.g. stricter .includes, or additional fields) on that smaller result set before returning / paginating manually (e.g. via getPage or a custom cursor). [Complex filters; [Take control of pagination]

But this becomes custom pagination logic—you lose the simplicity of usePaginatedQuery.

So for your specific “I want .filter().paginate() semantics and search on name and email”:

There is no built‑in contains in q to do it purely with DB .filter.
The supported scalable pattern is to denormalize multiple fields into one search field and put a searchIndex on that. That’s how you keep pagination correct and still search multiple logical columns.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale knot Kapa.ai is still learning and improving, please let me know how I did by reacting below.

pale knot Nov 20, 2025, 1:04 AM

#

@weak falcon lol this is such a ballache, this is trivial in postgres or SQL. If i denormalise I now have a another point of failure, i now have to remember say I have search: v.string() which maps to search: ${name} ${email} ${phone} if i add a new tenet, I now have to probably doa migration to populate that field to have that new piece of data. If i use the filter() convex helper with my paginated table, users will get wack results where things that exist dont appear. And i cant filter cause its methods are restrictive

weak falconBOT Nov 20, 2025, 1:04 AM

#

Let me check the knowledge base for you again. One moment! :books:

#

You’re not missing anything; you’ve run into real current limitations, and the trade‑offs are exactly as painful as you describe.

From the Convex side, what you want is:

Multi‑field, case‑insensitive, substring search
That runs in the query planner (not in TypeScript)
And works cleanly with .paginate (dense, “correct” pages)

The docs and Discord threads confirm:

DB .filter(q => …) only has comparisons like eq, lt, etc., not string .contains / substring operations. [Complex filters]
searchIndex only supports a single searchField per index, plus equality filterFields. [Text search indexes]
When people ask for “search two columns”, staff explicitly recommend denormalizing into one field and indexing that, or doing JS filters. [Searching two columns; [Advanced filtering thread]
TypeScript/filter‑style predicates do work with pagination, but you get “small or empty pages” and may have to scan a lot of rows. [Complex filters – pagination; [Merging streams – filtering]

So the situation really is:

Correct, index‑backed pagination
→ Requires expressing your filter in terms of indexes / search indexes.
→ For “search name + email + phone”: recommended pattern is a denormalized field + searchIndex. [Searching two columns]

#

Simple code + flexible predicates
→ Use filter helper / TypeScript filtering on top of an indexed prefix (firmId, deletedAt).
→ Pagination can miss/under‑fill pages, because it’s effectively “paginate then filter”. [Merging streams – filtering]
Keep DB “pure” (no denormalized field)
→ There’s no documented way today to get what you want: multi‑field substring search integrated with the Convex paginator.

On the denormalization pain:

Staff explicitly point to “denormalizing a field” with triggers as the way to build these computed search fields. [Searching two columns; [Triggers / denormalizing]
They also acknowledge you have to backfill & maintain it yourself on schema changes. [Advanced filtering thread]

So unfortunately, your options really are:

Take on the denormalized search column (with a backfill + trigger/mutation discipline) to get sane, index‑backed, multi‑field search + pagination; or
Accept “wack” pagination behavior when using filter/TypeScript predicates; or
Scale out to a dedicated search system (Algolia/Elastic) if search is central enough, which Convex folks also suggest for more advanced text search. [Dashboard search thread]

#Paginated query missing values when filtered

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

1. Keep using the `filter` helper, but accept the pagination behavior

2. Use full‑text search for name instead of substring includes

3. Trade correctness for performance by reading more rows

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

Because of that, your intuition about how it should work is reasonable, but the current helper doesn’t integrate with the internal pagination logic the way core `.filter` does—hence the behavior you’re seeing with “Emma Thompson”.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

1. Denormalize into a single searchable field, then use a search index

3. Hybrid: search index for coarse match, TS filter for extras

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

I don’t have a nicer hidden API to offer from the docs: there just isn’t a built‑in multi‑field, substring, index‑backed filter operator today.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#Paginated query missing values when filtered

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

1. Keep using the filter helper, but accept the pagination behavior

2. Use full‑text search for name instead of substring includes

3. Trade correctness for performance by reading more rows

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

Because of that, your intuition about how it should work is reasonable, but the current helper doesn’t integrate with the internal pagination logic the way core .filter does—hence the behavior you’re seeing with “Emma Thompson”.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

1. Denormalize into a single searchable field, then use a search index

3. Hybrid: search index for coarse match, TS filter for extras

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

I don’t have a nicer hidden API to offer from the docs: there just isn’t a built‑in multi‑field, substring, index‑backed filter operator today.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

1. Keep using the `filter` helper, but accept the pagination behavior

Because of that, your intuition about how it should work is reasonable, but the current helper doesn’t integrate with the internal pagination logic the way core `.filter` does—hence the behavior you’re seeing with “Emma Thompson”.