I have complex counters with multi properties tags, agents, sources and stuff. How would you recommend doing this?
#How to do advanced search in convex?
rich crown
solid lagoonBOT
Hi @rich crown. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Great question! For complex filtering with multiple properties (tags, agents, sources, etc.) in Convex, here's what's recommended:
1. Use Indexes as Your First Line of Defense
Always narrow down your dataset with indexes before any further filtering. For a field like userId or agentId, define an index and use .withIndex() to cut the data significantly before applying other filters. [Reading Data]
2. For Array Fields (like Tags) — Use a Join Table
Convex indexes don't support array containment queries directly. The recommended scalable approach is a join table:
// schema.ts
export default defineSchema({
leads: defineTable({
name: v.string(),
agentId: v.id("agents"),
source: v.string(),
tags: v.array(v.string()),
userId: v.id("users"),
}).index("by_user", ["userId"]),
lead_tags: defineTable({
tag: v.string(),
leadId: v.id("leads"),
}).index("by_tag", ["tag"]),
});
When inserting/updating a lead, sync the lead_tags table. Then query by tag efficiently:
const taggedLeads = await ctx.db
.query("lead_tags")
.withIndex("by_tag", (q) => q.eq("tag", args.tag))
.collect();
const leads = await Promise.all(
taggedLeads.map((t) => ctx.db.get(t.leadId))
);
3. For Text Search (name, email, etc.) — Use Search Indexes
Define a search index per searchable field with userId as a filter field:
leads: defineTable({ ... })
.searchIndex("search_name", { searchField: "name", filterFields: ["userId"] })
.searchIndex("search_email", { searchField: "email", filterFields: ["userId"] })
Then query:
const results = await ctx.db
.query("leads")
.withSearchIndex("search_name", (q) =>
q.search("name", args.searchTerm).eq("userId", ctx.user._id)
)
.take(20);
4. For Remaining Complex Filters — Use TypeScript Filtering
After narrowing with an index, apply arbitrary TypeScript logic. This has the same performance as Convex's built-in .filter() for unindexed fields:
import { filter } from "convex-helpers/server/filter";
const result = filter(
ctx.db.query("leads").withIndex("by_user", (q) => q.eq("userId", ctx.user._id)),
(lead) => {
const matchesSearch = !args.search ||
lead.name.toLowerCase().includes(args.search.toLowerCase()) ||
lead.email.toLowerCase().includes(args.search.toLowerCase());
const matchesTags = !args.tags ||
args.tags.some((tag) => lead.tags.includes(tag));
return matchesSearch && matchesTags;
}
).paginate(paginationOpts);
5. For Counts — Use the Aggregate Component
If you need filtered/unfiltered counts without re-fetching all documents, the Aggregate component is recommended. It tracks counts in a btree and supports subsets. [Reading Data]
Summary of recommendations:
| Need | Solution |
|---|---|
| Filter by userId/agentId/source | Index + .withIndex() |
| Filter by tags (array) | Join table |
| Text search (name, email) | Search index |
| Complex multi-field filter | TypeScript filter via convex-helpers |
| Counts | Aggregate component |
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
rich crown
@solid lagoon agents, tags and so on are not predefined, how would this be implemented with the sharded counter, are you sure that this is going to work?
solid lagoonBOT
Let me check the knowledge base for you again. One moment! :books:
Great follow-up question. Let me clarify what the knowledge sources actually say about the Sharded Counter and whether it fits your use case.
Sharded Counter — What It Actually Does
The Sharded Counter is a key-value store from string to number. You can use arbitrary string keys, so it can work with dynamic/non-predefined keys like agent IDs or tag names. [Sharded Counter]
For example, you could do:
// Increment count for a dynamic tag
await counter.inc(ctx, `tag:${tagName}`);
// Increment count for a dynamic agent
await counter.inc(ctx, `agent:${agentId}`);
// Read count for a specific tag
const count = await counter.count(ctx, `tag:${tagName}`);
This would work for simple counting of dynamic keys.
Important Caveats
However, the Sharded Counter has a significant limitation for your use case:
This relies on the assumption that you need to frequently modify the counter, but only need to read its value from a query, or infrequently in a mutation. If you read the count every time you modify it, you lose the sharding benefit. [Sharded Counter]
Also, the Sharded Counter only does counts — it doesn't support sums, ranges, percentiles, or filtering across multiple dimensions simultaneously.
What's Better for Your Case
If you need counts filtered by multiple dynamic properties (e.g., "count of items with tag X AND agent Y"), the knowledge sources recommend the Aggregate component instead:
If you need high efficiency counting, you can use the Aggregate component to track all the leads, and depending on your sort order, allows you to get overall count, count for various subsets, as well as percentiles & more. [Paginated Queries]
The Aggregate component supports prefix-based bounds on tuple keys, which is more suited to multi-dimensional dynamic filtering. [Aggregate]
Bottom Line
- Sharded Counter: Works with dynamic keys, but only for simple per-key counts with high write throughput. Not suited for multi-property filtered counts.
- Aggregate component: Better fit if you need counts across combinations of dynamic properties.
The knowledge sources don't provide a concrete example of using either component with fully dynamic multi-property combinations, so I'd recommend reviewing the Aggregate component docs directly for your specific schema design.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
rich crown
@solid lagoon i need a counter that works this way, how can i achieve it?
solid lagoonBOT
Let me check the knowledge base for you again. One moment! :books:
Based on the conversation so far, it seems like you want a counter that works with dynamic, non-predefined keys (like tag names, agent IDs, sources) and supports filtered counts across multiple properties.
Here's how to achieve this using the Aggregate component, which is the recommended approach for this use case: [Aggregate]
Setup
npm install @convex-dev/aggregate
Define the Aggregate with a Tuple Key
You can use a tuple key combining your dynamic properties (e.g., [agentId, tag]) so you can count by prefix:
import { TableAggregate } from "@convex-dev/aggregate";
import { components } from "./_generated/api";
import { DataModel } from "./_generated/dataModel";
const aggregateByAgent = new TableAggregate<{
Key: [string, string]; // e.g. [agentId, tag]
DataModel: DataModel;
TableName: "leads";
}>(components.aggregateByAgent, {
sortKey: (doc) => [doc.agentId, doc.tag], // dynamic values
});
Keep It in Sync on Every Write
// On insert
const id = await ctx.db.insert("leads", { agentId, tag, ... });
const doc = await ctx.db.get(id);
await aggregateByAgent.insert(ctx, doc!);
// On update
const oldDoc = await ctx.db.get(id);
await ctx.db.patch(id, { tag: newTag });
const newDoc = await ctx.db.get(id);
await aggregateByAgent.replace(ctx, oldDoc!, newDoc!);
// On delete
const oldDoc = await ctx.db.get(id);
await ctx.db.delete(id);
await aggregateByAgent.delete(ctx, oldDoc!);
Query Counts by Dynamic Prefix
// Count all leads for a specific agent (dynamic agentId)
const countForAgent = await aggregateByAgent.count(ctx, {
bounds: { prefix: [agentId] }
});
// Count leads for a specific agent + tag combination
const countForAgentAndTag = await aggregateByAgent.count(ctx, {
bounds: { prefix: [agentId, tag] }
});// Total count across everything
const totalCount = await aggregateByAgent.count(ctx);
[[Aggregate Grouping](<https://www.convex.dev/components/aggregate#what-are-aggregates-for>)]
## Important Notes
- **You are responsible for backfilling existing data** if you already have documents in the table before adding the aggregate. [[Aggregate Operations](<https://www.convex.dev/components/aggregate#operations>)]
- To avoid write conflicts between nearby keys, consider using **namespaces** if your partitions (e.g., per-game or per-user) are truly independent. [[Aggregate](<https://www.convex.dev/components/aggregate#what-are-aggregates-for>)]
- Every mutation that modifies the table **must** also update the aggregate, otherwise counts will drift out of sync.
This gives you O(log n) count lookups for any combination of your dynamic properties. [[Aggregate](<https://www.convex.dev/components/aggregate>)]
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------@solid lagoon wouldnt it be better to use algolia?
solid lagoonBOT
Let me check the knowledge base for you again. One moment! :books:
That's a fair consideration. The knowledge sources do mention Algolia as an alternative for search, but it depends on your specific needs.
When Algolia (or similar) Makes Sense
For complex search + filtering + sorting scenarios, some community members have found external search services like Algolia, Typesense, or Meilisearch to be a better fit than Convex's built-in search. One community member noted:
"On my e-commerce like use case with hierarchical and refinement list filters + search + aggregates, I'm getting 3 to 6 seconds to get a result and setup is way too complex to be maintainable. Resorted to use Meilisearch. Typesense or Algolia are other alternatives." [Pagination & Filtering]
Convex's own team also acknowledges this:
"Search in Convex is meant to serve simple use cases currently, the recommendation being to either roll something like [streams] or use a service like Algolia." [Text Search Thread]
When Staying in Convex Makes Sense
If your primary need is counting and aggregating across dynamic properties (agents, tags, sources) — which is what the conversation has been focused on — Algolia won't help with that. Algolia is a search service, not a counter/aggregation system. The Aggregate component remains the right tool for efficient filtered counts. [Aggregate]
Bottom Line
- For full-text search with complex multi-field filtering + sorting: Algolia/Typesense/Meilisearch are valid alternatives worth considering.
- For counting/aggregating across dynamic properties: The Aggregate component is still the recommended Convex-native approach, and Algolia doesn't replace this.
You could also use both — Algolia for search/filtering UX, and Convex's Aggregate component for counts.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
rich crown
i need a counter for For full-text search with complex multi-field filtering + sorting @solid lagoon
solid lagoonBOT
Let me check the knowledge base for you again. One moment! :books:
For counting results from full-text search with complex multi-field filtering, the knowledge sources are honest that this is a genuinely hard problem in Convex.
The Core Challenge
Convex's search indexes return results ranked by relevance, and the result set changes dynamically based on the query. There's no built-in way to get a count of search results without iterating through them. The Aggregate component works well for predefined groupings (e.g., count by agentId, tag), but it can't efficiently count the results of an arbitrary full-text search query, because:
- The Aggregate component tracks counts based on sort keys you define at write time [Aggregate]
- Full-text search results are dynamic and relevance-ranked — you can't precompute their count
What the Knowledge Sources Recommend
For counting filtered results where the filter combinations are user-generated and unpredictable:
"Right now I need to use
.collect()and count all the documents. Is there a more efficient way of doing this?"
→ "Use the aggregate component" — but this only works when you can define the grouping keys upfront. [Paginated Query Count]
For truly arbitrary filter combinations, the knowledge sources acknowledge there's no perfect solution:
"This means that there is no way to cache the filter result, because there could be an infinite amount of combinations." [Paginated Query Count]
Practical Options
- Skip the count: Only show "load more" pagination instead of "X results found" — avoids the problem entirely.
- Use
.collect()on a narrowed index: Use.withIndex()to narrow byuserId/agentIdfirst, then count in TypeScript. Acceptable if per-user datasets are small. - External search service (Algolia/Typesense): These services natively support faceted counts alongside search results, which is exactly what you're describing. This is the approach the knowledge sources suggest for complex search + filtering scenarios. [Pagination & Filtering]