meager breach
I have this function to call all the items in a table
handler: async (ctx) => {
return await ctx.db.query('portfolios').collect();
},
});```
I want to implement infinite scrolling, so rather than overloading the app only show more items when scrolling wanted to know if there is a way I can do this server side rather than client side.
heady salmon
Does paginated query not work for your use case?
meager breach
yes it does, just looked it up wasn't aware of it 😅
will try it, thank you!
heady salmon
ah - no problem!
realtime paginated query is one of the best parts of Convex, you'll love it
meager breach
Wow paginated query is amazing!
Wanted to know if you can help, with in the paginated query I want to sort and filter if possible.
This is what I have right now, only the sorting. Not sure if it's right. Wanted to also do filtering,
export const getPortfolios = query({
args: {
paginationOpts: paginationOptsValidator,
sortType: v.string(),
filterType: v.string(),
},
handler: async (ctx, args) => {
const { paginationOpts, sortType, filterType } = args;
// TODO:Filtering
// Sorting
if (sortType === 'recentlyAdded') {
return await ctx.db
.query('portfolios')
.filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
.order('desc')
.paginate(paginationOpts);
} else if (sortType === 'mostPopular') {
return await ctx.db
.query('portfolios')
.filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
.withIndex('by_favoritesCount')
.order('desc')
.paginate(paginationOpts);
} else if (sortType === 'alphabetical') {
return await ctx.db
.query('portfolios')
.filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
.withIndex('by_name')
.order('asc')
.paginate(paginationOpts);
} else {
return await ctx.db.query('portfolios').paginate(paginationOpts);
}
},
});
here is my schema for portfolios
portfolios: defineTable({
name: v.string(),
link: v.string(),
tags: v.optional(v.array(v.string())),
image: v.id('_storage'),
favoritesCount: v.optional(v.number()),
})
.index('by_favoritesCount', ['favoritesCount'])
.index('by_name', ['name'])
.index('by_tags', ['tags']),
...```
stoic lantern
Yes, you can filter, like you have it or in JS:
https://docs.convex.dev/database/pagination#transforming-results
meager breach
Ah yes, I thouhght I saw that somewhere. Thanks @stoic lantern !
heady salmon
I believe you can output anything in your pages, as long as it's deterministic based on the original pages received from the query.
meager breach
okay so here is my dilemma
args: { paginationOpts: paginationOptsValidator },
handler: async (ctx, args) => {
const results = await ctx.db
.query("messages")
.order("desc")
.paginate(args.paginationOpts);
return {
...results,
page: results.page.map((message) => ({
author: message.author.slice(0, 1),
body: message.body.toUpperCase(),
})),
};
},
});
for this example, results only returns a subset of the data because of pagination. Lets say I want to filter and sort from All the data in the messages table . If I do it through this example, it will only do i from that subset from results right?
how would i do it the way I need to
heady salmon
pondering
ah I see what you mean
you can sort/filter the complete result in the frontend, but things will shift around as the user scrolls through, which I'm guessing you don't want
filtering isn't the problem, it's sorting
meager breach
no it's the opposite
heady salmon
filtering shouldn't be impacted I don't think, let me know if that's not the case
oh?
meager breach
sorting is fine, it's the filtering that's giving the issue
if you go here, and select one of the options that isn't All it gets glitchy
heady salmon
I don't see a filtering problem in your code there
okay looking
Can you share how you're filtering?
Filtering shouldn't cause an issue. I'd also expect the kind of filtering you're doing there to be index driven
meager breach
yup, filtering on the front end
const { results, status, loadMore } = usePaginatedQuery(
api.portfolios.getPortfolios,
{
sortType: selectedSort || 'recentlyAdded',
filterType: selectedFilter || 'All',
},
{ initialNumItems: 6 },
);
const filteredData =
selectedFilter === 'All' || selectedFilter === null || !results
? results
: results.filter(
(portfolio) =>
portfolio.tags &&
portfolio.tags.map((tag) => `${tag}s`).includes(selectedFilter),
);
It almost seems less like an actual glitch and more like the animations are appearing glitchy when loading partial pages
meager breach
So I initially did the sort/filter the in the frontend approach
it worked for me on desktop
but mobile i was coming across this issue https://github.com/vercel/next.js/issues/34455
so to avoid it i wanted to do infinite scrolling
heady salmon
when you're not filtering you always load a full page so it seems smoother
meager breach
okay gotcha, but you notice when you filter to another tab the load spinner shows up so im not sure if that's filtering based of the query and paginating to look for more results because all the data isnt there
heady salmon
It's getting more pages more often because you're filtering on the frontend. This will be much smoother if you filter on the backend.
And you'll use less data
(user is loading way more than is being displayed currently, unless viewing the All tab)
meager breach
right
heady salmon
I would index on that filter field (assuming that filter is driven by a single field) and use that in the paginated query
doesn't address the mobile issue though
maybe smaller pages or lower weight images would help?
Next is a monster lol, a lot going on in there
You shouldn't have to change too much, you can keep filter state in the frontend as you currently are, and pass that into your paginated query
sage bobcat
if you're just trying to paginate, you could try checking this out https://github.com/webdevcody/thumbnail-critique/blob/main/convex/thumbnails.ts#L167
meager breach
@sage bobcat i wish I was trying to just paginate lol, I want to filter and sort as well from the backend if possible
heady salmon
Did my last comment make sense, anything still unclear on how to proceed
meager breach
was the last comment just to stick with filter the frontend since I need access to all the data?
graceful canyon
Maybe https://github.com/get-convex/fullstack-convex provides some examples of filtering? we used it to progress some of our stuff. Plus it has infinite scroll. Its just old now, id love it if convex would take a once through on some of these projects and see how they might be progressed with the latest improvements.
heady salmon
was the last comment just to stick with filter the frontend since I need access ...
No, filter on the backend using the selected filter as an index
meager breach
so the selected filter is an array of strings
dont think I can do q.contains, no sure what other way there is
im trying to filter based on tags, so if the tab is in tabs of that item I want to include it
heady salmon
Ahh it's tags based
meager breach
Yeah 😅
heady salmon
I know they added support for general js filtering in the query filter
might require a library though
checking
twilit notch
heady salmon
Thanks @twilit notch that's it
So if you filter in the query (as opposed to getting the page of data and then filtering in your query function), you can use doc.tags.includes(tag), for example, and get full pages that abide by the filter. As opposed to filtering after the fact and getting less than a full page, or maybe even an empty page.
If you use the filter helper from the article, that is
meager breach
Ah yes that article looks like it’s it, let me take a look and try it thanks @twilit notch @heady salmon
twilit notch
the filter helper in the article is equivalent to filtering after the fact, btw. so the pages might be empty
heady salmon
does that also apply to ctx.db.query().filter()?
If so I misunderstood this part
If there isn't a great way to get complete pages, I don't think backend filtering will improve your current situation
You could either do some extra work to make indexing on these tags possible (search is one possibility, would probably involve adding a field with a string containing all tags and searching on that), or dig into the animations that are causing the glitchy presentation so that it's still smooth when loading small pages.
twilit notch
ctx.db.query().filter().paginate() does apply the filter before getting the page, so you won't get empty pages. but it also has the potential to read way too many rows and hit a query limit.
graceful canyon
https://stack.convex.dev/complex-filters-in-convex
It would be amazing if we could build a comprehensive master list of all things filtering and index, add it to this maybe https://labs.convex.dev/convex-vs-prisma . Complex queries is one of the few things I'm avoiding. It ends up being a lot of cross referencing and i still don't know if there is a more efficient way. Or i forgot the way i did it a week ago and do it a less efficient way, admittedly ents adds a layer to this. Now i need to try filter(..
heady salmon
When you have to make it work, denormalization (sometimes extreme denormalization) is often the way