merry halo
I have a collection with over 8500 documents and I'm getting an error in the payload client when trying to view the list of documents from our production mongo that says:
MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
I've tried setting indexSortableFields to true on the payload.config. Does anyone have any ideas on what can be done here? We are on an M5 cluster tier with MongoDB.
sturdy bone
ok when you say payload client, do you mean the List view in the admin UI?
we need to enable the aggregation property allowDiskUse
I also assume that this is on a collection with versions enabled
what is happening is that we need to sort a large set of documents, and you have too many which is causing memory issues
i would suggest setting your versions / drafts config property maxPerDoc to a low number
merry halo
Yes, I meant the list view in the admin UI.
And yes, this is also on a collection with versions enabled. We only have 1 version for each doc at the moment but I'll try lowering the amount in the config to see if that helps.
Is the property of allowDiskUse something we can enable on our end or would this require a payload update?
Also with some further testing if I set maxPerDoc to 1 it still has the issue.
If I disable versions for this collection the documents do render in the list UI however.
sturdy bone
allowDiskUse needs to be set by Payload but you could directly edit your node_modules/payload/dist to see if that will fix your issue (i am pretty confident it will)
file to be changed is here:
and here is a screenshot of me editing the /dist version:
clearly i am feeling like a 🤠 right now
merry halo
Hmm i've just tried editing the file and restarting my local server but still having the same issue:
sturdy bone
hmm, that seems strange
same exact error?
looks like M5 clusters don't support allowDiskUse, which is very good to know
maybe we can figure out a smarter way around this
merry halo
Oh
ah i see:
I'm going to try bumping to M10 in the interim but this interesting and also good to know!
sturdy bone
ok - keep me posted
i am not sure about how to get around this otherwise
one other thing to note, allowDiskUse seems to be true by default on Mongo 6.0+
so you might not need to hijack your node_modules like a maniac (me) if you are using 6.0+
merry halo
We're on Mongo 5.0 at the moment. Will confirm in a couple minutes.
@sturdy bone
Okay! I can confirm upgrading to an m10 cluster works!
sturdy bone
beautiful - did you still have your node_modules fix?
or should we set this in code? PS - would be a great PR for you to make if you're up for it 👼
sturdy bone
ok awesome
sucky that the database has gotta be more expensive - i will keep this on the radar to see if it's possible to do this in a better way
merry halo
or should we set this in code? PS - would be a great PR for you to make if you'r...
Awesome thanks for the support! Yeah this will work fine for us in the interim but would be great for others if a similar issue didn't require the higher dedicated server.
merry halo
or should we set this in code? PS - would be a great PR for you to make if you'r...
Sure I could. Would it make sense to hard code allowDiskUse in the query or have this as an optional parameter somewhere? I guess if the m10 cluster seems to solve it at mongo 5.0 though maybe not quite necessary yet?
merry halo
