coral shoal
I have a large dataset and I need to split these up for processing in bulk. I want to try and mitigate memory or performance-related issues so would it be better to chunk the results after querying or do it as part of the query? We're talking ~65k records here.
i.e. Model::where()->get()->chunk(5000) vs Model::where(...)->chunk(5000, fn ($chunk) => //... )
heavy osprey
I'd chunk on query
coral shoal
Yeah after mulling over it a bit it does seem best to do it on query. There obviously will be quite a lot of queries but this should be fine on performance seeing as the results will be smaller, right?
royal cosmos
Model::where()->get()->chunk(5000) would first get all results, storing the result in memory, then you're chunking that collection. Model::where(...)->chunk() would execute separate queries for each chunk, only fetching the 5k results at a time
golden gate