elfin beacon
I’m looping over array of file urls (around 100) and passing each url to the below curl code. This curl code is downloading the files in 2 MB chunks to save the memory. However, I’m facing Out of memory error in Redhat environment but same code working perfectly fine on Mac M1 for the same number of files. On Mac it’s consuming maximum 40 MB of memory for the whole data. While on Redhat it’s consuming the data it has already downloaded. For example, if the total data is 1GB then it's holding 1GB of memory until the function is completely executed. I have tried to unset variables, ran gc_collect_cycles() and reduced chunks size but no difference.
Does anyone have any idea what might be the issue in this case?
granite ivy
stoic sinew