🔒 Python process hogging file IO resources when operating on large files in Windows?

[obtuse prairie]

Running this in Windows Server 2016 Datacenter.

I have this fairly simple bit of python code that aggregates files into combines files, looping through each file, line by line, and based on the file name it merges them into the appropriate merged file. Anywhere from a 5Gb to 200Gb.
But when running this script on a big dataset(150Gb+) it appears to hog resources and other versions of the script running on the same server.

For example, another script trying to merge just 2Gb of files will just hang until the job operating on 150Gb of files is done. Which doesn't make sense to me. Considering this server has 4 virtual processors.

Idk if this is a python issue or a windows issue or what. Example of the sort of script in pseudocode:

Merged_Name = "Example"
with open (Merged_Name+".txt", "w+") as outf:
  for i in os.listdir():
    if i.contains(Merged_Name) and i != "Merged_Name.txt":
      with open (i, "r") as inf:
        for line in inf: 
          outf.write(line)

[iron slate]

@obtuse prairie

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[winged spruce]

This script confuses me - to start with, won't one of the files it reads from be the outfile, resulting in it trying to write lines from the outfile back to it in an endless loop?

[obtuse prairie]

oh sorry. I wrote it from my phone. let me fix it

it has a check for that, but I didn't include it out of brevity

was sorta important info

[winged spruce]

Generally speaking though, this kind of loop should be okay. Can you elaborate on the "other versions of the script running on the same server" part? You have several processes running doing this? How are they launched, and do they share an outfile?

[obtuse prairie]

No, they don't share an outfile, the other versions would be operating on files in other directories and there should be no overlap of multiple scripts running on the same dir at the same time

/root
├─ site1_files
├─├─ "Examplefile20240101.txt" (50Gb)
├─├─ "Examplefile20240102.txt" (50Gb)
├─├─ "Examplefile20240103.txt" (50Gb)
├─ site2_files(1)
├─├─ "Examplefile20240101.txt" (100Mb)
├─├─ "Examplefile20240102.txt" (100Mb)
├─├─ "Examplefile202401013txt" (100Mb)

given that file system. say, I have two scripts, one that runes in /root/site1_files and one that runs in /root/site2_files

[drowsy forge]

Are you writing to multiple physical disks? Is this a multiprocessing system?

[obtuse prairie]

and say that the script for site1_files is running for an hour. it seems to prevent the script for site2_files from running even though it would normally finish in seconds

nope, same disk

[drowsy forge]

Can you clarify what you mean when you say "it appears to hog resources and other versions of the script running on the same server"? Do you mean the other scripts aren't working while this one is?

[obtuse prairie]

correct

the small script will start up while the large one is running, but end up doing nothing. it'll just hang until the large script is complete or killed

this kinda throws a wrench into our automation process because if we're merging a huge dir of files, we can't merge smaller dirs of files in a timely manner. So we schedule it for off hours for now, but that's not a solution

If the merge script was trying to use multiple cores then this would make more sense, but each script is single threaded, no multiprocessing.

[winged spruce]

Hmm. My only guess is that when the file is closed, all the changes need to be flushed to disk, and if the disk is very IO-starved this can take a while.

If this were on a unix system I'd suggest trying to mess with ionice (e.g. increasing the IO priority of scripts that already did all the writes and now just need to close the file), but I think that's not a thing on windows

[obtuse prairie]

Hmm, I don't explicitly flush the outfile at any point, but I thought that the end of the with open(file, 'r') as inf: block flushes the buffer

hmm, maybe not

yeah I guess it wouldn't. I'll try doing outf.flush() after each input file is finished reading/writing to the outf.

hmm, it looks like when the with open finishes and closes the file, that's when IO resources get released. I'm wondering now if just the individual size of the merged files is an issue, where the merged file is 50Gb in size, and it being open for so long...

[winged spruce]

Yup, it should, that's why it hangs there

[winged spruce]

That's reasonable, it should have less stuff getting buffered

[iron slate]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.