Mutliprocessing destroy saving data.

[fickle sorrel]

Hello, I would need some advice on how I could use multiprocessing in my use case.

Basically, my program grabs a list of keywords from an excel file. Then I want to start 50 tasks in parallel( multiprocessing) to process each of these keywords).
Then each process will save up the result in a csv file independantly. So like once the process finished, it willl save it.

This caused issues when I was using multiprocessing, because sometimes they would try to write the file simultaneously and it breaks the data.
Notice in the picture how after 800 successful data insertion, it just broke for no reason.
Thank you for any help

[regal grail]

How much are you reading/processing vs writing? Might make sense to write results to a queue and then dump that queue into a file (alternatively you could have another thread popping elements off the queue and writing them during processing)

[sturdy pond]

in these situations you're better off writing to independent files then appending them together if you can

so if you have 3 tasks, they write to result1, result2, result3

then you can run something else to append them together into a larger file and maybe even write the task number above each section. This will also let you see each result on their own

[regal grail]

The point of multithreading that program is to not have different files

The goal was to speed up an existing computation while keeping the entire result (and not looking at chunks)

Saving to intermediate files and then combining them works but adds lots of I/O, imo keeping the results in memory is fine (or streaming them to disk via a queue + writer thread as they come in) depending on the result size

[fickle sorrel]

my whole processing is based on different api calls and some data manipulation. I am doing 50 processes.

Now, I realize I could have just used asyncio for everything instead of multiprocessing

[fickle sorrel]

keeping it memory is dangerous as the task is very long (a 3-5 hours). I might because of some error or mistake by my part lose all progress

I would lose progress and money, because my operations are not free

[fickle sorrel]

that's the solution I implemented, but its going very heavy on ressources.

I am thinking of moving on from multiprocessing to something like async Semaphore

[regal grail]

Why does it take that long?

[regal grail]

API calls + async yes, async + data manipulation no

[sturdy pond]

he said multiprocessing
and no that's not really the entire point either, the point is to do the work in parallel, you can still have different files for output, depends on your use case.

[sturdy pond]

define heavy on resources

[regal grail]

if you're constantly writing to disk you might as well just stay on one thread kek

[sturdy pond]

so because you're writing output to multiple files for a 3-5 hour long task, you just want to remove the benefits of parallelizing the work entirely????

am i reading that right

[regal grail]

It doesn't have to be 3-5 hours long. You can mmap in the input file, stat it beforehand to get the size, break it up into chunks, process each chunk on a single core and push the results out to a queue in batches of X lines. Then you have another thread reading off the queue and dumping those entries into an output file

[sturdy pond]

the poster said the task is 3-5 hours long, hence the want to use multi-processing, which is easier to do with multiple output files instead of having to having to deal with concurrency of a single output file for the sake of single output file

[regal grail]

IO is expensive -> Don't do more IO than you need

[sturdy pond]

in this case he probably wants to do a lot of IO

[regal grail]

Doing more IO for the same result doesn't sound like the smartest thing to do

[sturdy pond]

"I don't want to lose my progress on a 3-5 hour task so i don't want to keep it in memory"
Having a single thread write the output file is not gonna decrease the amount of data you're sending, but is going to make it easier to make a mistake.

so you COULD have a single thread write to a file for no real gain at all, or you could just make it simple and just write to multiple files?

idk one seems like the more logical choice and the other is pre-optimization

just my 2 cents

[regal grail]

It is decreasing the amount by exactly half. No data is lost either since you're continuously writing to disk (also disk failure is more likely but whatever)

[sturdy pond]

if you say so ¯_(ツ)_/¯

[fickle sorrel]

I have a lot of data to process, I have rate limits so I can't go crazy.

[fickle sorrel]

no it has to. There is no way to do it faster. Since I have rate limits in some apis, I cant go crazy.

I used multiprocessing, because initally the program was very slow ( on one process). I would have took a full week or more. Now, with 50 processes, it runs in 3-5 hours.

Thing is multiprocessing takes a lot of RAM and CPU.

I have to do IO, because since the task is long, I dont want to risk losing progress for any reason. I pay money to process this data, so it would be unfortunate to lose everything because of an error or something I misconfigured. My program is still not 100% perfect for me to blindly trust it will end the task with no issues

For optimizing it, I though about using Asyncio.Semaphore

since it's mostly api calls. The only data manipulation I do is playing with strings and patterns.

So I could do async for all the tasks in one process.

well idk, I am kind of a newbie into concurrency programming.

[fickle sorrel]

right now, I implemented a temporary solution to write to multiple files( One for each process). Then I merge them using an independant code.

idk if it's the best solution or not. But my code definitely needs optimization.

as I plan to get better rate limits soon.

right now 50 processes take all my 16gb of ram and 100% of my cpu (I am on windows 10)

I am looking to getting up to 100 concurrency soon and I will need some help into designing my system correctly.

Thank you for any help

[regal grail]

Can you post the data source here? Tempted to optimize it

[fickle sorrel]

Like the original file that contains all my data or my concurrency code? If you are talking about the file, it's just a csv file containing 10K records

[regal grail]

Yeah basically your original file + the processing steps needed

[fickle sorrel]

Ill send the details once I get home.

[fickle sorrel]

Ok, so I kindly forgot (with school and exams I guess). But yeah, here are the requested files.

[primal moth]

@fickle sorrel

File Attachments Not Allowed

For safety reasons we do not allow files with certain file extensions.

mhmm0879 Said

An exemple of a source file to process

Code Formatting

You can share your code using triple backticks like this:
```
YOUR CODE
```

Large Portions of Code

For longer scripts use Hastebin or GitHub Gists and share the link here

Ignored these files due to them having disallowed file extensions

• Golden-Retriever_all-keywords_us_2024-03-30.xlsx

[fickle sorrel]

cant upload an excel?

anyway: https://wormhole.app/jzxyM#O2W6QXuC3-JdO7vmsxO-hA

I shared the excel

[primal moth]

@fickle sorrel

File Attachments Not Allowed

For safety reasons we do not allow files with certain file extensions.

Code Formatting

You can share your code using triple backticks like this:
```
YOUR CODE
```

Large Portions of Code

For longer scripts use Hastebin or GitHub Gists and share the link here

Ignored these files due to them having disallowed file extensions

• message.txt

[fickle sorrel]

https://hastebin.com/share/qivapalufu.python

this is the "brain" script that manages the content to then send it to workers who will process it( workers being processes)

[regal grail]

thanks, will take a look

[fickle sorrel]

Please tell me if anything is unclear.

[fickle sorrel]

@regal grail I dont want to bother, but any update?

[regal grail]

Got quite busy with work, back now

Not sure what that NAG3 library is, can't seem to find anything on it

[fickle sorrel]

no problem. I also get busy with school.

[fickle sorrel]

It's a python script that contains the work a worker should do

[regal grail]

Which you haven't shared yet?

Also the Excel file got deleted

[fickle sorrel]

I prefer to not share it here though.

[fickle sorrel]

ill upload it back

[regal grail]

No point then lol

If I can't even run your script there ain't no way that I'll be able to do actual performance work

I can do some guesstimations but the #1 rule for performance is measure before you optimize

[fickle sorrel]

https://wormhole.app/PPojr#4RJ1cJQRFaverR-w13fA9w

[fickle sorrel]

I mean, I could, but like its precious code you know.

can I just share it to you privately?

[regal grail]

That's fine as well lol

[fickle sorrel]

now, the code is ugly and not cleaned up

[regal grail]

Doesn't matter much to me

[fickle sorrel]

sent you in msg