🔒 PII extraction script

[woven saddle]

@opaque minnow

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[crystal forum]

@opaque minnow I recovered your message

I'm writing a personal identifiable information extraction script based on regex matching. What the script does is you provide the directory where the to be scanned files are located, afterwards it prompts you for an output directory for an excel file in which the detected PII is stored. I noticed that the script takes a long time to scan for PII taking up several hours which is not viable, I tried to implement threading, but for some reason now the script doesn't even run properly. It doesn't throw any errors, but it's getting stuck somewhere. I kept thread locking in mind, so idk what the problem could be.

[opaque minnow]

thank you

[opaque minnow]

uuhm can't i upload a script or file? I'm trying to include the code as in a text file but it automatically deletes it.

[crystal forum]

!paste

[woven saddle]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[crystal forum]

use a paste link if you want to share code

you can't upload files here

[opaque minnow]

understandable

[opaque minnow]

thank you

[crystal forum]

Read the embed

[opaque minnow]

ye srry i'm a dumb ass sometimes
my script:

https://paste.pythondiscord.com/2XLQ

[signal drum]

Do you know exactly where it gets stuck? I assume it's in the while loop, that they're always alive?

@opaque minnow pinging in case you're tabbed out :p

[opaque minnow]

I think it's either stuck in the process_file or process_directory function. If you're referring to the 'while True' loops for checking the validity of the provided input, it runs after that. It's getting visually stuck in the terminal after the progress bar is being shown it stays at zero percent, i also tried with a print statement to check which file it's currently processing but that gets stuck to.

[signal drum]

Try printing something before incrementing the counter in process_file

Just to see if it gets there

Also, I'm not sure creating one thread for each file is actually gonna be good in terms of performance, you probably need to send them in chunks

There's quite a few things here that depend on synchronization and require locks, I'd recommend you make groups, and then synchronize those groups later, otherwise your parallelism won't really happen

[opaque minnow]

I put

print(f"Processing {file_path}", end="\r")

before the increment and it's printing the current file being processed in the console

i thought it could get stuck when multiple threads are writing to the excel workbood but idk

[woven saddle]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.