🔒 Watching a directory for new file creation

[high sonnet]

Trying again - last thread did not get a response.

Hi, I'm working on a project that monitors log files for keywords (long story short).
This project requires watching for new files created in a directory. The log files are named by date in this format:

start_date = datetime.today().strftime('%Y%m%d')
main_run(start_date)```

I am using `sh` module to follow the files:
```py
# In "main_run" function:
tail_proc = tail("-f", "-n 0", path_1, _iter=False, _out=tail_callback, _bg=True)
tail_proc.wait()```
(I'm probably going to change the structure of the main starting function.)

The script is kept running by the loop, but that means I can't do anything else until I break it. How should I go about watching for a new file (while breaking the loop)?

[round nova]

@high sonnet

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[celest lotus]

Do you know about watchdog?

!pip watchdog

[round nova]

watchdog v4.0.1

Filesystem events monitoring

Released on <t:1716480748:D>.

[obsidian hedge]

or https://pypi.org/project/watchfiles/

[high sonnet]

crap, just noticed these

[high sonnet]

heard about it, is this the best way?

[high sonnet]

This one I haven't heard of. I will take a look shortly

also, misc question, how many modules can you import before things get inefficient? lol

this project has the most modules I've ever imported so far

okay so wait a minute - watchdog seems to be able to monitor when a file is open, modified, or closed. Not sure if I can see what was added... is it possible that I can replace my tail method with this? Should I?

I don't mind my tail method but curious

[obsidian hedge]

it's not a drag on efficiency, but memory footprint

[high sonnet]

huh, okay. Well, I have 13 modules imported

which seems like a lot

but I'm sure big projects have more

13 modules in one script

import time, os, logging, sys, re, subprocess, threading, json, asyncio, ipaddress, queue, sh
from sh import tail, iptables
from datetime import datetime, timedelta```

wait, why is asyncio imported... I'm not using it right now

also it looks like I may have tried using watchdog before? I've been on and off on writing this script for a while

#from watchdog.observers import Observer 
#from watchdog.events import LoggingEventHandler, PatternMatchingEventHandler

"""def on_created(event):
        print(event)

    def on_modified(event):
        if not event.is_directory:
            logging.info(event.src_path)
            #print(datetime.now() + " - " + event.src_path)
        #f = open(event.src_path)
        #print(f.read())
        #f.close()"""

    """evt_handler.on_created = on_created
    evt_handler.on_modified = on_modified
    observer = Observer()
    observer.schedule(evt_handler, path_1, False)
    observer.start()"""
    
    #print("observer started.")

    """try:
        while observer.is_alive():
            observer.join(1)
            
    except KeyboardInterrupt:
            observer.stop()"""```

by the looks of it, I might have to start learning how to use async. because I think observer holds things up too

or, use threading

I don't know which is best

[high sonnet]

bump - gonna be away for a sec

[round nova]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.