#🔒 Continuation of a previous post

34 messages · Page 1 of 1 (latest)

hoary tinsel
#

Hi, my old thread got closed since I got nauseous before I could properly try for a solution so I'm reposting. If the same happens yet again I don't know what to do really.

The old thread can hopefully found here #1270333189172367394 message but I'll reiterate everything for clarity

Current code

def _runScript(self,scriptloc:pathlib.Path):
        self.logger.debug(f"Running script at location: {scriptloc}")
        processAction=[]
        self._assignRunCommand(scriptloc, processAction)
        processAction.append(str(scriptloc))
        try:
            process=subprocess.Popen(["./"+scriptloc], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
            stdout_lines = []
            stderr_lines = []

            while True:
                output=''
                self.logger.debug("inside loop, reading output")
                output=process.stdout.readline().decode('utf-8').strip()
                if output != '':
                    self.logger.info(f"stdout: {output}")
                    stdout_lines.append(output)
                errors=process.stderr.readline().decode('utf-8').strip()
                if errors != '':
                    self.logger.info(f"stderr: {errors}")
                    stderr_lines.append(errors)
                if output == '' and process.poll() is not None:
                    self.logger.debug("output is empty and process is done")
                    break
            return {
                'stdout': '\n'.join(stdout_lines),
                'stderr': '\n'.join(stderr_lines),
                'returncode': process.returncode
            }
        except Exception as e:
            self.logger.error("Error in running script! Error message is as follows\n"+str(e))
            return (None)

What is it supposed to do
The function should run an arbitrary bash script at scriptloc, and return its output to stdout and stderr as well as print any and all prints the script being run has in real time as the script is running.

What is it actually doing
The function returns correctly in every attempted case. The problem is the printing part. When the script is a short quick one there are no issues, but if it takes some amount of time - for instance my test script echos the numbers 0 through 9 over a period of ten seconds - the printing fails. Instead of seeing a number every second, the program prints 0, then seemingly halts, until after about ten seconds every remaining print is shown at once, like so (note the timestamps):

2024-08-07 11:12:41,733 - Task - delay - Running test "delay"
2024-08-07 11:12:41,738 - Task - delay - stdout: 0
2024-08-07 11:12:51,760 - Task - delay - stdout: 1
2024-08-07 11:12:51,760 - Task - delay - stdout: 2
2024-08-07 11:12:51,760 - Task - delay - stdout: 3
2024-08-07 11:12:51,760 - Task - delay - stdout: 4
2024-08-07 11:12:51,760 - Task - delay - stdout: 5
2024-08-07 11:12:51,760 - Task - delay - stdout: 6
2024-08-07 11:12:51,761 - Task - delay - stdout: 7
2024-08-07 11:12:51,761 - Task - delay - stdout: 8
2024-08-07 11:12:51,761 - Task - delay - stdout: 9
vital lavaBOT
#

@hoary tinsel

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
tulip flint
#

output=process.stdout.readline().decode('utf-8').strip()

this could block 'indefinitely'

#

should look carefully at the documentation for Popen @hoary tinsel

hoary tinsel
#

So what would you suggest?

tulip flint
#

an enemy of 'real time' processing can be buffering

#

you're looking for any and all arguments that could make reading the streams blocking or buffered

#

bufsize could be one ticket, by default its -1 which makes the buffer some power of 2 normally

#

for real time processing you probably want to turn off buffering so 0

#

or it also says 1 gives line by line buferring

#

as long as you also set text=True

#

0 means unbuffered (read and write are one system call and can return short)

1 means line buffered (only usable if text=True or universal_newlines=True)

any other positive value means use a buffer of approximately that size

negative bufsize (the default) means the system default of io.DEFAULT_BUFFER_SIZE will be used.

hoary tinsel
#

tried it with bufsize 1 and 0 as well as without bufsize argument, same result

native ether
#

If you're running a shell script which uses echo, the lines won't be buffered.

#

Ah. You're reading a stdout line. Then a stderr line. Then a stdout line, round and round.

#

So you read the first stdout line promptly.

#

Then you block reading ... nothing from stderr. So you never read from stdout again until the stderr closes (with no output).

#

Only after that do you looks at stdout again.

#

Dispatch a thread for each of stdout and stderr and read them in parallel.

#

Buffering isn't your problem.

hoary tinsel
#

oooooooooooookay it stops at...... I see!

#

since there's nothing in errors, it tries to read, has nothing, so it waits?

native ether
#

Until the script your calling closes stderr.

#

i.e. when it finishes.

hoary tinsel
#

right, that does make a lot of sense

native ether
#

For the moment, just comment out all the stderr stuff and see how the stdout stuff behaves.

hoary tinsel
#

yup, that's it... now how to split the stderr off... I don't remember how threading worked...

tulip flint
#

if you wanted to get into the weeds you can possibly even do it with asyncio

hoary tinsel
#

yeah

#

did it with just threading library

# 
def _runScript(self,scriptloc:pathlib.Path):
        self.logger.debug(f"Running script at location: {scriptloc}")
        processAction=[]
        self._assignRunCommand(scriptloc, processAction)
        processAction.append(str(scriptloc))
        try:
            process=subprocess.Popen(["./"+scriptloc], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True,bufsize=0)
            stdout_lines = []
            stderr_lines = []

            stdout_thread = threading.Thread(target=self._read_stream, args=(process.stdout, stdout_lines, 'stdout', process))
            stderr_thread = threading.Thread(target=self._read_stream, args=(process.stderr, stderr_lines, 'stderr', process))

            stdout_thread.start()
            stderr_thread.start()

            stdout_thread.join()
            stderr_thread.join()
            return {
                'stdout': '\n'.join(stdout_lines),
                'stderr': '\n'.join(stderr_lines),
                'returncode': process.returncode
            }
        except Exception as e:
            self.logger.error("Error in running script! Error message is as follows\n"+str(e))
            return (None)

def read_stream(stream, output_list, log_prefix):
        while True:
            line = stream.readline().decode('utf-8').strip()
            if line:
                self.logger.info(f"{log_prefix}: {line}")
                output_list.append(line)
            if not line and process.poll() is not None:
                break
vital lavaBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.