🔒 redirecting io doesn't print properly | Python | Page 1

fierce ravine Sep 4, 2024, 8:30 AM

#

output = io.StringIO()
sys.stdout = output
results = parse_trick_and_run(trick_data, request.args)
sys.stdout = sys.__stdout__
output.seek(0)
print_content = output.read()
return Response(print_content)

torn girderBOT Sep 4, 2024, 8:30 AM

#

fierce ravine ```python output = io.StringIO() sys.stdout = output results = parse_trick_and_r...

#

@fierce ravine

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

fierce ravine Sep 4, 2024, 8:31 AM

#

b'Filesystem at / is mounted as read-only.\nFilesystem at / is confirmed as read-only.\n'

#

output ^

#

I want it to print \n as a newline, not actually print the new line ascii symbol

lavish light Sep 4, 2024, 8:32 AM

#

fierce ravine ```python b'Filesystem at / is mounted as read-only.\nFilesystem at / is confirm...

you have to decode bytes into string to print it properly, to do that you need to know its format

#

format is usually utf8 or ascii

fierce ravine Sep 4, 2024, 8:33 AM

#

when i do type(print_content), it shows it as str though

#

so I can't decode

lavish light Sep 4, 2024, 8:33 AM

#

fierce ravine when i do `type(print_content)`, it shows it as str though

if it says b' at the start it's very much not a string, it's a bytes object that must be decoded first

fierce ravine Sep 4, 2024, 8:34 AM

#

lavish light Sep 4, 2024, 8:34 AM

#

wait... you have a bytes object stringified?
so you actually have "b'...'"?

fierce ravine Sep 4, 2024, 8:34 AM

#

lavish light Sep 4, 2024, 8:35 AM

#

lavish light wait... you have a bytes object stringified? so you actually have "b'...'"?

you need to find where a bytes object is stringified and either make it decode correctly or return as raw bytes correctly, this is similar to accidentally stringifying a list when you needed an actual list

#

you can double verify this is indeed what you have by doing:
print(repr(your_stuff))

this will print a stringified bytes as
"b'stuff'"
note surrounding quotes around the b'stuff', indicating str type

keen berry Sep 4, 2024, 8:37 AM

#

Any reason you're not using https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout ?

Python documentation

contextlib — Utilities for with-statement contexts

Source code: Lib/contextlib.py This module provides utilities for common tasks involving the with statement. For more information see also Context Manager Types and With Statement Context Managers....

#

Also you have to be careful when using it, looking at the code you're returning a Response object, is this flask? Mutating sys won't work correctly if you have multiple threads

fierce ravine Sep 4, 2024, 8:40 AM

#

i am using flask

#

only one thread

#

it doesn't matter if i use flask or not

#

i'm still getting 'Filesystem at / is mounted as read-only.\nFilesystem at / is confirmed as read-only.\n'

#

before i return the response

#

    output = io.StringIO()

    with contextlib.redirect_stdout(output):
        results = parse_trick_and_run(trick_data, request.args)

    print_content = output.getvalue()
    return Response(print_content)

#

this is my new code

keen berry Sep 4, 2024, 8:44 AM

#

fierce ravine i'm still getting `'Filesystem at / is mounted as read-only.\nFilesystem at / is...

Yeah this is correct

#

It just appears like \n when you inspect the string. In fact it is an actual newline

fierce ravine Sep 4, 2024, 8:45 AM

#

i want it to actually put it on a newline

#

instead of showing the \n character

keen berry Sep 4, 2024, 8:47 AM

#

fierce ravine i want it to actually put it on a newline

Btw you're supposed to use make_response not create a Response directly

fierce ravine Sep 4, 2024, 8:48 AM

#

thanks, but that won't fix my problem

#

the problem is in the code above

#

before the return

keen berry Sep 4, 2024, 8:50 AM

#

fierce ravine thanks, but that won't fix my problem

There's no problem. The data that gets sent is a real newline

#

It just looks like \n

fierce ravine Sep 4, 2024, 8:51 AM

#

but in my terminal, it doesn't actually show a new line

#

and i don't get why there is a b'

#

if it's not a byte string

keen berry Sep 4, 2024, 8:52 AM

#

Show your whole file?

fierce ravine Sep 4, 2024, 8:54 AM

#

it's a big project

keen berry Sep 4, 2024, 8:55 AM

#

Use !paste

fierce ravine Sep 4, 2024, 8:56 AM

#

!paste

#

https://paste.pythondiscord.com/KUHQ

keen berry Sep 4, 2024, 8:58 AM

#

And what did you print to show the b'...?

fierce ravine Sep 4, 2024, 8:58 AM

#

print_content

keen berry Sep 4, 2024, 8:59 AM

#

You commented it out, so it will only print <class 'str'> right now

#

Show the output from your program and what you ran?

fierce ravine Sep 4, 2024, 8:59 AM

#

yeah i was testing

#

huzi@huzi:~/Documents/houdini/api$ python3 run_trick.py 
Container read_only_filesystem already exists. Removing it.
Image 'read_only_filesystem' built successfully. ✓
Container 'read_only_filesystem' started successfully.
b'Filesystem at / is mounted as read-only.\nFilesystem at / is confirmed as read-only.\n'

huzi@huzi:~/Documents/houdini/api$

keen berry Sep 4, 2024, 9:00 AM

#

That's impossible you didn't print anything out

fierce ravine Sep 4, 2024, 9:00 AM

#

i just changed my code lol

keen berry Sep 4, 2024, 9:00 AM

#

Is your paste out of date?

keen berry Sep 4, 2024, 9:00 AM

#

fierce ravine i just changed my code lol

If you changed your code you'll need to paste it again

fierce ravine Sep 4, 2024, 9:00 AM

#

https://paste.pythondiscord.com/KBMA

#

here

keen berry Sep 4, 2024, 9:01 AM

#

show the new output?

fierce ravine Sep 4, 2024, 9:02 AM

#

it's there

#

i just sent it

fierce ravine Sep 4, 2024, 9:02 AM

#

fierce ravine ```python huzi@huzi:~/Documents/houdini/api$ python3 run_trick.py Container rea...

here

#

only look at the last line

#

the other output comes from another file

#

this project is pretty big

#

hundreds of files

keen berry Sep 4, 2024, 9:03 AM

#

So I think your problem is you actually have this line still there https://paste.pythondiscord.com/KUHQ#1L56-L56

#

Show the output of cat run_trick.py

#

Are you sure you saved your file?

fierce ravine Sep 4, 2024, 9:04 AM

#

you're looking at the wrong code

fierce ravine Sep 4, 2024, 9:04 AM

#

fierce ravine https://paste.pythondiscord.com/KBMA

look at this

#

new url

keen berry Sep 4, 2024, 9:04 AM

#

Yeah I'm looking at both

fierce ravine Sep 4, 2024, 9:05 AM

#

i don't have encode in the new file

#

and it's still doing that

keen berry Sep 4, 2024, 9:05 AM

#

Run the cat command and the python3 command in the same terminal and paste everything

fierce ravine Sep 4, 2024, 9:06 AM

#

huzi@huzi:~/Documents/houdini/api$ cat run_trick.py 
def run_trick(trick):
    import requests
    import json
    from houdini_config import VM_URL, get_value
    params = {
        'container_name': trick,
        'file': "85c8de88d28866bf0868090b3961162bf82392f690d9e4730910f4af7c6ab3ee.txt",
    }

    response = requests.get(f'{VM_URL}/run-trick/{trick}', params=params)
    data = response.text

    print(data)

def for_host_debug():
    import requests
    import json
    PORT = 49153
    VM_URL = f'http://127.0.0.1:{PORT}'
    trick = "read_only_filesystem" # this will be the container name too

    params = {
        'container_name': trick,
        'file': "85c8de88d28866bf0868090b3961162bf82392f690d9e4730910f4af7c6ab3ee.txt",
    }

    response = requests.get(f'{VM_URL}/run-trick/{trick}', params=params)
    data = str(response.text)
    
    for line in data.split('\n'):
        print(line)


for_host_debug()

#

this file is irrelevant

#

import os
import subprocess

def is_read_only(mount_point):
    """
    Check if the filesystem at the given mount point is read-only.

    Args:
        mount_point (str): The path to the mount point.

    Returns:
        bool: True if the filesystem is read-only, False otherwise.
    """
    try:
        # Use the `mount` command to check filesystem options
        output = subprocess.check_output(['mount'], text=True)
        for line in output.splitlines():
            if mount_point in line and 'ro,' in line:
                return True
        return False
    except subprocess.CalledProcessError as e:
        print(f"Error checking filesystem mount options: {e}")
        return False

def test_write_permission(test_file_path):
    """
    Attempt to write to a test file to check write permissions.

    Args:
        test_file_path (str): Path to the test file.

    Returns:
        bool: True if write operation is successful, False otherwise.
    """
    try:
        with open(test_file_path, 'w') as test_file:
            test_file.write("Testing write permissions.")
        # Cleanup
        os.remove(test_file_path)
        return True
    except IOError:
        return False

def main():
    mount_point = '/'  # Change this to the actual mount point
    test_file_path = os.path.join(mount_point, 'test_write_permission.tmp')

    # Check if the filesystem is read-only
    if is_read_only(mount_point):
        print(f"Filesystem at {mount_point} is mounted as read-only.")
    else:
        print(f"Filesystem at {mount_point} is not read-only.")

    # Test write permissions
    if test_write_permission(test_file_path):
        print(f"Write permissions are available on {mount_point}.")
    else:
        print(f"Filesystem at {mount_point} is confirmed as read-only.")

if __name__ == "__main__":
    main()

#

this is where I get the output from

#

second paste is read_only_filesystem.py

#

this file runs in a docker container

#

and i use the io shit to get it to the host

#

this is very confusing stuff to be sharing my files

keen berry Sep 4, 2024, 9:08 AM

#

I need you to show me the cat output and the python3 output in one paste

fierce ravine Sep 4, 2024, 9:08 AM

#

idk what the problem is with my approach

keen berry Sep 4, 2024, 9:10 AM

#

fierce ravine idk what the problem is with my approach

Can you run the commands and show the output

#

The output you're getting doesn't match the program you're running at all

#

Oh I see what's happening. There's a bug in parse_trick_and_run somewhere

#

It's calling print with a bytes object

#

@fierce ravine ^

fierce ravine Sep 4, 2024, 9:15 AM

#

do you want to see parse_trick_and_run

keen berry Sep 4, 2024, 9:16 AM

#

Yes

fierce ravine Sep 4, 2024, 9:16 AM

#

i mean it doesn't print anything

#

def parse_trick_and_run(trick_data, args):
    container_name = args.get('container_name')
    # delete_docker_image(container_name)

    # next three function calls are generic. They will never be different
    check_if_container_is_running(container_name)

    original_directory = os.getcwd()
    os.chdir(trick_data['trick'][0]['path'])
    build_docker_image(trick_data['dockerfile'][0]['path'], container_name)
    os.chdir(original_directory)


    if bool(trick_data['dependencies'][0]['server']):
        # print("true")
        if not dependency_check.check_server():
            print(f"HTTP not turned on host. {x_button} \nQuitting Trick")
            return 1


    # third param to run_docker_container will always be for network. Waiting to find out what fourth, ..., nth is.
    run_docker_container(
                            container_name, 
                            container_name, 
                            str(trick_data['docker_config'][0]['network_mode']), 
                            trick_data['docker_config'][1]['read_only'],
                            trick_data['docker_config'][2]['security_opt'],
                            trick_data['docker_config'][3]['pid_mode'],
                            trick_data['docker_config'][4]['cpu_shares'],
                            trick_data['docker_config'][5]['volumes'],
                            trick_data['docker_config'][6]['mem_limit']
                        )

keen berry Sep 4, 2024, 9:16 AM

#

Well why do you bother with redirect_stdout if it doesn't print anything?

fierce ravine Sep 4, 2024, 9:17 AM

#

the run_docker_container is running a file in a container, and I want it's output

#

so i do that

keen berry Sep 4, 2024, 9:18 AM

#

redirect_stdout only captures stuff that gets printed to sys.stdout eg python calls

#

Subprocess output is not captured

#

You need to use a pipe for that

#

Show that whole file

fierce ravine Sep 4, 2024, 9:18 AM

#

but then how am i able to get the output

#

from the docker container to the host

#

with my method

keen berry Sep 4, 2024, 9:19 AM

#

Show where the b'Filesystem at / ... gets printed

fierce ravine Sep 4, 2024, 9:20 AM

#

import os
import subprocess

def is_read_only(mount_point):
    """
    Check if the filesystem at the given mount point is read-only.

    Args:
        mount_point (str): The path to the mount point.

    Returns:
        bool: True if the filesystem is read-only, False otherwise.
    """
    try:
        # Use the `mount` command to check filesystem options
        output = subprocess.check_output(['mount'], text=True)
        for line in output.splitlines():
            if mount_point in line and 'ro,' in line:
                return True
        return False
    except subprocess.CalledProcessError as e:
        print(f"Error checking filesystem mount options: {e}")
        return False

def test_write_permission(test_file_path):
    """
    Attempt to write to a test file to check write permissions.

    Args:
        test_file_path (str): Path to the test file.

    Returns:
        bool: True if write operation is successful, False otherwise.
    """
    try:
        with open(test_file_path, 'w') as test_file:
            test_file.write("Testing write permissions.")
        # Cleanup
        os.remove(test_file_path)
        return True
    except IOError:
        return False

def main():
    mount_point = '/'  # Change this to the actual mount point
    test_file_path = os.path.join(mount_point, 'test_write_permission.tmp')

    # Check if the filesystem is read-only
    if is_read_only(mount_point):
        print(f"Filesystem at {mount_point} is mounted as read-only.\r")
    else:
        print(f"Filesystem at {mount_point} is not read-only.")

    # Test write permissions
    if test_write_permission(test_file_path):
        print(f"Write permissions are available on {mount_point}.")
    else:
        print(f"Filesystem at {mount_point} is confirmed as read-only.")

if __name__ == "__main__":
    main()

#

this is running in the container

keen berry Sep 4, 2024, 9:20 AM

#

Ah right ok. Show how you're capturing output from the container

#

Probably the whole file that contains run_docker_container

fierce ravine Sep 4, 2024, 9:21 AM

#

def run_docker_container(image_name, container_name, network_mode, read_only, security_opt, pid_mode, cpu_shares, volumes, mem_limit):
    try:

        container = client.containers.run(
            image_name, 
            name=container_name, 
            detach=True, 
            network_mode=network_mode, 
            read_only=read_only, 
            security_opt=security_opt, 
            pid_mode=pid_mode, 
            cpu_shares=cpu_shares,
            volumes=volumes,
            mem_limit=mem_limit
        )

        print(f"Container '{container_name}' started successfully.")

        output = container.attach(stdout=True, stream=True, logs=True)
    except docker.errors.APIError as e:
        print(f"Error running Docker container: {e}")

#

container.attach(stdout=True, stream=True, logs=True)

#

this is how I get the output

keen berry Sep 4, 2024, 9:22 AM

#

Show the whole file

fierce ravine Sep 4, 2024, 9:22 AM

#

ok sorry

#

https://paste.pythondiscord.com/XCXA

#

here

keen berry Sep 4, 2024, 9:23 AM

#

Why do you assign to output but not use it? https://paste.pythondiscord.com/XCXA#1L59-L59

fierce ravine Sep 4, 2024, 9:24 AM

#

ah you're right

#

i got it to work

#

        # Attach to the container's logs
        for line in container.attach(stdout=True, stderr=True, stream=True, logs=True):
            print(line.decode('utf-8'), end='')

#

with this

#

so i don't think i need the other io lines in that other file

#

hmm looks like i still need it

#

but anyways it works now

#

huzi@huzi:~/Documents/houdini/api$ python3 run_trick.py 
Container read_only_filesystem already exists. Removing it.
Image 'read_only_filesystem' built successfully. ✓
Container 'read_only_filesystem' started successfully.
Filesystem at / is mounted as read-only.
Filesystem at / is confirmed as read-only.

#

thanks man

keen berry Sep 4, 2024, 9:28 AM

#

You shouldn't decode the lines like that

fierce ravine Sep 4, 2024, 9:28 AM

#

then what should i do ?

keen berry Sep 4, 2024, 9:28 AM

#

You need to use codecs.getincrementaldecoder("utf8")

#

Eg

for chunk in  codecs.iterdecode(container.attach(...), "utf8"):
    sys.stdout.write(chunk)

#

Otherwise you can get a utf8 character split across chunks

#

Or if you only want to return the output use stream=False and return it. Then you don't need that redirect_stdout hack

fierce ravine Sep 4, 2024, 9:32 AM

#

thanks

#

ok i got it

torn girderBOT Sep 4, 2024, 10:32 AM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 redirecting io doesn't print properly