#🔒 'ascii' codec can't decode byte 0xc3 in position 662565: ordinal not in range(128)

18 messages · Page 1 of 1 (latest)

digital ivy
#

Error that stumped me, on python 3.6.8, while reading data from .log.xz file:

'ascii' codec can't decode byte 0xc3 in position 662565: ordinal not in range(128)

Traceback (most recent call last):

...

data = file.read()

File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

Higher in the code where the file object is created:(Tried both)

file = lzma.open(path, "rt")

and also tried

file = lzma.open(path, "rt", encoding="UTF-8")

The files in question are text logs , 50MB each, compressed in XZ, (.log.xz)

Folder of these are zipped on Windows machine and uploaded to GIT.

On Build server, the python application is started as a part of a pipeline in corporate gitlab devops.

Python unzips the files and then processes file by file on the build server and reaches the error above.

Testing the same code on windows locally and deployed on a linux servers I see no issues.

Issue only happens on the build server agent, that I don't have access to.

Default encoding:

sys.getdefaultencoding() : utf-8

Tried many things, Any ideas would be very helpful. Thanks

manic kernelBOT
#

@digital ivy

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
lone bridge
digital ivy
# 
import lzma

def load_log_string(path : str, seek_to = None, limit_bytes = None):
    if seek_to == 0:
        seek_to = None
    if path.endswith("xz"):
        file = lzma.open(path, "rt", encoding="UTF-8")
    else:
        file = open(path, "r")

    with file as f:
        if seek_to is not None:
            f.seek(seek_to,0)
        if limit_bytes is None:
            data = f.read()
        else:
            data = f.read(limit_bytes)
        return data, f.tell()
lone bridge
#

Full file including imports

#

And show the full output of your program

digital ivy
#

Info prints
sys.getdefaultencoding() : utf-8
['11_p', '12_p', '13_p', '14_p', '15_p', '16_p', '17_p', '18_p']
'ascii' codec can't decode byte 0xc3 in position 31574037: ordinal not in range(128)
Traceback (most recent call last):
File "src/log_archive.py", line 446, in load_delta
data, new_cursor_position = load_log_string(self.path_to_live_file,self.cursor_position, limit_bytes =limit_bytes)
File "src/log_archive.py", line 38, in load_log_string
data = f.read()
File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]

#

And I just realized that I fixed it with using the encoding parameter in

file = lzma.open(path, "rt", encoding="UTF-8")
#

I was just looking on the old output.

#

New output is

#

Info prints
sys.getdefaultencoding() : utf-8
['11_p', '12_p', '13_p', '14_p', '15_p', '16_p', '17_p', '18_p']
LOGD-2025-01-07 14:44:03.090805 :DONE_STEP ProcTime:17.261,log span 2892.551,CDs:0, LOGFILE sipserver_11_p_extended-001.20240626_072520_256 from 07:25:20.256000 to 08:13:32.807000, B:50738413
LOGD-2025-01-07 14:44:20.682600 :DONE_STEP ProcTime:17.592,log span 4113.143,CDs:0, LOGFILE sipserver_12_p_extended-001.20240626_070307_280 from 07:03:07.280000 to 08:11:40.423000, B:50726007
LOGD-2025-01-07 14:44:38.781086 :DONE_STEP ProcTime:18.098,log span 32134.374,CDs:0, LOGFILE sipserver_13_p_extended-001.20240625_232616_962 from 23:26:16.962000 to 08:21:51.336000, B:50721123
LOGD-2025-01-07 14:44:57.603330 :DONE_STEP ProcTime:18.822,log span 3630.454,CDs:0, LOGFILE sipserver_14_p_extended-001.20240626_073132_759 from 07:31:32.759000 to 08:32:03.213000, B:50731097
LOGD-2025-01-07 14:45:16.044204 :DONE_STEP ProcTime:18.441,log span 1838.581,CDs:0, LOGFILE sipserver_15_p_extended-001.20240626_074916_614 from 07:49:16.614000 to 08:19:55.195000, B:50734162
LOGD-2025-01-07 14:45:34.505812 :DONE_STEP ProcTime:18.462,log span 6762.731,CDs:0, LOGFILE sipserver_16_p_extended-001.20240626_060948_571 from 06:09:48.571000 to 08:02:31.302000, B:50721788

#

I don't understand, why it wouldn't correctly use the UTF-8 on the linux build agent...

#

Anyway, sorry for wasting your time, and also thanks, because I would probably keep on staring on the old output for a while

lone bridge
#

Remember to use code blocks for tracebacks

digital ivy
#

Okay, will do next time, thanks

#

!close

manic kernelBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.