#🔒 What is the smartest way to iterate over the lines of a text file while preserving line endings?

49 messages · Page 1 of 1 (latest)

static aurora
#

...without reading the whole file into memory at once.

I currently have this:

def get_file_stats(input_file: pathlib.Path):
    with input_file.open() as f:
        line_count = 0
        word_count = 0
        character_count = 0

        for line in f.readlines():
            line_count += 1
            word_count += len(line.split())
            character_count += len(line)

    return {"input_file": input_file, "byte_count": input_file.stat().st_size,
            "line_count": line_count, "word_count": word_count,
            "character_count": character_count}

and the character_count ends up being wrong because the line endings get stripped out by f.readlines(). What's the smartest way to fix this while accounting for the possibility of both \n and \r\n line endings?

wind heronBOT
#

@static aurora

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
spiral sleet
#

Just use for line in f:

river nymph
#

I mean \n and \r\n should be both only count as 1 character

crystal gyro
#

use newline='' to disable open from doing newline translations

static aurora
static aurora
river nymph
#

I don't know what wc is

crystal gyro
#

word count

#

command line utility

static aurora
river nymph
#

I don't know pithink

#

But for me doesn't make sense to count them seperate

vagrant needle
#

doesn't .readlines() read the whole file into memory at once anyway?

crystal gyro
#

yes

static aurora
#

Really? Damn.

crystal gyro
#

you can do for line in f

static aurora
#

Actually I already switched to that since it seemed to produce the same results anyway 😅

#

So does that not read the whole file into memory at once?

crystal gyro
#

nope, it will read untill it encounters a newlilne

static aurora
#

Awesome. Thank you very much.

limpid vapor
#

what about for line in file: pithink

polar sinew
#

A different way to do this would be to read the file into chunks of the same length, ignoring lines. The line count is indicated by the amount of '\n' characters, and you never run into a problem of a line being too long for memory.

static aurora
#

I think I'll try that too. Thanks for the tip.

polar sinew
#

I'm not too familiar with pathlib, but to start you can do this to read the file in uniform chunks of bytes:

with input_file.open('rb') as f:
    while True:
        content = f.read(5000)
        if not content: break```
vagrant needle
blazing rune
topaz quartz
#

and a chunk size to of 4096 or 8192 or similar might be desirable

polar sinew
#

bytes.isspace is appropriate to check that

rapid tide
#

how big is file, if very very large you can consider chunking into a threadpool

topaz quartz
#

for large files you can also get significantly better speed if you read in something like 8 MB chunks instead of for example 8 KB chunks

static aurora
#

Also all of this is giving me the dreadful feeling that I'm trying to be too clever for what is ultimately just a simple exercise, since I doubt any of this would work with multi-byte utf-8 graphemes anyway 😅

polar sinew
# static aurora Also all of this is giving me the dreadful feeling that I'm trying to be too cle...

Yes, we're discussing a ludicrously overengineered solution for what's supposed to be a pretty simple exercise.

Here's how you'd do accurate word counting with the fixed size buffer idea:
To get the first and last bytes, you use slicing. First: content[:1], last: content[-1:]
Here are some spoilered lines for the solution:
||Initialise prev_end_space to True before the loop||

||this_start_space = content[:1].isspace()||
||this_end_space = content[-1:].isspace()||
||skip = 1 if not prev_end_space and not this_start_space else 0||

||words_n += len(content.split()) - skip||
||prev_end_space = this_end_space||

polar sinew
static aurora
#

It looks like what I wrote should be equivalent to what you wrote, but I now get a lesser word count than wc -w does with the same test file.

def get_file_stats(input_file: pathlib.Path):
    with input_file.open() as f:
        line_count = 0
        word_count = 0
        character_count = 0

        previous_chunk_ended_in_whitespace = True
        while content := f.read(8192):
            line_count += content.count("\n")

            current_chunk_starts_with_whitespace = content[:1].isspace()
            word_count_in_chunk = len(content.split())
            if not (previous_chunk_ended_in_whitespace and current_chunk_starts_with_whitespace):
                word_count_in_chunk -= 1
            word_count += word_count_in_chunk
            previous_chunk_ended_in_whitespace = content[:-1].isspace()

            character_count += len(content)  # doesn't work yet

    return {"input_file": input_file, "byte_count": input_file.stat().st_size,
            "line_count": line_count, "word_count": word_count,
            "character_count": character_count}
topaz quartz
static aurora
topaz quartz
static aurora
topaz quartz
#

sneaky, they just want you do think that you need to sign in if you don't look too closely

static aurora
#

It's pretty unfriendly design for sure.

topaz quartz
wind heronBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.