🔒 Read files in dictionary that have .txt or .log and count the amount of lines with err* messages

[glass pond]

Iterating through a directory, storing the extensions of files as a key and the files themselves as their values in a dictionary.
Now I want to add another function called "scanfilesforerrors(filegroups)" where I want to take all keys with .txt or .log extensions and read their contents and print the amount of lines my program found that included an "err*" message.

Though I'd need help with that. This is what I currently have but it gives me an error message:

[alpine agate]

@glass pond

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[frail stratus]

Whats the error message?

[glass pond]

One sec- im trying to get the thing done above 😭 IT doesnt embed properly

from collections import defaultdict
from pathlib import Path

def validate_path():
    while True:
        path = Path(input("Enter the path: "))
        if not path.is_dir():
            print("Please enter a valid path.")
        else:
            return path

def group_items(path):
    file_groups = defaultdict(list)
    for item in path.iterdir():
        if not item.is_file():
            continue
        extension = item.suffix.casefold()
        file_groups[extension].append(item)
    return file_groups

def scan_files_for_errors(file_groups):
    for ext in file_groups.keys():
        if ext == ".txt" or ext == ".log":
            with open(file_groups[ext]) as file:
                for line_count in file:
                    if line_count.casefold() == "err*":
                        line_count += 1
    print(f"Number of errors found in all .txt and .log files: {line_count}")
    return file_groups

def print_groups(file_groups):
    for extension, files in sorted(file_groups.items()):
        print(f"{extension}: {len(files)}")
        for file in sorted(files):
            print(f"        {file.name}")

def main():
    path = validate_path()
    file_groups = group_items(path)
    scan_files_for_errors(file_groups)
    print_groups(file_groups)

if __name__ == "__main__":
    main()

ah now

great

[glass pond]

Error message:

dont mind the folder name, that was from ages ago

[frail stratus]

what does file_groups[ext] show you?

[silk trench]

saying your supplying a list..

[glass pond]

It shows me the values to each key

(i think)

[frail stratus]

its a list.

open() needs to open 1 file. Not a list of them.

[silk trench]

need a new loop for going thru that list

[glass pond]

Okay so wait

what im doing then is:

for files in file_groups[ext]:
  if files == ".txt" or files == ".log"
      open(files)

Or does this technically open multiple files again ?

No wait it does open multiple files

let me rewrite that

[steel abyss]

btw iterating over a dict iterates over their keys, so for ext in file_groups.keys(): can be shortened to for ext in file_groups:

[glass pond]

def scan_files_for_errors(file_groups):
    for ext in file_groups():
        if ext == ".txt" or ext == ".log":
            for file in file_groups[ext]:
                with open(file):
                    for line_count in file:
                        if line_count.casefold() == "err*":
                            line_count += 1
    print(f"Number of errors found in all .txt and .log files: {line_count}")
    return file_groups

So now im saying:

• iterate through my dictionary.
• if the extension equals txt or log, iterate through their values (which is a list) and open each file on by one.
• while you open each file one by one, i want you to take the line_count of each file and add +1 to linecount every time you see an "err*" message in there

confused / clueless about the print(...) because i have all my prints in the last function - though i had absolutely 0 idea how to get the print(...) of this functions down there.

probably has something to do with the "return" at the end?

What do i want to return though. The linecount ? A bit clueless here

[steel abyss]

Why did you add parentheses? for ext in file_groups():

file_groups is a dict, not a callable, yes?

[glass pond]

Yes i forgot to remove them from the old code where i had done .keys()

[steel abyss]

if there are no file groups, where does line_count get defined as it is used at the end of the function?

also you're using this variable to iterate over the file, yet you're still incrementing it separately?

[glass pond]

Line count gets defined in my for-loop where i say "for line_count in file:"

[steel abyss]

and what happens if there are no file groups?

there are so many bugs in this code...

[frail stratus]

line_count in that case is a string, not an integer

[steel abyss]

huh?

I don't see anywhere where it would be set to a string

[frail stratus]

its in the for loop

[steel abyss]

but if there are no groups to iterate over, it doesn't get set at all

even then you're trying to increment a string in the loop

[glass pond]

Can you go over them with me please?
I want to learn from this

[frail stratus]

Zen, just define a new variable line_count = 0 and use a separate variable when iterating lines in the file after opening it

[glass pond]

Alrighty

[steel abyss]

you understand that incrementing a string doesn't work?

!e

foo = "bar"
foo += 1

[alpine agate]

:x: Your 3.14 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 2, in <module>
003 |     foo += 1
004 | TypeError: can only concatenate str (not "int") to str

[glass pond]

Yes based on what agent q said the program doesnt know that its supposed to be an int starting at 0 so we can then count it up one by one through each iteration

[frail stratus]

Right. so define the line count before the for loop happens

[glass pond]

Yep done that, at the very beginning right after i defined the function

and changed the loop to "for lines in file:" and "if lines.casefold() .... "

[steel abyss]

so each line in the file is named lines? a bit misleading is it not

[frail stratus]

Does your log error messages show err* or err or error?

[glass pond]

Im a bit confused as to what you mean.
Im trying to iterate through the lines by calling each line one by one and thats why i do "for lines in file" - or am misunderstanding? (i prolly am)

[glass pond]

Yeah I just realized that too.
Im telling it to find messages "err*" right now and not err or error

[frail stratus]

We tend to be particular with what is singular or plural. If you say lines, we expect a list of strings.

[steel abyss]

well, when you iterate over the file, what do you suppose the variable lines contains?

[glass pond]

Ahhh

its not clean code and someone else might misunderstand

or be confused

[glass pond]

Mhm I understand, it does make sense

Id think that each line would be stored in a list of "lines"

and that im therefore iterating through this list

which in this case thats not the case as im iterating through each line itself and not in a list

Okay so:

def scan_files_for_errors(file_groups):
    line_count = 0
    for ext in file_groups:
        if ext == ".txt" or ext == ".log":
            for file in file_groups[ext]:
                with open(file) as content:
                    for line in content:
                        if line.casefold() == "err" or line.casefold() == "error":
                            line_count += 1
    print(f"Number of errors found in all .txt and .log files: {line_count}")
    return file_groups

Changed it to this.
Now things work and my program doesnt crash.
BUT it says that it couldnt find any lines containing an error message.

I made two .txt files that both contain the follow content:

ERROR ERROR
ERROR
ERR
APSODFJAPSFA
SDFAJSFASD
FAS
DFA
SDF
ASD
FA
SFDASDF
ERR
ASERASERERRR
ASERASFA
ERR
AESD
FA

So, something is still not going as it should go

what i changed:
when doing with open(file) i had to save it as something cuz in my loop before i would iterate through "file" i believe which was declared somewhere else

that made the program run at least

Okay with the help of google I made it work but its a bit confusing

def scan_files_for_errors(file_groups):
    line_count = 0
    for ext in file_groups:
        if ext == ".txt" or ext == ".log":
            for file in file_groups[ext]:
                with open(file) as content:
                        line_count = sum(
                            1 for line in content
                            if "err" or "error" in line.casefold())
    print(f"Number of errors found in all .txt and .log files: {line_count}")
    return file_groups

line_count = sum(
    1 for line in content
    if "err" or "error" in line.casefold())

This I dont really understand.
Why is there a 1 at the beginning?

[devout belfry]

So sum has something to add up

[frail stratus]

Do you know how list comprehensions work?

[glass pond]

I had my first touchpoint with those a couple weeks ago but it was quite a bit abstract for me so i decided to leave that.

So no, I dont know / remember how they work

[glass pond]

but I defined line_count = 0 at the top already, no? So it can start there

[frail stratus]

What do you think the following show?

x = [1 for y in range(3)]
print(x)

[glass pond]

OH @devout belfry you mean that while iterating through the content it adds 1 to the "sum counter" basically.
Its basically like a += 1, right?

Still confusing why i would put that at the very start in front of a "for" loop

[glass pond]

3

[frail stratus]

Nope. It shows [1,1,1]

[devout belfry]

Maybe you meant to aggregate sums for the total of all lines? If so then it should be line_count += sum(...

[frail stratus]

Its effectively the same as doing this:

x = []
for y in range(3):
  x.append(1)
print(x) # Also outputs [1,1,1]

[glass pond]

Ah because its a list, yes.

[frail stratus]

But instead of writing this, I kept it short with just a single line

Then, sum() just takes the iterable list of numbers and summs them up

!e

x = []
for y in range(3):
  x.append(1)
print(sum(x))

[alpine agate]

:white_check_mark: Your 3.14 eval job has completed with return code 0.

3

[glass pond]

No no, just the total amount of lines. YOu understood it correctly the first time

[glass pond]

so the 1 at the beginning is like an .append(1) for the list comprehension

just shortened

[devout belfry]

Then the print is misleading

[frail stratus]

thats right

[glass pond]

Ahaaaaaa

💡

wait wait let me explain it with my function

[glass pond]

line_count = sum(
    1 for line in content
    if "err" or "error" in line.casefold())

Okay so here, this is basically the same as:

line_count = []
for line in content
  if "err" or "error" in line.casefold()
  line_count.append(1)
print(sum(line_count))

Is that right?

[frail stratus]

second one is missing a : at the end of the if and the end of the for line, and the indent is incorrect for line_count.append(1). But other than that, yes, thats right

Not sure about that if condition.... seems off

[alpine agate]

The or-gotcha

When checking if something is equal to one thing or another, you might think that this is possible:

# Incorrect...
if favorite_fruit == 'grapefruit' or 'lemon':
    print("That's a weird favorite fruit to have.")

While this makes sense in English, it may not behave the way you would expect. In Python, you should have complete instructions on both sides of the logical operator.

So, if you want to check if something is equal to one thing or another, there are two common ways:

# Like this...
if favorite_fruit == 'grapefruit' or favorite_fruit == 'lemon':
    print("That's a weird favorite fruit to have.")

# ...or like this.
if favorite_fruit in ['grapefruit', 'lemon']:
    print("That's a weird favorite fruit to have.")

[glass pond]

Alright gotcha, yea sometimes (too many times) i forget about the :

[frail stratus]

Thats alright. It takes practice

[glass pond]

ahhh okay understood

i just have to be more clear with what im telling the program

[glass pond]

Okay okay but I do believe I understand the list comprehension above.
Its super super abstract but im glad you found a way to explain it to me clearly by showing an example

makes much more sense now

[frail stratus]

practice it a few times so it sticks

[glass pond]

Yep yep, its not the only way of "doing list comprehensions" if that makes sense.
THere are different ways also, but for now im happy i understood this one. Its really useful

Okay so now that this is done:
Is there a way for me to have my output be "nicer" ?
Or throw the print(...) into my print function?

because right now when i run the program the first line is "amount of errors found ... " and then it prints the dictionary.

Is there a way i can put the print to the very end and also print out which files were found that contained error messages ?

[alpine agate]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.