🔒 Count the amount each word appears in a .txt file (efficient)

[lusty flame]

I don't want to just iterate over the words, but to get advice (maybe a nice library) that can iterate through a LONG text file (.txt).

Thanks in advance 😄

[hot kayak]

@lusty flame

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[toxic comet]

I don't want to just iterate over the words
Uh, why not?

[fluid jackal]

import pandas
with open("file.txt", 'r', encoding='utf-8') as f:
    text = f.read() # case sensitive, will count "word" and "Word" as 2 different words, add .lower() to avoid this
words = text.split()
word_series = pandas.Series(words)
word_counts = word_series.value_counts()

you could use pandas to do this efficiently, while there is iteration under the hood, there really isnt any other way

[toxic comet]

Not at all sure that this is actually faster than a Counter.

[fluid jackal]

i did run a test using timeit, it preformed slightly better, but it wasnt a big text file so it wasnt the best test

[weak sluice]

You would need to iterate the entire file's words. Even if you used a library, you would still need to load the data into the library.

If you need to consider all the data, you'd need to iterate over all of it at least once.

[lusty flame]

thanks all

I know I have to iterate, but if I could use pandas, it better (it uses C++ or something behind the seances).

[lusty flame]

can I do something like:

import pandas as pd
from pandas.core.frame import Dataframe
data:Dataframe = pd.read_csv("file.txt", sep="[ \n]")

?

[halcyon dock]

I think Python uses C behind the scenes as well.

[toxic comet]

not in a sense that lets you avoid iteration on the python side, though.

[hot kayak]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.