My code currently takes 15-25 seconds to run and I want to know how I can improve its speed.
https://paste.pythondiscord.com/ZK2A
The files I am trying to process are anywhere between 300,000 lines and a million lines. 30mb to 100mb.
#🔒 How can I improve this?
rancid tulip
weak wingBOT
My code currently takes 15-25 seconds to run and I want to know how I can improv...
@rancid tulip
Python help channel opened
fallow ivy
My code currently takes 15-25 seconds to run and I want to know how I can improv...
your annotations are wrong here
https://paste.pythondiscord.com/ZK2A#1L16-L29
(presumably they are wrong elsewhere as well)
autumn tundra
was also going to mention those shouldn't be async
rancid tulip
I actually did consider removing the async from those.
Also removed the unused imports (aiohttp and time)
short linden
I actually did consider removing the async from those.
are you familiar with async?
rancid tulip
are you familiar with async?
I don't know a great deal, i was mostly forced to use it due to discord bots.
It's basically a function that runs and when it has to wait for something it frees up the process for something else to run
You can run them concurrently if you want, however I don't in this code
short linden
You can run them concurrently if you want, however I don't in this code
You asked how I can speed up the code. Running the code concurrently could be your answer.
tardy onyx
Your database methods appear to be async, as well.
What kind of database is this running?
rancid tulip
You asked how I can speed up the code. Running the code concurrently could be yo...
That is one option. But I was more concerned with the process of everything. Running it concurrently might not necessarily speed it up if the process is what is slowing it down.
!paste
tardy onyx
Ah, sqlite.
rancid tulip
I have to use ASYNC as this will be part of a discord bot eventually.
short linden
That is one option. But I was more concerned with the process of everything. Run...
Reading from a file is an io bound task and async is meant for io bound tasks.
tardy onyx
I have to use ASYNC as this will be part of a discord bot eventually.
Well, no. Discord bot's methods are async, yours don't need to be for that reason.
autumn tundra
could you take a profile of your code? there's a lot of it, so it's hard to know what the slow part is a priori. what operating system are you on?
rancid tulip
Well, *no*. Discord bot's methods are async, yours don't need to be for that rea...
But it's generally recommended things be async wherever necessary to avoid blocking code.
rancid tulip
could you take a profile of your code? there's a lot of it, so it's hard to know...
I can upload my code to github if you want. I am running on windows 10
tardy onyx
That would generally be sound advice, if it's shown that the awaitd calls actually wait.
If they run CPU bound, they likely never give up control.
short linden
But it's generally recommended things be async wherever necessary to avoid block...
True. Thing is you aren't utilizing the full potential of async code.
autumn tundra
I can upload my code to github if you want. I am running on windows 10
instead, could you add some timers to your code to narrow down what the slow part is? it would look something like this ```py
top of file
import time
allusers_start = time.time()
allusers = await database.get_all_users()
print('allusers took', time.time() - allusers_start)
Well, one concern I see would just be here:
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
print("Getting player IDS!")
for line in file:
await player_ids(line)
allusers = await database.get_all_users()
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
print("Getting events!")
for line in file:
next_line = next(file, None) # Read the next line
You say these files could be 1million lines long. So you run through them twice.
And the second pass through you're skipping each other line. 🤔
rancid tulip
instead, could you add some timers to your code to narrow down what the slow par...
The slow part is when it processes the events XD
autumn tundra
i'd mostly do that to the chunks of your main() function
you mean the for loop that calls await player_ids(line)?
rancid tulip
Well, one concern I see would just be here:
```py
with open('./testfiles/log1.lo...
Because when I put it all in 1 loop, i get issues.
and the 2nd loop I am making sure the "next line" has the required text otherwise it's not worth looking at.
I could try and put them back onto 1 loop. That would definitely speed it up.
tardy onyx
Your first one gathers player IDs, but is it possible to work through the events without needing them all?
A single pass would be more efficient, yeah
rancid tulip
rancid tulip
Your first one gathers player IDs, but is it possible to work through the events...
I'll give this a shot now
autumn tundra
what part of the program is slow, though?
rancid tulip
I don't know the specific part. I'd have to do that timing thing suggested earlier to know specifically. But if I were to hazard a guess I'd say it's the part that checks for events lol
autumn tundra
definitely measure before optimizing!
once you know what part in main is slow, you can continue down into the leaf functions
it's very possible something like database.get_user_by_name could be the slow part, if there's no index on the column and you have a lot of users
but there's also a bunch of other parts that could be slow, so it's hard to say what's actually going wrong
if i had to guess though, you're probably db bound
rancid tulip
it's very possible something like `database.get_user_by_name` could be the slow ...
Uhh, there's only 80 users. I doubt that's slowing it down
I'm honestly thinking it could just be the sheer size of the file tbh
autumn tundra
no, that size of file shouldn't take that long to process. you should measure your code to see where it's slow
could you share your log file
autumn tundra
Uhh, there's only 80 users. I doubt that's slowing it down
that does make it less likely, but you sort of have an N+1 problem, so the overhead of talking to the database may cause your slowdown
rancid tulip
I can't coz the the smallest file I have is 30mb big lol
autumn tundra
could you share it using google drive/dropbox/github
rancid tulip
Nothing was a second or longer to process
rancid tulip
could you share it using google drive/dropbox/github
I can upload to git? 🤔 gimme a second to scrub any passwords then
autumn tundra
you mean no individual function call in your main took more than 1s? that would make sense. what part of main did it take the most time executing?
rancid tulip
i'll have to print out every event if I want to know exactly which ones took the longest.
autumn tundra
ah, you don't want to do it like that
rancid tulip
nope lol
autumn tundra
put the timers outside the for loops
how much time does it take to Getting player IDS! vs Getting events!
rancid tulip
i got this at the end of every function:
if (time.time() - start) >= 1:
print(f"Wound event took: {time.time() - start}")
and this is my main:
#This is just to test.
async def main():
start = time.time()
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
print("Getting player IDS!")
for line in file:
await player_ids(line)
next_line = next(file, None) # Read the next line
if "revived" in line:
await revive_event(line=line)
elif next_line and "Die()" in next_line:
await kill_event(line=line)
elif next_line and "Wound()" in next_line:
await wound_event(line=line)
elif next_line and "suicide" in next_line:
await suicide_event(line=line)
print(f"Main took: {time.time() - start} seconds!")
try something like this ```py
async def main():
player_ids_start = time.time()
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
print("Getting player IDS!")
for line in file:
await player_ids(line)
print('player ids took', time.time() - player_ids_start)
allusers_start = time.time()
allusers = await database.get_all_users()
print('allusers took', time.time() - allusers_start)
events_start = time.time()
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
print("Getting events!")
for line in file:
next_line = next(file, None) # Read the next line
if "revived" in line:
await revive_event(line=line, allusers=allusers)
elif next_line and "Die()" in next_line:
await kill_event(line=line, allusers=allusers)
elif next_line and "Wound()" in next_line:
await wound_event(line=line, allusers=allusers)
elif next_line and "suicide" in next_line:
await suicide_event(line=line, allusers=allusers)
print('events took', time.time() - events_start)
you want to start at the root of your tree, and then work your way into the leaf functions
how long does each section take
rancid tulip
Updated the repo to include log file
autumn tundra
and then when you're in your leaf functions, instead of having a print you would have a global value that you add to
autumn tundra
perhaps i'm missing something, but on my machine your code executes in 1 second
rancid tulip
did the database populate properly?
you should have 2 tables in the database 1 events and 1 users, and a shitload of events
autumn tundra
should i start with an empty database
rancid tulip
Na.
autumn tundra
ah, after running i have this
rancid tulip
So you telling me my pc is just shit?
rancid tulip
is a shitload of events only 1k hahaha
:( you be nice
this is 1 log file out of like 15 lol
autumn tundra
what are the specs of your machine?
rancid tulip
good question. Whats the command I run again?
oh
i cant send that
lemme take a screencap lol
halcyon sedge
just search "view ram info" in the search bar maybe
rancid tulip
halcyon sedge
that's not bad at all
autumn tundra
and you're running python test4.py to run your code?
rancid tulip
yes
autumn tundra
hmmmm
autumn tundra
can you run your main function with the entire if "revived" in line: block commented out
like all of the if statements
how long does that take
then can you add back the if statements 1 by 1
and say how long it takes after adding each one back
autumn tundra
first thought is maybe sqlite is slower on windows or something
but that seems unlikely. and definitely not 35x slower
rancid tulip
This is when I added them 1 at a time. so the first is none, then "if revived" then added die then added wound and then finally added suicide
autumn tundra
wow!
autumn tundra
ok can we drill down on the revive_event case
yeah that's the case on my machine too
rancid tulip
removed just wound and it suddenly went faster lol
autumn tundra
though it only takes .5s longer when i enable wound
could you comment out all the if stmts except for if "revived" in line?
on my machine to run that it takes 0.11s
but on yours it looks like 3 seconds
so we want to bisect the revive_event function
rancid tulip
Just the revive revent
autumn tundra
then how long does it take if you comment out this section in the revive_event function
rancid tulip
Maybe I don't commit after each entry then?
rancid tulip
It'd probably be both.
The my code is committing to the database after each entry.
So it's probably waiting for that to commit first.
autumn tundra
could you try measuring it?
rancid tulip
I'll change my code now to store everything in memory and then process it all into the database at the same time.
make it 1 commit instead of thousands
autumn tundra
but yeah, that was going to be my suggestion from up above
very common problem with dbs
but i'm curious if it's one or the other, or both that are making it slow
rancid tulip
Wouldnt be an issue if I could use motor instead of aiosqlite lol
autumn tundra
update vs insert
rancid tulip
update vs insert
It's the insert.
autumn tundra
what do these 4 runs represent
rancid tulip
Order:
add_event
first update user
second update user
both update users
rancid tulip
It's definitely anything to do with modifying the database
I'll just store everything into a dictionary and send them to the database at the end.
autumn tundra
inside of add_event, how long does it take if you remove the commit at the end of the function?
(since the context manager should handle that for you)
rancid tulip
Without the commit it wont save to the database. but I'll do that anyway
autumn tundra
the context manager handles that, no? but i don't think that's the slow thing just curious
rancid tulip
Without the commit, but it didnt save to the database
It's fine. I'll modify my code now. Gimme 10
We've identified where it's going slow
autumn tundra
similar issue described here, https://stackoverflow.com/questions/72514674/why-sqlite-queries-run-slower-in-windows-server-machine-compared-to-ubuntu-m, which is probably the same one you have, given that aiosqlite seems to be optimizing for a multithreaded use case (https://github.com/omnilib/aiosqlite/issues/97#issuecomment-748520700)
autumn tundra
btw are you still doing multiple inserts just not starting a transaction for each one, or are you doing one singular insert
12s is stil very surprising!
rancid tulip
The 12s actually comes from the fact that I have to query steam sometimes
autumn tundra
ah
rancid tulip
async def store_events(self, events) -> None:
# Define the SQL query with placeholders
query = '''
INSERT INTO events (playerone_steamid, playertwo_steamid, kill, revive, wound, suicide, date)
VALUES (:playerone_steamid, :playertwo_steamid, :kill, :revive, :wound, :suicide, :date)
'''
# Define the dictionary of values for placeholders
async with aiosqlite.connect(self.config['database']['test_location']) as db:
# Execute the SQL query with the dictionary of values
#print(f"Events before storage: {events}\n\n")
for event in events:
#print(f"\n\nevent: {event}\n\n")
await db.execute(query, event)
await db.commit()
ah yeah you could do this with a single insert and no for loop
rancid tulip
idk how lol
autumn tundra
hmm actually not sure how to do it with aiosqlite
no docs online lol
and i guess your perf is fine enough now
but should definitely be able to do this almost instantly
rancid tulip
does this next_line = next(file, None) # Read the next line actually skip a line? so im only processing half the file? or does it keep the current place in the loop and pull the next one for checking?
autumn tundra
yeah it skips a line
rancid tulip
AAHHH
rancid tulip
I cant have it skipping a line on me lol
autumn tundra
but i remember making note of that
rancid tulip
Processing only half the file is not ideal
autumn tundra
i just assumed that's how the log format looked
rancid tulip
especially since some events happen back to back lol
autumn tundra
so i guess you're back to 25s
rancid tulip
So how can I check the next line without actually skipping a line?
autumn tundra
turn it into a list and then check using indices
rancid tulip
The entire file?!
autumn tundra
it's already in memory
and even 100mb is actually not that big of a file, i promise!
if your concern is looping over it
if you really care then you can use a peekable iterator, but that's more work than necessary imo
because you're not bound by this in terms of perf
rancid tulip
So is there a fancy way to convert it to a list or do I have to first loop through each line and append them to a list and then loop again?
autumn tundra
list(lines)
rancid tulip
`list(lines)`
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
file_list = list(file)
```?that should work, yeah
rancid tulip
#This is just to test.
async def main():
start = time.time()
events = []
with open('./testfiles/log1.log', 'r', encoding='UTF-8') as file:
file_list = list(file)
count = 0
for line in file_list:
await player_ids(line)
try:
next_line = file_list[count] # Read the next line
except:
next_line = None
if "revived" in line:
event = await revive_event(line=line)
if event:
events.append(event)
elif next_line and "Die()" in next_line:
event = await kill_event(line=line)
if event:
events.append(event)
elif next_line and "Wound()" in next_line:
event = await wound_event(line=line)
if event:
events.append(event)
elif next_line and "suicide" in next_line:
event = await suicide_event(line=line)
if event:
events.append(event)
count += 1
``` Im gonna run this anyway, but opinion?wait
what
how the fuck
is this faster?!
autumn tundra
looks fine to me, though i might do next_line = file_list[count] if count < len(file_list) else None instead of try/catch
that seems surprising
rancid tulip
seems?!
it is surprising LOL
and remember that's 7 seconds mostly coz of the steam queries
autumn tundra
could be steam just responding faster
rancid tulip
2 without queries
And I havent fully optimized my code yet lol
This change has only been just the adding of events.
autumn tundra
awesome!
rancid tulip
I havent changed the update or adding user ids yet
im curious how fast it'd be on your pc now lol
Updated the repo
Fuck yeah. I love optimization when you go from like 35+ seconds to 2 seconds
autumn tundra
now under a second on my machine
rancid tulip
Dont spend that extra time all at once
;)
Now I got other issues lol
Explain something to me. I have changed nothing except for adding a few print lines.. and suddenly lol
nvm figured it out lol
Change one thing and shit goes to heck
weak wingBOT
Python help channel closed