🔒 Need to know why dividing my code into separate modules drastically improved performance

[shut sentinel]

I developed a Python web scraping script to scrape some data from a website. It was just one single script and a nightmare to extend or modify, so I refactored my code and divided the single script into different modules (main.py, config.py, scraper.py, parser.py, storage.py, utils.py) and the performance just improved drastically.
I have tested the old and new programs side by side 2 times (on the same laptop, with the same internet connection and run one immediately after the other).
The new modular program is taking 24s, while the old single script is taking 80s to do the same task. How on Earth could this happen? Please help me understand the root cause in detail and point me to any resources that could help me learn important concepts on how to improve my code's performance by myself. I was probably making some horrible newbie code performance mistakes in the old script, and I don't want to repeat them in the future.
Both the old and new programs are present in this folder

[drifting oar]

@shut sentinel

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[craggy jetty]

I'm not getting that behaviour. Running main.py vs old_top...

both varied between 5-7 seconds

[shut sentinel]

So it has in fact something to do with my environment then. I got excited for no reason at all.
Thank you for taking the time to run the code.

I'll check the running times again.

[shut sentinel]

Can you please tell me if there might be anything in my environment that is causing this behavior (the nearly 60s difference) other than pure coincidence (my network speed drastically reducing right as I run the old script)?

[craggy jetty]

connection speed will be my first suspect. There's a tool to help you time all function calls, it's call cProfile. You may want to look it up if you're interested

[shut sentinel]

Let me look it up right now.
And do you have any suggestions for books to learn in-depth about code performance and finding performance bottlenecks? (Not too advanced as I consider myself an intermediate level programmer)

[craggy jetty]

cProfile is a start imo. It's quite simple to use

import cProfile

and in your name == main you just do this to call your main() instead

    cProfile.run("main()", sort=1)```

and it gives you this detail timing of each of your process

this is from your old_top_transfer...

so majority is just from fetching data

1 area you can improve right away imo is when you loop to fetch data/images

switching those to async will speed up the process quite a lot

[shut sentinel]

In the old script?

[shut sentinel]

I can't make any sense of this right now 😅
I'll look it up on Google

[craggy jetty]

old_top_transfer_scraper.py I assume that's the one before refactoring?

see the column that says tottime

[shut sentinel]

Right. That's the one before refactoring.

[craggy jetty]

that's the main one you're interested in. The second column

the first 2 suggested they're data transfer stuff. taking 6.8 and 2.2 seconds

the rest takes just a fraction of that

[shut sentinel]

Ah! I see

[shut sentinel]

This problem is only for the images, right?
I get the HTMLs for each page in the get_all_page_soups() function on line 62. That is async. Do you see any problem in that?
I must have forgotten to do async requests for pictures as well.

[craggy jetty]

I can't tell that much but looking at the prints I would think so

[shut sentinel]

One last request.

Can you tell me a more efficient way to write the parse_row() function on line 100?
I am writing a huge number of try except blocks as I need to write the missing value for that key in the dictionary and print a custom error message.

That seems really inefficient.

And only one try except won't work as I need to write to a different field for each exception and print a different error message.

[shut sentinel]

OMG. Just realized that the old script fetches all 250 images synchronously (forgot to do async here) while the new one does this asynchronously. That must have been causing the behavior.
Thank you for pointing that out!

[craggy jetty]

oh that sounds huge

curious because the timing when I ran the 2 weren't that much different

[shut sentinel]

Now that's odd

Let me run the scripts again right now

[craggy jetty]

so strange lol. Yea the first time I ran, the lines about fetching images were very short

now it's longer. But it's like 21s vs 24s

[shut sentinel]

Only a 3s difference between the two scripts?

[craggy jetty]

yes

[shut sentinel]

This is the old script. 101s

Going to run the new script

[craggy jetty]

does cprofiler catch you other bottle necks?

[shut sentinel]

New script
46s

[shut sentinel]

Give me a sec

It printed a HUGE response. My terminal lost some of the response due to it being so large.

A tiny part of the response

[craggy jetty]

woa you cannot scroll to the top?

[shut sentinel]

Nope

[craggy jetty]

let me see if there's any argument to limit the output

[shut sentinel]

That's just a small part. There's literally hundreds of lines here.

Same case for the new script.

[shut sentinel]

It must be that my internet connection is much slower than yours, so it is making the impact of not using async requests much more severe as the program has to wait for a much longer time for my slow internet to fetch the image.

I am saying because I can visually tell by the terminal output that fetching the images from the old script is much slower than the new one.

[craggy jetty]

need to import pstats to limit the line

    with cProfile.Profile() as pr:
        main()

    stats = pstats.Stats(pr).sort_stats("tottime").print_stats(10)```

there's tottime and cumtime you may want to try

10 is the line limit

explanation I found >>

tottime is the total time spent in the function alone. cumtime is the total time spent in the function plus all functions that this function called.

[shut sentinel]

I assume the "stats = " line is meant to be written inside the context manager?

[craggy jetty]

you don't need that actually, sorry

[shut sentinel]

No problem at all.

[shut sentinel]

Can you please explain this?

My laptop just ran out of battery and the power is out so I can't actually run the program now.
So I figure that I should ask this and we just call it a day.

[drifting oar]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.