🔒 Code review [bs4] [beautiful soup 4] [web scraping]

[noble sparrow]

https://paste.pythondiscord.com/CBIA
What i would like to know:
-any suggestion how to improve this code's efficiency
-how to speed the scraping / how to slow it down to avoid website banning my ip
-how to modularize(?) this code more so i can use it in other projects as well as other people can use it
-how to document this properly(inline documentation)

[wheat python]

@noble sparrow

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[snow tundra]

i can't really know that unless you run it on a web page, measure it, then run an equivalent script in JS or another language & compare tbh

as for documentation, you don't really need it bc the functions explain themselves for the most part

you label things clearly, so

[toxic oriole]

Look at docstrings
missing some type hints
Inconsistent name casing

For modularizing them, u could isolate all the functions in one file and import the ones you need to another file (main file or whatever).

To limit yourself from getting banned, you'll have to slow down your request. You can store a datetime object to variable when you made your last request and when you initiate a new request subtract the current datetime to the last datetime and see how much time has elapsed.
If you want a rate limit of 3 sec per request; if the difference between both the time object is only 2 seconds you can time.sleep for one second before sending out your second request which is

ratelimit = 3

ratelimit_stamp = datetime.timedelta(seconds=ratelimit) + datetime.datetime.now()
diff = last_req - datetime.datetime.now()

elapsed = (ratelimit - diff).seconds
if elapsed > 0:
    time.sleep(ratelimit - elapsed)

You'll have to tweak that a little with your code, just an example

[noble sparrow]

thank you for the review! i will try to make this better!

[toxic oriole]

although if you're planning on making your stuff async, you'll have to completely change how the ratelimit would work

[noble sparrow]

if i was to make it async , i will have to use different IPs?

[toxic oriole]

I think there are some libraries to handle specifically async ratelimit tho

[toxic oriole]

Uh not really

[noble sparrow]

what is the point of using async with one ip?

[toxic oriole]

Well you could do it with one ip but it'll be pretty slow

Cuz you'll have to handle the request for all threads

[noble sparrow]

i dont understand

[toxic oriole]

Basically if you're doing it async, you can initiate two requests at the same time, unlike in sync code everything works line-by-line so you wouldn't be able to send a second request without the first one being completed. But in async you'll have to somehow "queue" all request from all threads and send each of them one-by-one

Well there are different ways of handling ratelimits tho

One of them is the leaky-bucket method

https://en.m.wikipedia.org/wiki/Leaky_bucket

But either way making an async scraper is pretty non-practical

It's better to use their api of whatever service you're getting the data from

[snow tundra]

what about my thanks?

:[

[noble sparrow]

i wanted to ask u , how to measure it?

also i am not sure how to code exact same in JS

[snow tundra]

sorry my friend mentioned it but he's not around so i had to find it

[snow tundra]

sorry what are you using? i'm like multi taskijng so i thought you were using scrapy

ur using vanilla python?

is there a reason ur not using scrapy, though?

[wheat python]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.