how would I make a portion of requests per minute from a list of urls using threadpoolexecutor?
#🔒 threadpoolexecutor request
flat tree
oak vortexBOT
how would I make a portion of requests per minute from a list of urls using thre...
@flat tree
Python help channel opened
orchid pendant
the per minute part is the hardest part, you'll have to figure something out yourself
flat tree
the per minute part is the hardest part, you'll have to figure something out you...
aw dang it
zinc flame
You could have one thread put that amount of URLs into a work queue per minute.
But I'm not sure how compatible this is with the ThreadPool API
flat tree
You could have one thread put that amount of URLs into a work queue per minute.
the per minute part is what I dont know
sturdy crown
just call .submit() N times per minute
let the executor figure out how to assign those to whatever thread it wants
flat tree
just call `.submit()` N times per minute
how would I make sure its one minute
sturdy crown
use time.sleep
flat tree
fair enough
sturdy crown
with executor(workers=x) as executor:
for url in urls:
executor.submit(request, url)
time.sleep(60 / requests_per_minute)
psuedocode, obviously
flat tree
alright ty 👍 , I'll give it a try
sturdy crown
np
flat tree
```py
with executor(workers=x) as executor:
for url in urls:
executor.subm...
wait, wouldnt this only do one a minute?
sturdy crown
if requests_per_minute is 6, you'd want to wait 10 seconds between requests
60 / 6 = 10
flat tree
but then I wouldnt need to do the executor
wait, I dont think I need it
would it just work if I did them one by one
sturdy crown
you could just use threading.Thread here, but that API is worse
sturdy crown
would it just work if I did them one by one
Well, then you need to factor in how long the request takes
zinc flame
would it just work if I did them one by one
Depends on how high your request limit is and how long the request takes, yeah
sturdy crown
i.e. if the request takes 5 seconds then you're limited to 12 requests per minute and cannot go higher
flat tree
rate limit is 100 per minute, im planning on doing 20 per minute to be safe as I have another hourly programme requesting from it too
all I need it to do is save that data in a database
sturdy crown
but if you don't care - here's how you do that:
for url in urls:
before = time.perf_counter()
request(url)
after = time.perf_counter()
request_duration = after - before
ideal_delay = 60 / requests_per_minute
remaining_delay = ideal_delay - request_duration
time.sleep(remaining_delay)
(note that it's possible for remaining_delay to be negative)
flat tree
i.e. if the request takes 5 seconds then you're limited to 12 requests per minut...
what does the threadpool executor do in this case?
sturdy crown
what does the threadpool executor do in this case?
It has multiple requests running at the same time
sturdy crown
it will START a request every N seconds, even if there are already multiple requests that are in-progress
sturdy crown
```py
with executor(workers=x) as executor:
for url in urls:
executor.subm...
This example can have multiple requests running in parallel
sturdy crown
but if you don't care - here's how you do that:
```py
for url in urls:
before ...
this cannot
sturdy crown
executor.submit(request, url) only STARTS a request, doesn't wait for it to finish
flat tree
`executor.submit(request, url)` only STARTS a request, doesn't wait for it to fi...
ah okay
I think that would be better
flat tree
```py
with executor(workers=x) as executor:
for url in urls:
executor.subm...
how would I save the data to a variable?
im still kinda new to threading
sturdy crown
you can eventually call ```py
future = executor.submit(request, url)
data = future.result()
but calling future.result() will wait for the request to complete (if it hasn't already)
flat tree
how would I save all of them and get the result after its finished?
sturdy crown
with ThreadPoolExecutor(X) as executor:
futures = []
for url in urls:
futures.append(executor.submit(request, url))
time.sleep(60 / requests_per_minute)
for future in futures:
print(future.result())
you could also do ```py
for url, future in zip(urls, futures):
print(url, future.result())
np
oak vortexBOT
Python help channel closed