#🔒 pybaseball rate limiting

25 messages · Page 1 of 1 (latest)

livid coral
#

Hi! Not sure if I’m going to be able to get any help with this one, since it’s for a specific library, but I’m having some trouble with pybaseball (and in specific, being rate limited by it). I’m calling the function pybaseball.schedule_and_record() in order to get data to create a graph of team W/L data, and up until now I’ve been able to do this no problem, and generate as many charts as I desire.

Recently, I’ve expanded my code to also be able to graph divisions. To do this, I send a separate request for each team in the division. Again, I’ve been able to do this as much as I desire.

The problem arose when I tried to optimize the division graphing, since sending all 5 of those requests was taking upwards of 30 seconds to complete. I tried to speed this up by using multithreading to send all of those requests at once, which worked like a charm! However after generating about 5 or 6 graphs within the course of about a half hour, I was rate limited. (Unhelpfully, pybaseball did not tell me this, and instead spit out a value error).

My question now is: what do I do to avoid being rate limited? Do I have to run the requests in a single threaded fashion?

Also typing this out, I do realize that pybaseball does have the option to keep a cache, though if I rely on that, my data won’t be updated with new games as they are played. Should I be keeping and purging a cache at regular intervals? And if so, how would I go about keeping track of when to purge? I imagine I’d have to keep a timestamp in some sort of log file, maybe with shelf?

hollow daggerBOT
#

@livid coral

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
ruby crane
#

it sounds like whatever it's making a request to, has a rate limit, you need to figure out what the rate limit is, and then adjust accordingly

livid coral
#

Alright. Pybaseball does not have any documentation on this rate limiting (at least when it’s making queries to bbref and not retrosheet)

raw quail
hollow daggerBOT
#

pybaseball/datasources/bref.py line 11

"""```
raw quail
#

Which data source is the problem?

livid coral
#

It’s from schedule_and_record() so I’d imagine bref

#

Since its current season

raw quail
#

. I tried to speed this up by using multithreading to send all of those requests at once, which worked like a charm!

If this is traditional multithreading, maybe something is going wrong as you shouldn't be getting a speedup from bref since there is a singleton class managing the rate limiting of querying that api. You should only be able to send 1 request per 6 seconds, no matter what.

#

How are you doing your multithreading?

livid coral
#

Concurrent.futures.ThreadPoolExecutor()

# 
def fetch_team_records(team):
    data: pd.DataFrame = pybaseball.schedule_and_record(2024, team)
    return team, data
with concurrent.futures.ThreadPoolExecutor() as executor: 
    team_data = list(executor.map(fetch_team_records, divs[division].keys()))```
raw quail
#

I would make a wild guess that BrefSession.get isn't thread safe so you're blowing right past the api limit.

#

You'd need to get someone smarter than me to check this.

#

But if you are getting more than one response in a 6 second period, that's the problem I would guess.

livid coral
#

Alright. In any case though, is my idea of switching back to single-threading and using and purging pybaseball's cache() feature at set intervals a decent one? Sounds like it would take care of the rate limiting, since most of the time I'd just be pulling from cache

raw quail
#

yes, seems sensible

livid coral
#

Well that's good. I guess my only question left would then be how to make sure that things are purged at a regular interval. I'm assuming I'm gonna have to stick a unix timestamp somewhere and check against it when the file is run, but is there an accepted best way to store that timestamp? Would it be in shelf?

raw quail
#

Personally, I would write a json file that looks like {'last_purge': UNIXTIMESTAMP} because I would like it to be human readable.

#

Shelf is fine though.

livid coral
#

Alright. Now you mention it though, human readable is preferable for me as well so I'll probably go that route.

#

Thank you so much for your help!

hollow daggerBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.