#Rate Limit is surprisingly aggressive

40 messages · Page 1 of 1 (latest)

hearty ridge
#

I'm seeing surprising behavior with the rate limiter.

I have a script (https://github.com/NickHeiner/jscodemod/blob/nth/ai-batch-no-esm/scripts/openai.bash) that repeatedly calls the API with a 1 second sleep. After 8 calls over 12 seconds, I get an error.

Request 1 at Tue Jan 10 10:48:58 EST 2023
...
Request 8 at Tue Jan 10 10:49:10 EST 2023

{
  “error”: {
    “message”: “Rate limit reached for default-code-davinci-002 in organization org-uwqDMdROFGWuoV0xmUUaQI8j on requests per min. Limit: 20.000000 / min. Current: 30.000000 / min. Contact support@openai.com if you continue to have issues.“,
    “type”: “requests”,
    “param”: null,
    “code”: null
  }
}

The message is a bit surprising to me, because I haven't exceeded 20 calls in a single minute.

Of course, if I were to continue at a rate of "8 calls per 12 seconds" for a full minute, I would make 40 calls. So maybe that's what the API is reacting to? So maybe to abide by the rate limit, I need to stay below a steady-state rate which would result in hitting the rate limit if carried out for a minute?

Update: I tried modifying the "wait time between calls" from 1 to 3 seconds. This results in me getting the same rate limit error as above after 13 calls over 43 seconds. This undermines my theory above, because that's a rate of 18 calls/minute, which seems like is should be fine.

Update 2: I also saw an error with a delay of 4 seconds, resulting in 7 calls over 28 seconds, for a per-minute rate of 15.

I see this behavior with the token count rate limit as well.

GitHub
jscodemod/openai.bash at nth/ai-batch-no-esm · NickHeiner/jscodemod

Codemod runner. Contribute to NickHeiner/jscodemod development by creating an account on GitHub.

autumn ingot
#

Seeing this as well with rates as low as 10/min

wet bison
#

I am seeing the same issue on token limit. I measured calls from our side, not getting near the same rate as stated in the error msg. I sent an email to support. Did you guys get any resolutions?

hardy surge
#

I have exact same problem, hitting rate limit all the time, with very few calls. Pay-as-you-go account.

wet imp
#

+1'ing this; i dont have more than <80 requests a day and i can't even make two requests within a minute anymore

#

im currently lucky to get more than one request every five minutes

obtuse mauve
#

+1

#

This appears just recently

#

The response speed seems to be slower too

tame crag
#

@wet imp noticed the same

wet imp
#

@patent hedge appears to have the same issue

#

must just be system issues right now

#

i usually sleep at this hour but sleeping sometimes is hard heh

patent hedge
#

hmm thread goes back to a month ago? it's been fine most days

tame crag
#

I see this is widespread, on some open source projects see the same. It's issue on Ada and Davinci models. Hope it will be fixed, as it's making trouble for production apps. Maybe time to think about some load-balancing scenarious, but it will not help if we don't know exact limits on request rates\

sleek valve
#

Same here today for the past few hours. The rate limit for me is not specific:

openai.error.RateLimitError: The server had an error while processing your request. Sorry about that!

Seems the services are down?

tame crag
#

statusText: 'Too Many Requests', whatever I do

wet imp
patent hedge
#

I'm just going to figure it's scaling issues. And glad I'm not alone, don't think I did anything too wrong on my end

#

but it's kind of concerning, if I want to start a more serious project and can't get more than 2 requests in per minute

soft escarp
#

I get the same message and as a paying member, this is quite annoying.

soft escarp
patent hedge
#

Pay-as-you-go, you'd think they'd want a lot of requests

rich solstice
#

Hey all! Can you provide examples of the requests that you're making? I personally haven't run into this issue but here are some reasons this may be happening

Is API usage subject to any rate limits?
thorny pebble
fast oak
azure void
#

I haven't been running into actual rate limiting errors, but for the last 2 days i've noticed a steep decline in my fine-tuned Ada model speed on return

wet imp
# rich solstice Hey all! Can you provide examples of the requests that you're making? I personal...

as Chaddeus mentioned, per my specific usage, it would not be relevant to the requests per minute or the token ratelimit. I'm a pay-as-you-go user, however my usage is for a Discord chatbot which doesn't exceed more than three requests a minute generally, and each request as tracked never exceed 2,500 tokens per request; i use text-davinci-003 and i've found that removing some of my input prompt (such as chat history within my prompt) to be under 1,500 tokens eliminates the issue - my theory is that the text-davinci-003 model endpoint fails to actually accept larger token usage within it's request, despite it's limit being documented as 4,000: https://platform.openai.com/docs/models/overview

OpenAI API

An API for accessing new AI models developed by OpenAI

#

my usage to elaborate more is just for a chat-bot for a small friends server

thorny pebble
#

Sigh. This is an issue again -- and impacting my users.

I have already implemented quite a bit of delays/back off... but this is a persistent rate limit hit, no way my app would be doing that.

wet imp
thorny pebble
wet imp
#

oh.

#

yikes

#

that's significantly lower than mine, but uh

wet imp
#

ive found i can avoid the issue almost entirely by eliminating inappropriate prompts and limiting the token usage

patent hedge
wet bison
#

We are still have the same problem. I had long email threads with open ai support, trying to understand how they calculate our token usage rate, cannot get a straight answer.

#

Anyone tried to apply for increase rate limit? It asks you to specify how much increases we need, without knowing how it is calculated, no way to tell ☹️