429 Rate Limit error but we are not hitting our RPM limit

[sinful nebula]

Is there a different reason why we would be getting a 429 rate limit error when we are using GPT 3.5 and are nowhere near our rate limit cap. Request ID for one such error: 3182be14703cf4823de624b580fe7593

[warm sun]

Have the same issue once in a while. I believe we get a 429 not only when our account rate limit is reached, but also when GPT seems to be at capacity. For now, I just retry the request a few times if I'm getting a 429 🤷

[sinful nebula]

It appears that this is what is happening. It's totally random - not like a rate limit cap

[stray pond]

I see this happening in our systems as well. Also comes with a decent 30s timeout. It would be cool if the api responded immediately with the 429 and some headers that let us know if our account is at capacity or openai itself.

Either way make sure to add retry logic 🙂

[sinful nebula]

Retry logic added 👍 thanks for the feedback!

[fickle acorn]

Having the same problem for the past three days. General 429 all the time.. Already lost a bunch paying users due to that… Will move to Azure as I’ve heard it doesn’t have that problem. Very annoying.

[brittle dirge]

@fickle acorn Let me know if Azure solves the problem, very frustrating

[fickle acorn]

Migrated to Azure and ran a test with the same task.

Azure - 39 sec
OpenAI - 1 min 31 sec

Azure is almost 2/3 faster and hopefully doesn't have the general rate limit problems. One thing to keep in mind is that RPM and TPM is a lot lower for users than OpenAI.

[brittle dirge]

Nice one. Think the pricing is different as well, slightly more expensive via Azure?

[crimson sapphire]

@fickle acorn You're hosting GPT in azure or are you using a different azure product? I'm facing the same issue and while retry logic mitigates the issue, the speed just becomes a bigger problem as wait time increases at every attempt.

[fickle acorn]

Pricing is the same

[brittle dirge]

Azure is offering OpenAI as a service

[fickle acorn]

No we use GPT through Azure.

[crimson sapphire]

Ok, thanks. Will look into that.

[fickle acorn]

https://website-754fwhahs-humanloopml.vercel.app/blog/open_ai_talk?utm_source=bensbites&utm_medium=newsletter&utm_campaign=openai-s-roadmap

Seems it's a general problem for OpenAI atm and will be a focus in 2023.

[brittle dirge]

Interesting. Also understandable how NVIDIAs share price is 🚀

I like this quote:

He suggested that a lot of people thought they wanted their apps to be inside ChatGPT but what they really wanted was ChatGPT in their apps.