Hitting API Rate limits consistently

[high grail]

I'm attempting to start building a new app, and within less than 20 messages, I'm consistently getting an "[Provider] is temporarily overloaded" both for OpenAI Api and Anthropic API usage. Is this an issue with Chef, or are the default API rate limits for both providers not sufficient for Chef's token throughput when operating?

[ebon pilot]

hi @high grail this means we're hitting 429 or 529 rate limits from anthropic and openai. are you still running into these constantly/consistently? we can take a closer look if there's a broader issue

are you using the API keys built in to Chef or using your own?

[high grail]

Using my own. Will try again now.

[ebon pilot]

ah, yes this is the issue

[high grail]

Hit the same "temporarily overloaded" message for both providers just now on first new prompt attempt

[ebon pilot]

the key you're using has limited tokens per minute throughput. we have a key with a higher token rate we use for chef when a key isn't provided

i'm guessing you likely have a limit of 40,000 TPM on your anthropic key?

[high grail]

Yeah that's right

[ebon pilot]

yep given that it's hard for individuals to obtain keys with higher limits you might need to wait a minute every now and then, or upgrade to Pro and use the keys provided by Convex

tokens are hard to come by these days!

[high grail]

Just to clarify, to use the Convex-provided keys I just need to sign up for a regular convex Professional plan on your main website?

[ebon pilot]

yes you have three options right now:

1. use the free Chef plan without a custom API key but be subject to the lower limits
2. upgrade to Convex Pro via our main website (we'll add a more convenient link soon)
3. keep using your own key and either find a way to get a higher TPM rate limit or wait for us to add better backoffs and retries so it just slows down when you hit limits rather than throws an error

[high grail]

Sounds great thanks, excited to play around with this more.

[fast tangle]

Hi James, any schedule for adding better backoffs and retries so it just slows down when hitting limits rather than throws an error?

[ebon pilot]

cc @rose cloud since he's been working on token usage ^

[rose cloud]

hi @fast tangle ! we automatically retry requests with backoff. are you noticing issues with the chef provided API keys, or are you using your own?

[fast tangle]

hey Jordan, thanks for getting back, with the pro subscription of chef I do not have this kind of issues, but using my own Anthropic API key i'm getting this message very frequently: Anthropic is rate limiting your requests