#Speed up the process of calling the API

9 messages · Page 1 of 1 (latest)

hot jewel
#

Hi,
I have a task for which I need to call the completions API several thousand times and expect this to take several hours.

I don't need help with optimising the prompt itself, but to understand what's the more recommended way to parallelise the calls to the api so that the task completes a little faster.

With a paid account I understand that I can do up to 60 calls per minute. Making the calls one at a time, I currently do about 10 calls /minute so I assume I could theoretically 6x this if I parallelised the calls.

Does anyone have Python libraries or code to suggest? (assuming openai allows this practice)

Thanks!
Alex

grave ether
#

Hey! What things have you tried so far? I think this is achieveable by making requests with AIOHTTP and keeping track of the total amount of requests you're making within a 1 minute window, if you want to keep it under a certain limit

#

So, when requests are happening, you'd check how many running requests there are and only launch new requests if it's under a certain threshold

#

You can also achieve similar results using multiprocessing in python, or by creating a thread for each request (the internal request should still be an asyncio/aiohttp request tho)

hot jewel
#

Many thanks for the tips @grave ether . Can you confirm that openAI would allow multiple calls via a single ip and single api key? If so among the two methods you suggest which would you recommend the most? I 'm mainly an R user and in the R world there is mainly one way to do this via a package called furrr. In python (by asking chatgpt) I see that there are at least 4-5 different ways (asyncio, threading, multiprocessing, concurrent.futures, request-futures and more) As a side note, I 've noticed here https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits that with paid account after 48 hours the text completion request/min limit jumps from 60 to 3000 which I assume suggests there is a massive option for parallelisation.

Is API usage subject to any rate limits?
grave ether
#

As long as you stay under the rate limit

hot jewel
#

Good to know, thanks very much for this feedback

grave ether
#

My pleasure