Threaded/Concurrent/Parallel Efficient API Requests

[red compass]

Hey everyone, I am aware of this implementation of parallel API requests to the OpenAI API: https://github.com/openai/openai-cookbook/blob/970d8261fbf6206718fe205e88e37f4745f9cf76/examples/api_request_parallel_processor.py

However, this code is too low-level for me and I'd rather have something that uses the "openai" module itself for easier maintanance and get rid of the low level stuff.

I already have many implementations for a threaded approach, however threaded approach doesn't make use of the token quota as good as the implementation above.

Does anybody have code written to fully utilize the token limits, but easier to understand and implement than the above approach? Thanks!

[mighty merlin]

For tokenlimit issue, If you use Langchain, Gpt-index, you can solve problem

[lunar acorn]

You're not going to find a simple way to do this without using other libraries. They aren't "low level", they just need to be able to access concurrent processes.

[hoary gate]

lol, i dont know what you mean with low level here, as it seems a little over-complicated to me... me myself integrated just a split asyncio queue that handles the requests in parallel as long as the limits are not full yet, and waits then just as long as it takes to free some space again. do you mean that?

[lunar acorn]

That's not what low level vs. high level means in the realm of programming.

It basically relates to how much control the programming language has, by default, over the hardware and architecture. C is a very low level language, which is why it is used a lot in firmware, for example.

[hoary gate]

Ohh, thanks a lot for the patience and the kind explanation😅 i am still a noob, especially in terminology😅

[lunar acorn]

No worries! :P