I am running into an issue where we occasionally see one of our requests to gpt4-turbo take more than 90+ seconds. Most of them take 2-3 seconds but we are seeing an occasional one take 90-92 secs fairly consistently and its very reproducible. It almost seems like one of the openai backends is not healthy so when it receives one of the n requests we send, it ends up taking a long time. Can this please be looked into?
#GPT4-turbo requests taking 90+ seconds
grim dagger