Hi guys, we observed 10x increase in latency compared to march for 3.5 turbo. I don't know what the issue is but something somewhere changed. Its gotten to outrageous levels of 50secs or more. Is this going to be rectified, is there any reason for this? You can check yourself https://ask.lawyerz.com
#Massive increase in latency
forest otter
spice isle
@forest otter Can you please share some example prompts and settings that are causing your requests to hit 50+ seconds?
forest otter
@spice isle The prompt is tailored to be law chatbot with user query and user queries can be anything, same prompt was working before, havinng said that, it got better now so something happened or someone changed something - may be is related to api key or something ?but it was hitting 30- 50 secs for almost last 10 days.
Having*
Setting is 0.6 temp. 300 tokens.
spice isle
Yeah, tbh 30-50 seconds has been somewhat normal recently for some requests, especially requests that use a lot of input text and ask for a lot of output text, and especially requests that use 3.5-turbo
I don't think there are any issues on your end with this
forest otter
Yeah, but the latency after optimization - we got it to 6 secs, so it's a gigantic jump.
But, it looks fine now. So maybe some settings update happened?
spice isle
How did you optimize?
spice isle
But, it looks fine now. So maybe some settings update happened?
The response time fluctuates widely during the day and week
spring jetty
Yeah, but the latency after optimization - we got it to 6 secs, so it's a gigan...
How did you optimize? Mind to share it?🥺 As Kaveen said , the response time isn't consistent.
daring bobcat
\startlette\routing.py line 671
flint fossil
Davinci-003 and gpt-3.5-turbo will timeout even after 10 retries when using a non-English prompt.
For example.
“Write a paragraph (in German) about trees.”
shadow hinge
Anyone still seeing this trend?
coral sparrow
<@213045272048041984> The prompt is tailored to be law chatbot with user query ...
Is the latency problem consistent with what the openAI CEO recentlt said to the US Congress: "We don't have enough GPUs" to process the user demands. Your app seems worthwhile. Do you have a way to sort law answers according to location/jurisdiction. Have you done any fine tuning? Can you merge your AIApp with the Casemine.com service/database? Have you integrated a citation generator or citation checker?
flint fossil
Last time there was an issue like this, a team member said it was because people were abusing the API.
It took 10 days for enough people to notice, hundreds of likes across Discord and the forum for OpenAI to do anything.
It makes me wonder if the OpenAI staff even use their own product. How can we get insane latency, constant timeouts, yet the Status page still says everything is operational… and the team doesn’t respond to anything.