bold wedge
Hey, we just signed up to try the api and currently have about 6 timeouts (after 30 seconds) and 521 Responses per 100 api requests. Our prompts are small and request normally take less than five seconds to complete.
Is there currently a problem with server load?
We are using the mistral-small model
bold wedge
API Timeouts or 521 Response
Additional note: We are not using streaming
Originally we did about 3 requests in parallel, but these problems still persist when only doing one request at a time
oak pelican
Hi Jan, we are actively working on improving that, I’ll ping you when I have news
bold wedge
Alright, thank you! Just read that the api is not stable yet, so this is understandable 🙂 Any issue if we just retry the request when the api times out or returns a non successful status?
oak pelican
that’s what you should do 👍
merry geode
same issue here.
I ran a few benchmarks earlier on mistral-medium and I was getting around 40% error rate. It was sequential calls, not streaming. The errors happen in a sequence, so there seems to be a continuous issue - full capacity?
It always works fine the second time with a little break in between.
oak pelican
This should be better now @merry geode , can you try again?
merry geode
I've done the benchmarks I needed for now, but I'll try after work
merry geode
Yes, this was much improved. All my benchmarks yesterday evening and this morning went smoothly, but it came at a cost of "worse" performance (see the thread in channel models)
bold wedge
This should be better now <@816256227439935508> , can you try again?
It seems better now, but we still get occasional errors - even after retries. The api also seems very slow, does this also show up in your analytics/monitoring?