Is there a way to cancel/stop a completion request mid-stream? Or does simply not reading the rest of the completion off the stream mean you aren't billed for the tokens?
Thank you.
#Is there a way to cancel/stop a completion request mid-stream?
unreal prism
halcyon field
Based on my knowledge, the process can’t be stopped
the rest of completion will just be sent to you and calculated no matter whether you receive it
silent coyote
So there is no need to implement an cancel fucntion
gusty blaze
I have been able to cancel streams and have also confirmed that the tokens stop being generated by double checking my usage in the OpenAI dashboard.
This doesn't work with all providers though. I have confirmed that it will not work with Vercel (Next.js), but I have gotten it working using Deno and am pretty confident I could do it in Node if I tried.
quartz jackal
Do you have github code that we can refer to? 🙏
gusty blaze
You are interested in an example of the Deno server that handles the Open AI stream and the cancel code on the client?
graceful river
You need to make sure that your client can actually abort the connection and close the stream. Many clients will keep the connection open.
gusty blaze
You need to make sure that your client can actually abort the connection and clo...
In order to abort the connection I use the AbortController on the client fetch. On the server I have a cancel callback that will be called in this situation.
graceful river
That may work, in my client side I had to implement my own axios adapter 😅
gusty blaze
Hey quick update. Seems like a solution is on its way for nextjs. Can check the discussion here: https://github.com/vercel-labs/ai/issues/90
I ended up shipping a Fasitfy server on Node 20 using AWS, but glad that vercel should be able to support this soon.