is there a way to stream from OpenAI response directly to frontend app without using realtime database? because reactive database approach consumes a lot of bandwidth which isn't ideal for us.
#Streaming without realtime database updated.
near dagger
solar oak
sorry about kapa.ai not being helpful. there is currently no way to stream data out of an action directly. i've worked on this and it's not done yet. the only ways to get intermediate data out of an action are to store it with a mutation or send it to some other server with a fetch (which does support streaming requests and responses)
tacit needle
Mitigations for a lot of data writes I can think of:
- Stream it one line or sentence at a time rather than per token.
- Stream it from the API using a next.js api route, then write the result at the end to the DB (but accept that you might never write it if the request gets interrupted). You can use the streamed response as ephemeral data on the client until the message comes down after being written. Like an optimistic update. Note: you can't see the stream on multiple clients or after a page refresh in this case.
- Paginate the data from the frontend, so each query page isn't loading as much data on the fast-refreshing data. You could even have a separate subscription / query on the "in-progress" message so it isn't in the default query fetching path.
#3 is related to concepts from https://stack.convex.dev/queries-that-scale
near dagger
Thanks @tacit needle! Already tried #1 but it still taking a lot of bandwidth and tried #3 too, unfortunately it was affecting UX which required more frontend work to fix it. Before trying #2 I wanted to confirm it with you, as @solar oak mentioned that they are already working on this, do we have an estimate when this could be ready?
tacit needle
I can't remember the timeline on streaming from http actions. @spark dock might have a better idea. but to be clear, this would not be streaming from a regular action / mutation, and you'd want to store the result alongside into the DB (maybe writing the incremental values or just the whole value at the end), and filter that result out using an index, from the client's query, until it's done
spark dock
No timeline on streaming http actions. I vaguely remember it being somewhat non-trivial when we discussed it so it'll take a moment before we figure this out. But as Ian said you can probably try this with a different http service. The important part is to save the result to the DB at some regular interval.
near dagger
okay! thank you 🙂
valid mirage
I've also been looking at this, as my bandwidth has been eaten up from this. Number 2 seems like what I'll go for.
tacit needle
We have streaming from http actions working internally on a branch as of today. No guarantee on when it'll ship, but it's progress!
valid mirage
Awesome to hear that it's in the pipeline, one of the reasons you guys are awesome is your awareness of generative Ai, the vector Db etc
coarse wedge
@tacit needle @spark dock will streaming be live soon I am revamping the convex code and thinking of going with 2nd approach but if http actions streaming is coming soon then I can wait. Thanks
spark dock
In progress! I'll check with the team and get back to you.
coarse wedge
Great THANKS
spark dock
There are some edge cases we're working through. Hopefully in a few short weeks so we can make sure it's working well.
tacit needle
@near dagger @valid mirage @coarse wedge we now have http response streaming - so you can stream directly to a client and only periodically write to the db (or not at all). Check it out: https://news.convex.dev/announcing-convex-1-12/
And sample code: https://github.com/sshader/streaming-chat-gpt/blob/sshader-streaming/convex/http.ts
rocky thistle
<@681163390344757260> <@523246075646181376> <@260825949979803649> we now have ht...
I haven't had a chance to get into llama-farm example yet but does it streaming work different or the old way as it was released before 1.12?
tacit needle
It's using the normal flow - it writes to the DB at the end of sentences/clauses, since the user request isn't being piped all the way to the worker. I wrote this post on the implementation: https://stack.convex.dev/implementing-work-stealing
near dagger
That's supercool! Thanks @tacit needle & team convex 🙂
valid mirage
Yuss, awesome! Been waiting for this!!
tacit needle
Sarah recently wrote a post on it: https://stack.convex.dev/ai-chat-with-http-streaming
and a quick video of using the ai npm library:
https://www.youtube.com/watch?v=kP0HYN6NpA0
