#OpenAI streaming without Reactivity

1 messages · Page 1 of 1 (latest)

stark pike
#

<@&1228088700710293556> is there a way to stream from OpenAI response directly to frontend app without using database reactivity? because reactive database approach consumes a lot of bandwidth which isn't idea for us.

#

@reef sparrow

reef sparrowBOT
#

Let me check the knowledge base for you. One moment! :books:

#

Sorry @stark pike, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

pale notch
#

@reef sparrow try again?

reef sparrowBOT
#

Let me check the knowledge base for you. One moment! :books:

#

I'm sorry, but your question is a bit unclear. If you're asking about retrying queries or mutations in Convex, it's important to note that Convex doesn't have a concept of "retrying" if your query function hits an error. This is because Convex functions are deterministic, meaning if the query function hits an error, retrying will always produce the same error. Therefore, there's no point in running the query function with the same arguments again [source (https://docs.convex.dev/functions/error-handling#errors-in-queries)].

However, Convex will automatically handle internal Convex errors. If there are problems on Convex's end, it will automatically retry your queries and mutations until the problem is resolved and your queries and mutations succeed [source (https://docs.convex.dev/functions/error-handling)].

If you're encountering errors, it's recommended to handle these with a React error boundary component. This allows you to catch errors thrown in their child component tree, render fallback UI, and send information about the error to your exception handling service [source (https://docs.convex.dev/functions/error-handling#errors-in-queries)].

If you need further clarification or have a different question, feel free to ask!

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale notch kapa.ai is still learning and improving, please let us know how it did by reacting below

pale notch
#

@reef sparrow does Convex support streaming responses from an httpAction?

reef sparrowBOT
#

Let me check the knowledge base for you. One moment! :books:

#

Based on the provided knowledge sources, Convex does support HTTP actions, which allow you to build an HTTP API right within Convex. HTTP actions can manipulate the request and response directly, and interact with data in Convex indirectly by running queries, mutations, and actions [source (https://docs.convex.dev/functions/http-actions)].

However, there is no explicit mention of support for streaming responses from an HTTP action in the provided sources. The HTTP actions in Convex take in a Request and return a Response following the Fetch API [source (https://docs.convex.dev/functions/http-actions)]. The httpAction function in Convex receives an ActionCtx as its first argument and a Request as its second argument, and it returns a Response [source (https://docs.convex.dev/generated-api/server#httpaction)].

While there is a mention of streaming responses in the context of ChatGPT responses into Convex [source (https://stack.convex.dev/gpt-streaming-with-persistent-reactivity#summary)], this does not directly relate to HTTP actions.

Therefore, while Convex does support HTTP actions, it's not clear from the provided sources whether it supports streaming responses from an HTTP action. For a definitive answer, you may want to refer to the official Convex documentation or reach out to the Convex community or support.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@pale notch kapa.ai is still learning and improving, please let us know how it did by reacting below

pale notch
#

Mitigations for a lot of data writes I can think of:

  1. Stream it one line or sentence at a time rather than per token.
  2. Stream it from the API using a next.js api route, then write the result at the end to the DB (but accept that you might never write it if the request gets interrupted). You can use the streamed response as ephemeral data on the client until the message comes down after being written. Like an optimistic update. Note: you can't see the stream on multiple clients or after a page refresh in this case.
  3. Paginate the data from the frontend, so each query page isn't loading as much data on the fast-refreshing data. You could even have a separate subscription / query on the "in-progress" message so it isn't in the default query fetching path.

#3 is related to concepts from https://stack.convex.dev/queries-that-scale

Queries that scale

As your app grows from tens to hundreds to thousands of users, there are some techniques that will keep your database queries snappy and efficient. I’...

#

@stark pike ^