Stream using ReadableStream (SSE - Server Sent Events) | Runpod | Page 1

stuck ruin Jul 12, 2024, 5:31 PM

#

How can I stream results out of my runpod serverless worker?

Some background:
I am **NOT **using VLLM
I **am **using yield in my handler
Hitting the v2/{endpoint_id}/stream/{job_id} endpoint gives me back a JSON containing the partial response chunks and "consumes" the "queue". Am I expected to poll the /stream endpoint? Because if so I'm going to poll it every 50ms since it doesn't appear to be rate limited.

What I was expecting was a ReadableStream... did I misconfigure something? Should I mark my generator handler as async? Does setting return_aggregate_stream: True return a ReadableStream under /run?

daring wyvern Jul 12, 2024, 5:41 PM

#

No check general 👌

stuck ruin Jul 13, 2024, 12:00 AM

#

Answering my own question for posterity:

No, this is not supported, you have to poll the /stream endpoint, but this is not so bad, because it is blocking, meaning it will wait until a chunk arrives and reply immediately

lost mantleBOT Jul 13, 2024, 1:37 AM

#

Thank you for marking this question as solved!

Learn more

https://answeroverflow.com

daring wyvern Jul 13, 2024, 1:37 AM

#

Yay

#

I'll try to write a piece of feedback about this

#Stream using ReadableStream (SSE - Server Sent Events)