#What's the right way to iterate over chunks of data in a streaming request body?

30 messages · Page 1 of 1 (latest)

silent cradle
#

I'm currently doing something like this, but I instead want to iterate over chunks of data, in the same chunks they were sent to the server:

async function consumeRequest(request: Request) {
  if (request.body) {
    const reader = readerFromStreamReader(request.body.getReader());
  
    for await (const line of readLines(reader)) {
      channelSend.postMessage(line);
    }
  }
  return new Response("Goodbye");
}
frigid ruin
#

you can do

for await (const chunk of request.body) {...}
#

although you don't really have any guarantee that the chunks you receive are at the same boundaries that the ones the client sent

#

there could be proxies along the way chunking the stream differently

silent cradle
#

@frail nexus hmm, tricky. I'm trying to pipe command output to my server via curl, and then render it using xterm.js.. it may be that the chunks don't matter and I can just write them to the terminal as and when they come through and everything will work itself out

#

the thing I'm not sure about is ANSI escape codes.. what they look like when you pipe them between processes (I'm guessing it's just the ascii representation?).. and then what would happen if I fed xterm a partial escape code, and then later gave it the rest

#

I guess I'll just have to experiment

frigid ruin
#

I suspect that depends on curl to some extent

#

btw, you pinged the wrong person

silent cradle
#

oops, very sorry other andrew!

#

not even other andrew.. my clumsiness knows no limits!

frigid ruin
#

(I mean, most English speakers will pronounce my name as "Andrew")

silent cradle
#

@frigid ruin phew! .. hmm, so say my data is utf-8 encoded text, I guess I'm not even guaranteed that the chunks will match up to character boundaries.. or am I?

frigid ruin
#

not really

#

but if you're using TextDecoder on the chunks, you can instead pass the stream through TextDecoderStream

#

which will buffer partial characters across chunks

#

or, well, partial code points

silent cradle
#

ah great, that's exactly what I want! I want to deal with it as a text stream and have it handle the partial code points

#

not really clear what the api is.. trying something like this const reader = readerFromStreamReader(request.body.pipeThrough(new TextDecoderStream()).getReader());

frigid ruin
#

you don't need the reader

# 
const decodedStream = request.body.pipeThrough(new TextDecoderStream());
for await (const chunk of decodedStream) {/*...*/}
silent cradle
#

ah bingo

#

but I don't seem to be allowed to iterate it in the same way

#

Type 'ReadableStreamDefaultReader<string>' must have a 'Symbol.asyncIterator' method that returns an async iterator.

frigid ruin
#

decodedStream in my code sample above should be a ReadableStream<string>, not a ReadableStreamDefaultReader

#

... weird, TS's default typings for ReadableStream say it doesn't have an async iterator, but it should have one

#

I'm pretty sure Deno's implementation does have it, at least

silent cradle
#

I guess deno does its own stuff with inlay hints so it stomps over other addons

#

ah, I was doing an unnecessary .getReader()