mighty bay
I'm trying to stream an array from my database (clickhouse) to remix UI.
The clickhouse client gives me batches of rows, which should then send to remix, and refresh dependent components.
But I can't figure out how to send the batches as such, I can only accumulate in the loader and then send the whole thing, which means users wait for less relevant data, even though the most important data has already been send.
Is there some way to send along batches using a mechanism similar to defer?
vernal pendant
Is there a way for you to get an array of promises back from that api? If so you could take the first few slices as the important chunks each as an individual promise to return and then have a final “catch-all” that returns whatever remaining unimportant chunks you have
mighty bay
I mean I could make that, but that'd be a sad solution TBH.
I want to show this as graphs down the road, and it'd be much preferable if they'd smoothly load in
If everything goes right I can get them to smoothly animate from right to left in typical network conditions
vernal pendant
I mean to use the loader to do it, you’d need to return individual promises to the loader’s defer, and I’ve only seen that done as a specifically named item on an object, so I’m not sure if there’s any way to spread N many promises that have no name into the defer to then be grabbed later
You’d probably want/need to manage it on the front end with fetches or a web socket to slowly return the results
You could try out what happens if you return an array of promises to the defer as a key and see if you can Await individually on the front end. But I’m not sure defer will understand what to do with an array of promises. I have a hunch it would see it has an array and think there’s nothing for it to do there and just give you an empty array on the front end
mighty bay
Hmmm... Is there maybe a way to get Server Sent Events to work via a remix loader? That would fit the bill, I've used it before in vanilla JS but not sure how/if possible in remix
vernal pendant
Yes there is, check out remix utils for an easy implementation of it. Or you can hand roll it