#Parallelization in TypeScript / Node.js?

1 messages · Page 1 of 1 (latest)

trail snow
#

Hey all, I looked for similar questions around here but found answers only for Go. How do you achieve parallelization of tasks while using the Node.js / TypeScript SDK? The runtime is obviously single-threaded (one event loop), and all most of the chaining functions return concrete types, and not promises of results (with a couple of exceptions, output and mounts come to mind).

muted gust
#

you want to use async/await for parallisation - once you await, you're waiting until a specific task is complete

#

if you start a bunch of tasks without awaiting them, then they'll all be kicked off in parallel - then you just need to await them all together to get them done in parallel

#

using something like https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

MDN Web Docs
Promise.all() - JavaScript | MDN

The Promise.all() static method takes an iterable of promises as input and returns a single Promise. This returned promise fulfills when all of the input's promises fulfill (including when an empty iterable is passed), with an array of the fulfillment values. It rejects when any of the input's promises rejects, with this first rejection reason.

trail snow
#

Thanks, I know this. But as I mentioned in my question, the SDK does not return promises in the first place. dag.container().from("alpine") is of type Container, not Promise<Container>.

muted gust
#

ah, yeah, so in that case, this operation is actually lazy - nothing is happening in this case

#

the from("alpine") is not actually executed until you try and do something "concrete" with it

#

e.g. reading a file's contents, exporting the container, manually calling sync on it, etc

trail snow
#

In my current use case, I have a dag.container().publish("...", { platformVariants: [...] }), and I see the platform variants being built sequentially and not in parallel. Is there a default parallelism of 1 that I missed somewhere?

#

If I manually call .sync() on the variants, then Promise.all(...) on them, I see the two builds running concurrently

muted gust
#

ohhh

#

yeah, nice find 🤔

#

i think we know about this, @reef bison?

trail snow
#

So somehow I assume the runtime is simply waiting for the variants to be realized in sequence?

muted gust
#

we currently don't do evaluation of arrays in the api in parallel

#

i think this is a known bug

#

which we've never prioritised because i don't think anyone's noticed before 😄
but having expensive containers in publish is a real use case for why we should fix it 🎉

muted gust
trail snow
#

Interesting. So if I make a function return an array of things, the array will be passed to the API as-is, and those will happen in sequence?

muted gust
#

currently yup

trail snow
#

Actually not 😄

#

They do run in paralle.

muted gust
#

🤔

#

@fathom hare might be in scope for perf optimizations project you're looking at?

trail snow
#

Situation is as follows:

  • a function that returns an array of containers builds them in parallel when invoked from the CLI
  • same function, invoked by another fn that simply combines those containers as platform variants, then builds sequentially
  • reducing the example, a function that first creates variants then combines them in a publish call, builds sequentially

For the last two cases, pluggin in .sync() and Promise.all(...) at the right places restores the concurrency

reef bison