#Multithreaded build script

17 messages · Page 1 of 1 (latest)

rough wigeon
#

Basically, I want to run some npm scripts for nested projects in the build script. They are awfully slow (maybe skill issue?) so I want to multithread in the most simple and straight forward way. Here's what I currently have:

fn main() {
    let current_dir = env::current_dir().unwrap();

    let widget_api_dir = current_dir.join("widget-api");
    let widget_api_handle = std::thread::spawn(move || npm_run_build(widget_api_dir));

    let widgets_dir = current_dir.join("widgets");
    std::fs::read_dir(widgets_dir)
        .unwrap()
        .flatten()
        .filter(|entry| entry.metadata().unwrap().is_dir())
        .map(|dir| {
            let path = dir.path();
            println!("cargo:rerun-if-changed={}/src", path.display());
            println!("cargo:rerun-if-changed={}/package.json", path.display());
            std::thread::spawn(move || npm_run_build(path))
        })
        .chain(std::iter::once(widget_api_handle))
        .for_each(|h| h.join().expect("failed joining build command"));

    println!("cargo:rerun-if-changed=server/widget-api/src");
    println!("cargo:rerun-if-changed=server/widget-api/package.json");
}

fn npm_run_build(path: PathBuf) {
    // change dir to path
    // npm run build
    // change dir back
}

The widget-api build command doesn't seem to run. Note that all these projects output to the same directory, I don't know if that's relevant for any filesystem locking issues. Also, widget-api and widget directory contents don't have a direct dependency relationship, as it may mislead you, so they don't have to execute in order. Otherwise, I'm a bit clueless of why this is not working, but I sense a dumb mistake. Also, taking any suggestions on how to improve this.

wide latch
#

This doesn't work because the build script ends before the thread even starts up I missed the join

#

You can't use a build script to do operations while cargo continues on

terse inlet
#

but if you want to wait, std::thread::scope can fix that for you

#

also, you wrote

    // change dir to path
    // npm run build
    // change dir back

but changing directory will not work in multithreading because there is only one current dir for the process, so they will interfere

wide latch
#

Wait there is a join

rough wigeon
rough wigeon
#

Still, I'm a bit confused by the results, either way I'm going to make the changes tomorrow and if it works, it works

rough wigeon
#

Now working, thanks! It was definitely the changing directories

rough wigeon
#

@here On another note - cargo:rerun-if-changed doesn't seem to be behave as I expected, the build script is running every time and it's slooooow.
Are the paths maybe incorrect? How do I debug this?

terse inlet
#

cargo build --verbose will print info about why Cargo thinks a rebuild is needed

#

also, should be cargo::rerun-if-changed (the single colon is deprecated but works, though, so that's not the problem)

rough wigeon
terse inlet
#

What was it? I'd like to know to have better suggestions for the next person.

rough wigeon