caching failures with rsync. blocker for putting in production.

[glacial olive]

hi team, we have a docker container with rust and node. to parallelize the build part we split the rust and node build parts and then do an rsync to merge these. but that rsync essentially is never cached 😦

first the code that is the culprit:

    runner = runner.with_directory("/backend3", full_backend).with_exec(
        ["sh", "-c", "rsync -av /backend3/ /backend"]
    )

Is there a way to make things work here without making rsync go bonkers ;((
This is the main question

details:

The idea is we want to copy over a large part of our build before we copy in the sources. we first copy the pnpm lock and then rust artifacts before we copy over the full_backend which are just the main sources we have to build. here is the full code for the build:
https://gist.github.com/sualehasif/ba4fd5c386188fffc96fe40725184b32

Can a dagger pro plese let me know what i am doing wrong here. what is a better way to do this that doesnt use rsync.

caching failures with rsync. blocker for putting in production.

how do people usually handle merging built dependencies into the code that is being copied in.

[glacial olive]

thanks for all the help in advance

[feral ruin]

Hey! When you use with_directory, you're effectively doing a copy. No need for rsync. You can also use methods like with_new_file instead of relying on with_exec for creating directories and files.

See basic concept of multi-stage build in https://docs.dagger.io/cookbook#perform-multi-stage-build

[glacial olive]

mmm so what would you replace this line with:

runner = runner.with_directory("/backend3", full_backend).with_exec(
    ["sh", "-c", "rsync -av /backend3/ /backend"]
)

assuming you mean we should just do

runner = runner.with_directory("/backend", full_backend)

doesnt this completely overwrite the backend dir?

[feral ruin]

I don't think so, no.

It overwrites, but doesn't completely replace.

[glacial olive]

so its equivalent to rsync?

[feral ruin]

Doesn't remove what's diff either.

[glacial olive]

runner = runner.with_directory("/backend", full_backend)

does this look correct to you?

[feral ruin]

Yes, it does.

[glacial olive]

    include_files = [
        "package.json",
        "patches",
        "pnpm-lock.yaml",
        "pnpm-workspace.yaml",
        "server/package.json",
    ]
    package_files = client.host().directory(
        "backend",
        include=include_files,
    )
    runner = runner.with_directory("/backend2", package_files)
    for include_file in include_files:
        runner = runner.with_workdir("/").with_exec(
            ["cp", "-r", f"/backend2/{include_file}", f"/backend/{include_file}"]
        )

mmm when you replace this with

    include_files = [
        "package.json",
        "patches",
        "pnpm-lock.yaml",
        "pnpm-workspace.yaml",
        "server/package.json",
    ]
    package_files = client.host().directory(
        "backend",
        include=include_files,
    )
    runner = runner.with_directory("/backend", package_files)

it doesnt cache at all and bugs out but i might be getting it wrong.

thankyou! i will try it out!

[feral ruin]

Doesn't cache? How so?

[glacial olive]

any other thing we are getting wrong? i.e. that we should fix.

i.e. things that are not mmmm idiomatic

[feral ruin]

Well, looking at your code I think that can be simplified quite a bit! But let's take one step at a time.

[glacial olive]

gotcha. lemme simplify the first parts and post here.

[feral ruin]

By the way, you no longer need to pass around the dagger.Client instance: https://dagger.io/blog/simpler-reusability-with-the-global-client

Here's a simple example of installing from requirements.txt first, before adding the rest of the files: https://github.com/dagger/dagger/blob/6a3689eeb680920fe5f830ac972be3dc1fa4f29b/internal/mage/sdk/python.go#L251-L253

Creating the dummy main.rs files doesn't seem like the proper thing to do, but I'm not familiar with Rust to understand the constraint.

But for curiosity, this:

runner = runner.with_exec(
        [
"mkdir",
"-p",
"/backend/packages/pinecone-rs/pinecone_client/src",
        ]
    )
runner = runner.with_exec(
        [
"sh",
"-c",
"echo 'fn main() {}' > /backend/packages/pinecone-rs/pinecone_client/src/lib.rs",
        ]
    )

Is the same as:

runner = runner.with_new_file("/backend/packages/pinecone-rs/pinecone_client/src/lib.rs", contents="fn main() {}")

[glacial olive]

i dont remember why that code was written but probably it was because cargo fetch wasnt doing the right thing (i.e. compiling the code in the dependencies)

and that is crucial to good caching for rust atleast for us.

[feral ruin]

I'm sure there's a proper solution to that 🙂

[glacial olive]

yep i buy that.

im working on fixing the current bugs to try and improve caching!

last q:

how do you recommend setting up ci so that tests/lint/integration run in parallel

in one dagger run via python parallelism or via running it as different github actions?

thank you so much @feral ruin

[feral ruin]

Dagger already knows what can run in parallel based on the fact that you have dependent inputs or not, you just need to make sure to make concurrent requests otherwise you'll need to wait for the response of the first thing (e.g. lint), before starting to request the next (e.g., test). You can see an example of making concurrent requests in Python in: https://docs.dagger.io/cookbook#execute-pipeline-operations-concurrently

Of course you can also delegate to your CI executor and put it in different tasks. The downside is unless you control the runner, the cache will be less optimal then.

For example, in github actions, if you're using free runners, every job will be allocated to a different machine. Different dagger engines, not persisting cache. So both will have a cold start.

If you use concurrency and a single job, they'll both be executed in the same runner, and will benefit from the same cache, assuming there's some things that are shared (even if a base image).

[glacial olive]

we are using dagger cloud mmmm

shouldnt that be resolved with a distributed cache?

[feral ruin]

Oh, if you have Dagger Cloud's distributed cache than 👍

[glacial olive]

ya we plan to keep using dagger cloud until yall kick us off. in which case feeels like things should just be cached by default

last question for things like

    # create a cache volume
    node_cache = client.cache_volume("node")

do i need to configure anything or does dagger cloud handle this automatically?

[feral ruin]

It's handled automatically but I recommend choosing a more granular name for the cache volume rather than just "node".

[glacial olive]

mm

[feral ruin]

If you change to another node version, or use alpine instead of slim, that cache key should change appropriately to avoid incompatibilities.

[glacial olive]

if you already have a lib.rs in a location and you copy over a lib.rs does it override?

[feral ruin]

Yep

[glacial olive]

mmm @feral ruin somehow things are working kind of weridly locally for me as well. in that running the same thing twice pulls the postgres and redis containers again and again?

i.e. this part

https://gist.github.com/sualehasif/ba4fd5c386188fffc96fe40725184b32#file-dbs-py

everytime i run it pulls these images from scratch?

unclear if this is a storage problem or a disk problem?

it tries to resolve them everytime??

i also rewrote https://gist.github.com/sualehasif/ba4fd5c386188fffc96fe40725184b32#file-build-fast-py

but it doesnt seem to help?

[feral ruin]

Good morning! It helps to debug if you create a reproducible example, as minimal as possible, so I can run it too. Then it's easier to check what works minimally and build on top of that.

Are you connected locally to Dagger Cloud? And which dagger version are you using?