Sharing Build Artifacts

[elder obsidian]

Hi,

I am researching moving our team to Dagger from Earthly (which is unfortunately going away). One of the features we use a lot of is being able to share build artifacts across repositories. For example, we may have a build artifact in one repository that is needed during the build process of another. Earthly made this trivial by being able to reference the other repository and pull the artifact into the current build context.

Does Dagger have something similar to this?

[lean rock]

@elder obsidian what CI compute platform are you currently using? Jenkins? GitHub actions?

Dagger doesn't have a native Artifacts feature per ser, but it should be straightforward to create a module on top of your CI compute provider artifacts feature. LMK if that makes sense and if you'd like some guidance for that

[elder obsidian]

We are using GitHub Actions. I’m not trying to store artifacts long term, just build them from one repo and bring them into the build context of another repo.

[lean rock]

I think that's exactly the use case for GHA artifacts, to pass around data from one job to the other while adjusting the retention as needed. Is there anything particular around the Earthly artifacts which GHA doesn't support?

[elder obsidian]

Would i be able to do the following:

• Run Dagger to start a build for a Rust binary
• During the build, build a dependent WASM module from another repository
• Bundle the Rust binary and WASM module together into the final container

We don't really use GHA other than a nice UI for developers.

As an example, in Earthly we could do something like this:

IMPORT github.com/org/repo/wasm:v3.2.31 AS wasm

build:
  FROM somebase:latest

  COPY wasm+build wasm.module

Then running earthly +build would cause the WASM module from the remote repository to be built and stored in the current build context as wasm.module

I'm wondering if it's possible to get similar functionality in Dagger as a lot of our codebase is dependent in this way

The other reason we want to avoid GHA is because we want to keep Dagger's promise of "if it works locally, it works in CI." If we start adding custom GHA logic in builds, then developers can no longer reliably build locally.

Hope this makes sense.

[lean rock]

oh, yes this a first class feature of Dagger. I wasn't thinking about this use-case when you mentioned artifacts, I was mostly picturing the typical GHA case where you need to pass assets (files, folders, etc) across multiple GHA jobs.

Now, going back to the Dagger case, do you own both repositories? or is the second repo a third party one?

[elder obsidian]

We own both

[lean rock]

if you own this repository, the Dagger way to do this is via creating one module for each repository

so,you create a module in the "wasm" repo in your case with a function that returns the wasm.module *dagger.File

and then in the parent repo you do dagger install "url of wasm module repo" and that's pretty much it. Your parent's module will be able to call functions defined in your wasm module.

It's as if you were consuming any regular module from the Daggerverse (https://daggerverse.dev)

more info about that here: https://docs.dagger.io/api/remote-modules

LMK if you have any questions 🙌

[elder obsidian]

Ah, ok, that makes sense. Thanks! Does the install command do a global install or does it just add it to the local module?

I'm still a bit fuzzy on what constitutes a module. I'm hoping we can avoid having to have module source code in every single project in our monorepos. Can a module be as simple as the JSON config file?

[lean rock]

You can definitely avoid this if the process of building the wasm module is generic in a way which you can just fetch the code and just run a cargo build command to generate the file

[elder obsidian]

Yes, I am head of the platform team, so that's more or less what I'm hoping to do in the longterm

The goal would be to avoid devs having to write their own builds unless their doing something truly off the beaten path

[lean rock]

Yes, that's another way you can architect your pipelines. You have a centralized repo with Dagger modules that your devs can use remotely without the need of having anything in their repos

In this model, if your central module has a dagger.Directory field or function argument, you can just pass a git repo as a reference for that

I think we have an example in the docs. Let me check it out

[elder obsidian]

So in that case, we wouldn't even necessarily have to have modules in every project? The CI could just call dagger run ... against a remote module and pass in the local code.

[lean rock]

Yes, exactly

dagger call -m $url $functionName

https://docs.dagger.io/cookbook#copy-a-directory-or-remote-repository-to-a-container

^ there it is. Literally the first item in the cookbook

[elder obsidian]

Yeah, I saw that, just hadn't put together how to think about it from a more holistic standpoint. Appreciate your time in helping me wrap my head around it.

I think the last thing I'll need to figure out is chaining. With this more centralized (platform) approach, it's likely a specific task would require calling multiple modules in order to get the desire flow.

[lean rock]

You can have a wrapper function that just calls all the modules for you

It really depends on what APIs you want to expose to your devs

The best thing above all is that since everything is code, you have total control and flexibility about what you can express

There are other big teams following your same model. If you're looking for ideas and would like to get some feedback about your approach, I'd surely share that in #general

[elder obsidian]

Thanks, really appreciate your time and look forward to getting some prototype work done 🙂