I am planning to test this idea, but can

[ripe parcel]

Might work often, but not guaranteed because cache volumes can be pruned from local disk at anytime if the engine needs to cleanup disk space usage. They are meant to be best effort only.

One reliable way of accomplishing what you're describing is to have a module with a function that returns the binary (i.e. as a File type). Then anyone that needs the binary can call that function. At that point the engine cache will store the binary for you, but if the binary ever needs to be rebuilt that will happen again on the fly.

cc @unique sun @modest ibex in case you have any other thoughts on this sort of pattern

[unique sun]

The most common pattern today is what @ripe parcel described by having your container build function call your binary build function instead of relying on the caller to orchestrate the two functions.

We've also been noodling on a first class Artifacts concept which could do what you're looking for automatically, but that design is still in progress

[tulip depot]

In order to avoid the cache cleanup, I would envision this pattern being in a pipeline. Step 1: build the thing, Step 2: package the thing. So, one module called right after another. I wouldn't rely upon this kind of pattern if there was any time between step 1 and step 2. Interesting dagger allows it though.

A nice idea about this pattern is that it feels more flexible than a module calling a function in another module. I suppose to really make this function the build module would have to provide a function that returns the name of the cache volume, so it's really the same pattern as you suggested just probably less reliable. 😄

[unique sun]

A nice idea about this pattern is that it feels more flexible than a module calling a function in another module.

Can you elaborate on that? Like because you don't have to worry about calling another function, just that you expect a file to exist somewhere?

I suppose to really make this function the build module would have to provide a function that returns the name of the cache volume

This way wouldn't work because cache volumes are scoped to a module, so you can't share a cache volume by name. This prevents modules from needing to worry about cache collisions with other modules

[tulip depot]

A nice idea about this pattern is that it feels more flexible than a module calling a function in another module.

Can you elaborate on that? Like because you don't have to worry about calling another function, just that you expect a file to exist somewhere?

Well, it feels more flexible in that I can have an arg to the module where I can pass the name of the cache volume. In my example where a pipeline could build a binary from one of many language options (go, c++, etc) but the packaging is constant, you could have the name of the cache controlled by the pipeline. Pass the cache name into the build module, it produces the binary on the cache volume, then the package pipeline can retrieve the binary from the cache volume. It's more like a soft link, whereas it feels more like a hard link if the package module calls the build module. In this scenario the package module would have to have calls to all the build modules and decide which one to use.

I suppose to really make this function the build module would have to provide a function that returns the name of the cache volume

This way wouldn't work because cache volumes are scoped to a module, so you can't share a cache volume by name. This prevents modules from needing to worry about cache collisions with other modules

Ah, this is what I wondered about. I presume then to do this you would have to pass the actual cacheVolume and not the name. At which point, imo, a function that returns a cacheVolume seems no value over one that returns the dir/binary.

[unique sun]

Just to make sure I understand your scenario, you have

1. Several build functions that produce binaries from all kinds of source and build processes
2. A single unified packaging function that can take any of those binaries to produce a Container

[tulip depot]

Correct

[modest ibex]

Sounds like a good use case for a composite artifact: a custom object type with each component file as a field. Then you can attach high-level packaging to that object

It's generally a good idea for designing your Dagger code, to start from modeling artifacts, then backwards to functions

[unique sun]

Or if you don't need a composite artifact, like if there's no single lens that knows about all builds and their packaging, it could be the inverse of the original suggestion: each module that has a build can import the packaging function and use that