Seeding cache volume from host

[proud tinsel]

Curious if there's a method to "seed" a specific cache volume. Working on some embedded linux builds, and I have about 300GB of source code etc that I need available in my build container prior to running builds.

I'm working with the new modules sdk, and I know there's restrictions around host volume access, but it seems like a natural (albeit large) fit for cache volumes. Wondering if there's an api where I could load up a cache vol and then find it via something like dag.LoadCacheVolFromID(foo)

[sleek flicker]

Hey Steven! Are you still looking for a solution to this?

[proud tinsel]

yeah - ish... I mean it's not really the most critical path for this build. I ended up just seeding into a built container

but I think more visibility/control over cache vols would be nice

[sleek flicker]

so, @proud tinsel if you're using modules this is pretty straightforward.

cache volumes are globally referenced by name across all the pipelines so as long as you use the same dag.CacheVolume("name") across different pipelines, you can seed those with whatver data you need 🙏

LMK if that makes sense and if that helps your use-case 🙏

[proud tinsel]

nah not using modules - tried it and devs on my team actually hated it 😅

[sleek flicker]

wow would love to know more about why devs hated it if you haven't shared that feedback yet 🙏 . cc @jaunty mason

re: cache volumes, you can also do the same with the plain SDK since cache volumes are always referenced by name. You can basically create a function that just mounts that cache volume and puts whatever is needed there.

[proud tinsel]

Sure so I think it's a few things.

1. Our prior workflow (no modules) involved just calling go run . - and we could add some args in the cli to do specific things. I think the modules workflow intended to have a similar flow, but it ended up requiring a ton more args and knowledge of the dagger cli. (Also in our case they didn't even have dagger installed previously 😅 )

Example: In our custom cli, I put all our secrets in vault, and just fetch the local ~/.vault_token or VAULT_TOKEN env var from the host. This isn't possible in modules - they have to be passed in. So now our command becomes something like dagger call build TARGET --dir MY_DIRECTORY --token FOO. Developers now need to know or discover more dagger specific args or language to accomplish tasks.

2. Similar to above. Passing in source is no longer obvious or straightforward. We have a couple monorepo style projects, and I want to be careful about the context we're passing in. Devs might have huge files in their local directory, and that can nuke build times locally. With the "advanced" golang cli, we can specify in code which directories we want, and remove the responsibility of knowing the proper directories etc from the developer.

Example: A command that was previously go run . publish PROJECT_NAME - is now dagger call build-project --dir project_name --dir2 project_dependency_folder --token FOO - if they pass in the wrong directories, builds fail. If we just pass in the top level directory, now we have a huge context that we have to pass to the intermediate dagger engine before we do any host excludes.

3. Additions / modifications to our CI. In the old , go native workflow, devs could literally just open up the ci directory and autocomplete and everything, justwork.jpg. Now they need to have dagger, call dagger develop and do a few other things to get going.

and building on 3 cause I ran out of characters - in the prior workflow they could get stuff from the host, or call other golang libraries directly within our CLI. not being able to access the host directly (without args) made it another learning curve compared to natively executing go code

oh and our workflows around SSH agents kind of broke so auth for terraform git modules or go modules got cumbersome

Still a huge fan btw, hope I dont sound to critical, I know I've been very active over the past couple weeks. We've been building out a new product/pipeline that had some challenging requirements

Also really excited for v11 and integration with OTEL tracing, would love to start adding some tracing to my calls outside dagger. (things like s3 uploads)

[jaunty mason]

Thanks so much for all this great fedback Steven. I am capturing all of it and will share with the broader team.

[jaunty mason]

oh and our workflows around SSH agents kind of broke so auth for terraform git modules or go modules got cumbersome

Hey Steven, following up on this one in particular, was this using the old approach where you can pass sockets around, or something else?

[proud tinsel]

Yeah passed host socket / agent

[iron summit]

Found this thread doing a search around sockets... 😅
@proud tinsel What's the alternative you are using now to overcome the ssh agents workflows for private go modules/terraform? (if any!)

I'm on my early stage of discovering dagger and finding a workflow that is easy to introduce for my devs.
We're using docker + ssh agent to access private git repos during the build phase. Initially I was trying to simply consume the same dockerfile in order to slowly start introducing dagger without them having to do much...

[proud tinsel]

@iron summit - start a new thread, might make it easier to find for others