#Docker host mounts?

1 messages · Page 1 of 1 (latest)

broken cypress
#

Is there a way to make a host directory mount the same way the Docker “run -v” options does? I have a large monorepo and I don’t want to copy it into a layer on every dagger run, just mount it from the host.

haughty panther
#

no, that's not possible. Dagger works similarly to docker build where all the artifacts to run your workflows require a "shared nothing" environment since builds should also be able to happen remotely.

Having said that, in your case of the monorepo, the context will only have to be sent once since Dagger keeps track of the files uploaded to the context and only transfers deltas each time

broken cypress
#

Hmm, how would I verify that it's sending deltas? Experimentally, the upload host directory process takes some minutes every time. (the total directory size is around 6G).

#

I guess my followup question is, if this isn't possible, then that means there's no way to bind a host directory to a writable mount point? So a build process that runs inside the container can't ever update files in the host directory (without me explicity writing SDK code to copy those files back)?

haughty panther
haughty panther
broken cypress
#

Is there a fundamental limitation that means traditional container mounts can't be done in dagger? Or it's just not a feature that is currently available?

haughty panther
# broken cypress Is there a fundamental limitation that means traditional container mounts can't ...

it's not a limitation, it's a design decision of the system. For Dagger / Docker build and any other container building tool, it's important to guarantee the input/source of what you're building plus the ability to be able to do it over the network if required (imagine targetting a remote builder with different CPU architecture, etc). Doing this while allowing to bind mount directories from the host, goes against this design principle

broken cypress
#

That definitiely makes sense from a CI perspective. Lemme describe the experiment that I was making ...

#

We have a large monorepo, mainly Go and Python , and lots of committed binaries. This only builds on Linux. There is an effort to switch over to a nix+bazel toolchain. So that's complicated to get the tooling set up right and some people (most importantly, me!) only have macOS dev systems, so depend on containerized builds. The nix+bazel stuff really depends on caching build artifacts, since it takes a super long time to build all the dependencies.

frozen sedge
#

Oh hey, good to know this is Nix related, that second issue link is even more relevant then

broken cypress
#

So I was experimenting to see whether I could wrap a build command in dagger. The workflow would be to checkout the monorepo, then do "some-tool bazel build //..." and that would invoke dagger, with caching volumes and do all the right thing in a canonical biuld environment, but operating on the user' current working copy of the repo.

#

So knowing that this won't work, is a good result for my experiment! 🙂

frozen sedge
#

I've had some success with using Nix in Dagger by using a persistent cache volume mounted to /nix/

broken cypress
#

Yeh, we have some existing tooling, shell scripts + Dockerfiles. I was hoping to improve on that.

frozen sedge
broken cypress
#

Thanks for all the info @haughty panther @frozen sedge !

frozen sedge
#

This approach is all within Dagger, no scripts/Dockerfiles necessary

broken cypress
frozen sedge
#

Ah cool - are you running into gotchas there?

#

Or is this mostly about the large host dir syncing performance issue

broken cypress
#

The problem is with mounting the working copy. Since you can't mount a host volume read/write, you can't do things like run gazelle inside the container to update the bazel build.

frozen sedge
#

It's been a while since I've used it, does Gazelle do codegen? And you want a way to sync those changes back down?

broken cypress
#

yeh, gazelle generates bazel bulid files for go

#

But in general, the user could run any build command that generates anything. I suppose that I could run git inside the container to figure out all the new and changes files and copy them back ...

frozen sedge
#

Maybe you could take Diff from the original mount and then Export that to the host dir?

broken cypress
#

hmm ... would it make a difference whether I used WithMountedDirectory or WithDirectory for that case?

#

oh wait, the "other" for the Diff can be a host directory?

frozen sedge
#

yeah, I think that should work

#

I think you'd need something like this?:

src := client.host().directory("/foo")
ctr := client.container.from("foo").withDirectory("/src", src).withExec(["gazelle"])
diff := ctr.directory("/src").diff(src)
diff.export("/foo")
broken cypress
#

Lemme experiment and report back tomorrow. Thanks, that's a great idea!

frozen sedge
#

np - curious how it goes! 😁

broken cypress
#

ugh, ever seen a bug where cache volumes don't mount?

haughty panther
broken cypress
#

yep, I have one now ... is there anything you would like me to run to get debugging info?

haughty panther
#

yep, if you run mount in your container you should see your cachemount in your target dir

broken cypress
frozen sedge
#

I've had that exact issue before with /nix, and eventually figured out why. Let me see if I wrote a helpful commit message

#

Also do you have a snippet of code that shows how that mount + container is set up?

broken cypress
frozen sedge
#

ah - I think this is because of the cache sharing mode

broken cypress
#

haha, I made a random choice of sharing mode 🙂

frozen sedge
#

lol - they are a bit confusing so I don't blame you

broken cypress
#

switching the cache volume mount type didn't seem to have any effect, so I'll try blowing away the cache. That will take some time 😂

haughty panther
#

Mount caches generally come from the device directly

#

Could it be that somewhere in your pipeline you're overlaying /nix on top of the cache volume?

#

overlay in X is generally an outcome of WithMountedDirectory

#

that's what it outputs in linux. Can you validate by running this please? Since you're running in a mac

package main

import (
    "context"
    "log"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        log.Fatalln(err)
    }
    defer func(client *dagger.Client) {
        err := client.Close()
        if err != nil {
            log.Println(err)
        }
    }(client)

    vol := client.CacheVolume("myvol")
    client.Container().From("alpine").
        WithMountedCache("/cache", vol).
        WithExec([]string{"mount"}).Sync(ctx)
}
broken cypress
#

I posted code in the previous gist; perhaps it's due to the mount options?

        WithMountedCache("/nix", nixCache, dagger.ContainerWithMountedCacheOpts{
            Sharing: dagger.Private,
            Owner:   "docker",
        }).
haughty panther
#

docs say this:

     // a user:group to set for the mounted cache directory.
    //
    // note that this changes the ownership of the specified mount along with the
    // initial filesystem provided by source (if any). it does not have any effect
    // if/when the cache has already been created.
    //
    // the user and group can either be an id (1000:1000) or a name (foo:bar).
    //
    // if the group is omitted, it defaults to the same as the user.
    owner string
broken cypress
#

Not sure that making a container user is actually doing me any good here, so I'll experiment with just removing all that.

haughty panther
#

could be that since your nix cache has been created previously it's just defaulting to an empty new volume

#

I recall coming across a confusing scenario regarding this before. We need to document this better

broken cypress
#

I haven't seen any problem after switching the cache so shared FWIW

#

thanks for your help @haughty panther 🏆

frozen sedge
#

how running Nix in Dagger feels at the moment

haters.gif
#

when it works it's great! but it's an intersection of a lot of complicated things 😛

haughty panther
broken cypress
haughty panther
#

I'd assume your next challenge will be to improve the mammoth file syncing situation. I'm wondering if playing with include and excludes could be a good enough approach to give your users a decent DX