Does Container.WithDirectory("", Container.Directory("")) allocate two copies of the files on disk?

[desert bone]

If I do Container.From("foo").WithDirectory("x", Container.From("bar").Directory("y")), and the bar image has 100GiB inside of its y folder, will the cache of the container produced by the query, and the cache of bar (and the cache of .Directory("y")) be backed by the same 100GiB of data on disk?

[bold cedar]

Yes I believe so. But might be worth double-checking given the size

[desert bone]

Given your lack of confidence, is it correct to assume this is an implementation detail of BuildKit?

I was also wondering if applying include/exclude filters change the answer. Docker certainly doesn't seem to dedupe things at the file level, but rather does so at the layer level.

[bold cedar]

well I'm not a core engine dev hence my lack of confidence 😁 but yes it's a buildkit implem detail.

I'm at 99% confidence

[narrow osprey]

@desert bone it will depend on some caveats on the WithDirectory call . More info about that in this discussion: https://github.com/dagger/dagger/discussions/9345#discussioncomment-11788963

The summary is that Dagger will try to use MergeOp when possible which will in consequence re-use the same underlying fs references when calling WithDirectory. However, there are cases where MergeOp can't be used (more info in the link above) and that will generate a duplication of the data

[desert bone]

Thanks, this is very useful. Claude claims that BuildKit de-duplicates at the file level, but I can't find any supporting documentation for that on the internet.

    if input.DestFileName == "" && input.SrcFileName != "" {
        input.DestFileName = input.SrcFileName
    }

Sweet Jesus, I could have been doing WithFile("foo/", other.File("bar")) instead of WithFile("foo/bar", other.File("bar")) this whole time.