Is there any advice for working with

[true widget]

Example:

✔ parsing command line arguments 1m4s
├╴✔ address(value: "./result/"): Address! = xxh3:cc587315802f4899 0.0s
├╴✔ .directory: Directory! = xxh3:1f267fe5498b0e28 1m4s
│ ╰╴✔ Host.directory(path: "./result/"): Directory! = xxh3:c51c3f71a200d1f0 1m4s
│   ╰╴✔ filesync 1m4s
│     ├╴✔ bzImage 3.7s ◆ Written Bytes: 64 MB
│     ├╴✔ initrd 3.1s ◆ Written Bytes: 24 MB
│     ├╴✔ cmdline 0.0s ◆ Written Bytes: 216 B
│     ├╴✔ init 0.0s ◆ Written Bytes: 102 B
│     ├╴✔ stereos.qcow2 31.4s ◆ Written Bytes: 2.7 GB
│     ├╴✔ mixtape.toml 0.1s ◆ Written Bytes: 869 B
│     ├╴✔ stereos.img 6m3s ◆ Written Bytes: 8.3 GB
│     ╰╴✔ copy 1.5s 

I've tried using WithMountedDirectory but seems that when I pass a dagger.Directory as an argument to a function, it does this filesync - is there not a way to "just use host file system" without shoving everything into the dagger context?

Is the problem that I'm trying to pass through a *dagger.Directory? Do I need a With... that mounts the dir by path vs. passing it in?

https://github.com/papercomputeco/daggerverse/blob/c36e73f623913a4f11383019ee13525356196e9f/bucketupload/main.go#L160-L174

[tardy compass]

@true widget if you want to process files from your host filesystem, they have to be ingested. But you can filter them pre-ingestion. Do you actually want those 10GB of files processed by dagger?

[true widget]

Yes! I want to use Dagger glue with Go to build OCI registries with google/go-containerregistry and not have to do it by hand.

[tardy compass]

Can you explain in a little more detail? So the files are oci images? Are they committed in your repo?

[true widget]

Very very TLDR: I'm backdooring some huge files into an OCI registry for clients to be able to pull latter (think ollama pull - those model layers are absolutely huge so I'm building the registry by hand and pushing it to a bucket).

In the end, I need to calculate the sha256 hashes, build the manifest, build the index, etc. That's pretty easy with Go using google/go-contaienrregistry and the types it provides. And I want to do it in CI, so, in the end, I have these large blobs and files that need to get aws s3 up to a bucket and I'd much rather use the conventions dagger call provides and:

awsCli := dag.Container().
    From("amazon/aws-cli:latest").

vs. try to do it by hand in some yamls in CI.

My mental model of how I expected this to work was Bind Mounts in Docker but seems that the dagger engine expects to slurp all those up which may be a non-starter.

[tardy compass]

yeah the problem with bind mounts is that they only work locally. So it breaks native remote execution. Case in point, docker remote execution is broken and always has been.

@true widget where do the files come from? git?

[true widget]

From a nix build output.

I know I can do this with Nix, I just would rather write Go 😂

and getting the types from google/go-containerregistry is basically what I need

[tardy compass]

ah. well you can run the nix build in dagger

or just get rid of nix and build natively with dagger - but maybe a conversation for another time 😛

[true widget]

ah. well you can run the nix build in dagger

That'll never happen. Dagger's not a package manager.

[tardy compass]

I mean execute nix itself inside dagger

[true widget]

Or, I suppose I should say, for what we need Nix for, won't happen

I mean execute nix itself inside dagger

I'll have to try that: I assumed the build and /nix/store would be too huge for the container

[tardy compass]

you should definitely keep that in a cache volume

same for any other build/packaging tool: go, npm, bazel.

[true widget]

👀 👀 👀 very very interesting.

Here's the question: can the dagger engine (which I suppose is just the contaienr runtime) use the whole CPU/RAM? If I did that, I'd want the NixOS builders we have to fully chug

[tardy compass]

sure. I don't anticipate any issues there.