Does dagger.Directory copy its contents?

[dreamy island]

... or just a pointer ?

[vernal salmon]

@dreamy island can you give an example? If you mean dag.Directory() that creates an empty directory, so nothing to copy

[dreamy island]

Does dagger.Directory copy its contents?

I was sloppy in my title. I meant a dagger.Directory passed from the command line to be used as an argument to .with_directory

[mighty iron]

nvm.. it "copies" files into the new container (edit)

dagger with try to optimize it depending on the src and dst locations so it uses some low-level overlay feature where it merges the different layers instead of performing a read/write syscall

that optimization will be applied transparently when possible

having said that, with_mounted_directory doesn't do a copy. So if you don't need the files permanently in the dst container, you can use this .

[dreamy island]

when does it start copying, as soon as it sees the dagger.Directory() parameter, or lazily? For example, does it respect the exclude parameter in with_directory or does it try to copy everything before it executes that statement?

[mighty iron]

yes, it'll honor the exclude in the with_directory

if you pass the src from the CLI here's what happens:

1. All files get uploaded from the host to the initial src directory
2. when you call ctr.with_directory($src)afterwards, a second "copy" will be done into $ctr

same thing that happens when using Dockerifles basically

the only thing we do differently is those extra optimizations which they need to be manually set explicitly in Dockerfiles

[dreamy island]

1. seems problematic. If my module has a Directory property; e.g., the Directory of the frontend, would it try to copy the whole thing, with its node modules and deps into it?

[mighty iron]

you can exclude that from your function definition

so it never gets uploaded in the firs place: https://docs.dagger.io/api/filters/#directory-arguments

[dreamy island]

let's say I defined

@object_type
class Frontend:
    frontend_dir: Directory
...

and frontend_dir gets used after container.with_directory(image, self.frontend_dir, exclude=['node_modules', 'ci/', 'build/'])

Did the whole dir get copied before this command?

[mighty iron]

you can set the exclude level at the frontend_dir

so it never gets uploaded to the build context in the first place. Check the pre-call filters in the above article

[mighty iron]

it's analogous to the .dockerignorething if you've used that before

[dreamy island]

Good thing I asked 🙂 Off to learn my lesson...

[mighty iron]

we're here if you need any help.

[dreamy island]

Shouldn't we see evidence of the pre-call filter in the cloud trace? I defined a class property dir: Annotated[Directory, Ignore(IGNORE_LIST)] but the trace merely says Host.directory(path:"foo"), and the copy operation still takes long. IGNORE_LIST=['node_modules/', 'build/', 'package/'] and these directories are right under dir.

[mighty iron]

Have a trace to share?

[dreamy island]

False alarm; I found it! I then created a terminal into the Container to confirm its contents.

[mighty iron]

Usually the ignore patterns should appear somewhere in the trace. Ok mobile now. Will check later 🙏

[mighty iron]

🚀 glad that worked

I was about to mention that the exclude patterns should appear in the Host step like this

which they weren't showing in the trace you sent me

[dreamy island]

In my case they were in the module step, since the Directories were properties of it.

[mighty iron]

yes but that means that the node_modules will still be uploaded to the build context

and you generally don't want that

[dreamy island]

why does it mean that, these directories are only in the Ignore annotation, not in exclude arguments.

[mighty iron]

^ that Host step should have the exclude clause

you can add the annotation at the top level function. In your case, the fresnel constructor / function call should have the excludes

otherwise, the node_modules folder will still be uploaded from your host machine to the build context

[dreamy island]

am I not already doing that when I define this?

@object_type 
class Frontend:
  directory: Annotated[Directory, Ignore(IGNORE_LIST)]

or must it be in the fresnel class that composes the frontend and backend? I think that is what you are saying. That would seem to invert the composition 😐

[mighty iron]

How is the Frontend class created?

It's clearly not picking that Annotation

[dreamy island]

Why do you say that? the frontend directory has the exclusion list:

host.directory(path:"/Users/emre/code/fresnel/client/web"), exclude:["node_modules/", "package/", "build/", "storybook-static/"], path: "/")

I do not directly instantiate Frontend. I compose it into another class with the Backend.

[mighty iron]

The trace you've sent me calla the runBackend function

[dreamy island]

Is the build context apart distinct from the container? When I enter it from the terminal it respects the filter.

[dreamy island]

That's because the composed fresnel module can run any combination of the backend and frontend.

[mighty iron]

I see in your trace that fresnel gets instantiated with the 2 dirs and a file

Those dirs don't have any exclusions

I'd assume in your fresnel constructor you have those fields, right?

[dreamy island]

No, I do not. They are in the child classes; one dir respectively. I can of course access them in the parent class.

[mighty iron]

I see . Well... that doesn't seem to be working. Since the exclusions are not appliedin the Host call

Can you try moving the web and monolith dirs to the parent class?

[dreamy island]

So the fact that the filtered directories are not in the container does not necessarily mean that they did not get copied to the build context ?

[mighty iron]

Correct. Build context != container.

Build context is whatever gets passed to your moduke

[dreamy island]

And I can't inspect it

[mighty iron]

Yes, you can. You can call .terminal() in a directory type

That will drop you into a shell to explore.