Why does DefaultPath trigger uploading the directory the file is in?

[mossy orbit]

I found that using DefaultPath causes the whole folder the file is in to be uploaded. This is unexpected and, in my opinion, a bug.

Here is a demonstration. Screenshots demonstrating the expected behavior, without DefaultPath, and unexpected behavior, with it:

The expected and desired behavior, without DefaultPath:

[light edge]

if you use dagger.File instead of dagger.Directory this works as expected if you just want to specify a single file

having said that, I agree the UX is confusing when specifying a file when using a Directory. If you have the time to open an issue that'd be much appreciated. Otherwise, I'll try to open that later today 🙏

[mossy orbit]

I did use a File, didn't I?

[light edge]

oh, I see what you mean. I misinterpreted the issue

let me check

@mossy orbit k, I see what you're seeing now

the upload happens as part of the module initialization. You can filter that as mentioned here: https://docs.dagger.io/configuration/modules/#file-and-directory-filters

[mossy orbit]

but it does not happen without the Annotated[DefaultPath], plus Ignore filters don't work with Files (I tried). Instead of this what should I do?

file: Annotated[File, DefaultPath("big_folder/README.txt")]

[light edge]

let me check the repro, that'd be easier

@mossy orbit so, I see the whole folder gets uploaded in your example regardless of which function I use ^

that's part of the behavior I was mentioning before

if you check the include pattern that uploads everything in the project directory

as part of the module initialization phase

if I add an exclude property in the dagger.json that gets me what I need

  "name": "dagger-defaultpath",
  "engineVersion": "v0.18.16",
  "sdk": {
    "source": "python"
  },
  "exclude": [
    "big_folder/large_file.txt"
  ]
}

in my case I created a large_file.txt file inside the big_folder path

and dagger ignores that file basically

[mossy orbit]

In my case I don't want to ignore it; I want to select ONLY it, without pulling in its directory. I think anybody would expect that when using File.

[light edge]

yes, I get that. It's not your module that's causing the issue. The current behavior from the module initialization phase when you have a module at the top of the repo (not source folder) is to upload everything to the runtime container. That's why you're seeing slow initialization times for that.

when you do have a source folder, we only upload that as part of your module init

if you do dagger init --sdk python --soruce .dagger, that should avoid uploading unnecessary stuff

[light edge]

can't remember if this is in the docs. cc @cerulean oyster ?

[mossy orbit]

dagger is actually in a subdirectory ci/ in the real case. Let me try moving it in the example.

[light edge]

that's fine. As long as it's not part of the .dagger folder when you do have a source, that should be ok

[mossy orbit]

Moving dagger to ~/ci, away from ~/big_folder, did not change anything. Did I misunderstand you?

[cerulean oyster]

this looks right, i think theres a bug with DefaultPath, trying to have a look

yeah @light edge the engine is doing a Directory().File() for DefaultPaths on files https://github.com/dagger/dagger/blob/main/core/modfunc.go#L644

so basically the desired behavior is instead of Host.Directory(dir).File(file) we should either do Host.File(file) or if we absolutely must load the directory first, Host.Directory(dir, include:file).File(file)

[light edge]

you're right Kyle. I got lost in the module initialization phase which was also uploading the whole dir

and never got to validate the e2e function call 🙏