Non-deterministic files being returned with relative paths

[fleet thorn]

I want to control the files that get returned while respecting their paths.

So essentially, if I run a function right now it returns the entire git directory, but this runs the risk of overwriting content I won't approve.

If I'm not using "--wipe", how do I provide back just the files matching my expression, like yaml/markdown files for example and still make sure i can export --path . but keep the paths they had in the directory internally?

I tested and with one mistake overwrite the git directory (whoops!). I want to protect from any files I didn't want touched, like source files, and just allow docs to be updated, but since file names are not determined till run, I need to control the paths/types and then still output to root of the repo with all the paths maintained.

Any examples?

Not sure of the right magic to perform here

[fleet thorn]

what about this? Is filtering results back and still returning the directory instead of a file good? This would ensure never touching anything I didn't opt into, and never touching the git directory right?

The removal of ignore=[.git] mention I think was the culprit meaning it zero'd out my git and ruined that clone.

Is there anything else I need to do to ensure NEVER to negatively overwrite files while exporting back directory and not a single file?
cc @gritty jacinth in case you get a quick answer on this one, maybe you can make sure I'm safe.

First run I did ruined a repo cause I mistakenly had git directory zeroing out files 😂

[fleet thorn]

can someone from <@&946480760016207902> just confirm if this is the "safe" way? Really embarrassing to wipe out someones .git on my first remote run after all my testing

Want to better control my outputs and so far my changes to use FilteredOuput seems good as I have been testing, just making sure this is right?

The key is

• protecting what comes back
• not having to know exact files
• ensuring they get exported out to the current project in the correct relative paths.

if this is confusing I can jump on quick huddle.
Thanks!

that's the approach i took after having agent mode dig for patterns most used

[gritty jacinth]

happy to huddle! but taking a look

[fleet thorn]

yeah if you don't mind maybe you can set me right if i'm doing something wrong 🙂

i'll try and make it quick

[gritty jacinth]

can jump in #911305510882513037

[dusky moss]

ahh yeah this is a big footgun at the moment. related issue, and a comment i posted on the other day proposing a solution: https://github.com/dagger/dagger/issues/8235#issuecomment-3152011433

tl;dr I think we could support returning a special type, Env (which has a Directory) and when the CLI gets one back, it 1. shows you a "diff" (what files were modified, removed, added), and 2. prompts to apply it

that way we wouldn't need --wipe and you'll always have an idea of what will be modified before applying it

[fleet thorn]

@gritty jacinth and I chatted about this...

Ran into an issue with ECI mode in Docker (https://docs.docker.com/enterprise/security/hardened-desktop/enhanced-container-isolation/) . Might have impacted this but still probably an issue to address.
Checking if this is enabled should probably be done and confirming if not supported.

When the source directory gets mounted and I don’t ignore .git, Docker copies back all contents, even if there’s no file changes. During this, files in the Git directory get zeroed out for a moment. If something interrupts that process, or somehow the Git contents change, you’re left with zeroed out files. This happened on Docker Desktop for Windows with ECI for the first run for the user.

If I control the mount explicitly, it’s expected, but default behavior shouldn’t impact your local Git repo unless you opt in.

Thoughts:

• Right now you have to deliberately opt out by ignoring .git, and that seems backwards. Default of --source . means git being copied in could be modified instead of opting into it. Maybe the best behavior, but maybe not?
• ECI not checked, guessing would be good to stop if needed.
• The fact it's zeroing out files means it could be interrupted/fail and at that stage you have corrupted the entire git or impacted files. Seems dangerous, so maybe there's something more graceful about file update operation that could happen?

OK BUT my main issue to confirm right now .....
Is the best way for me to guarantee the exact files to be impacted and protect everything else to do this?

[vernal terrace]

hey Sheldon! yes, this is currently the best way to handle this

[fleet thorn]

be good to know too if anyone has looked at enhance container isolation when y'all get a chance. telling folks right now to disable, but it breaking a dagger run and messing up git IF git was exported back out or changes is an edge case I'd definitely want to protect against.

[fleet thorn]

trying to make sure i use github more than discord for this now so it's traceable and useful in mcp/agent tooling lookup/debugging.

https://github.com/dagger/dagger/issues/10864

If someone can give me a basic confirmation this is likely a breaking issue would be great!

[fleet thorn]

<@&946480760016207902> fyi. would be good to get clarity on this or ack the issue since it's more a system requirements confirmation/graceful fail

[dusk heath]

do you have an error that you're hitting when running with this? the bug report is a bit vague, if you can share a reproduction example, or even just the snippet of error, that would help 🙏

supposedly the feature supports runing docker-in-docker setups, so i would imagine it should work? but also, i really have no experience with this feature

the docs page is annoyingly quite light on the details of specifically how it's performing additional isolation (and it's proprietary?, so I can't poke around), so it's difficult to say whether it's expected to work

[fleet thorn]

The walk through I did was an edge case, but I suspect a combination of ECI + my code mistakes. I don't have a way to reproduce which is why it's vague. Just determining the support or not.

The big issue is really just determining does it need to be disabled.
I suspect so as giving access in one container to do docker in docker like you mentioned or networking is probably a no go. I just think that if it's possible to determine this a dagger invocation time to avoid even running and dealing with unpredicticable results would be a nice safety net.

[dusk heath]

I don't think eci is the cause of the zeroing thing

if that's specifically the issue

I wouldn't go as far as to say we have "official" support for it, but it doesn't sound like the git dir issue that's in this thread is going to be affected by it

[fleet thorn]

i'm having dagger run a container so figured that might be a problem with container to container , something like that.

[dusk heath]

potentially maybe? if it does seem to work, it probably will keep working 🤔
it's just probably not the cause of the overwriting issue you're seeing