Dagger changes CWD silently and unintuitively

[viral furnace]

Why does dagger assume another CWD when running a script, and WHAT is the CWD it assumes?

E.g.

My repo has
./git/
./devops/scripts/
./extensions/db/sql/

I'm inside my repo's ./devops/scripts directory. And when I run a daggerscript in there, it assumes a repo root as a CWD (i.e. where .git/ is located). I know this because if in that directory (./devops/scripts/) I try to mount ./extensions/db/sql for liquibase execution, it succeeds. This tells me that when dagger runs my script it changes CWD to my repo's root.

Is this assumption correct?
It's VERY unintuitive. Not a single shell I've worked with does that. Why does dagger do it? Any way to disable it?

AND if I try to mount it with ../../extensions/<...> (i.e. explicitly stepping out of the ./devops/scroipts/) , it fails, because it exits my repo root (into a grandparent directory).

[mellow marsh]

👋 by running a "daggerscript" do you mean dagger call? or just the dagger shell?

[viral furnace]

dagger shell

[viral furnace]

#!/usr/bin/env -S dagger --progress=plain

this kind of shell

[mellow marsh]

ok, in this case you can use $(host | directory .) to get the cwd. That should work

[viral furnace]

Is this intended behaviour?

Where does it cd to if I don't do host|directory ? I.e. how low down the directory tree will it go?

[wintry basin]

Could you share the exact script, with intended vs. actual behavior?

[viral furnace]

@wintry basin

There.

So here's the scenario. In ./devops/scripts/ I store my scripts. If I cd into that dir and execute my dagger script, I expect it to respect my CWD for relative paths.

My script mounts a directory onto a container: with-directory "${WORKDIR}" "./../../extensions/database/ext-data/src/main/resources/schema" |. So if my CWD is ./devops/scripts/ (with '.' being my repository's root, i.e. where .git/ is located) , I expect the ./../../extensions/<...> to be relative to my cwd, i.e. ./devops/scripts/.

Instead, for some reason, dagger calculates the ./../../extensions/<...> segment relative to my repo's root, and NOT my CWD, effectively escaping from my repository to its parent's-parent directory and trying to find ./extensions/ in there.

repro.dagger

#!/usr/bin/env -S dagger --progress=plain


WORKDIR="${WORKDIR:-/src}"

ctr=$(
    container |
        from alpine:latest |
        with-directory "${WORKDIR}" "./../../extensions/database/ext-data/src/main/resources/schema" |
        with-workdir "${WORKDIR}" |
        with-exec -- ls -l
)

$ctr | stdout
$ctr | stderr

20  : ModuleSource.localContextDirectoryPath DONE [0.0s]
20  : [0.0s] | /home/REDACTED/workspace/REDACTED/REDACTED-ep-commerce

21  : ModuleSource.sourceRootSubpath DONE [0.0s]
21  : [0.0s] | devops/scripts

So it seems like relative paths are calculated starting from localContextDirectoryPath, which for some reason points to my repo's root directory. Why....? I did not specify it anywhere, I'd expect dagger to look up my CWD and work its way from there.

[wintry basin]

mmm yeah that does seem like a bug to me 🤔

@rugged otter wdyt?

@viral furnace are you trying to access something outside of the git repo?

[viral furnace]

no. I'm trying to access resources inside my git repo, just another directory. Why?

[rugged otter]

What path do you get with .pwd?

[viral furnace]

<...>/devops/scripts

the actual and correct cwd

[rugged otter]

Just to make sure, you've added it to your script, with the . included, not pwd in bash?

[viral furnace]

yes

WORKDIR="${WORKDIR:-/src}"
.pwd
ctr=$(
    container |
<...>

[rugged otter]

From what I understand, if you're in <repo>/devops/scripts, the path ../../extensions should point to <repo>/extensions?

[viral furnace]

That's my expectation, yes

at least that's how shells do 🙂

[rugged otter]

Yeah, that's correct. What's the command that you're using to run the scripts? Are you just doing ./repro.dagger?

[viral furnace]

1. dagger --progress=plain < repro.dagger
2. ./repro.dagger

both behave equally

[rugged otter]

Got it. Let me try to repro this locally for a minute.

Btw, do you have a dagger.json on the repo root or in devops?

[viral furnace]

nope...

 ( while [ "${PWD}" != / ] ; do LC_ALL=C ls -l dagger.json || cd ..; done )
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
ls: cannot access 'dagger.json': No such file or directory
↑18:50:37 [0]

[rugged otter]

Ok, I was able to repro, will dig in to figure out what's happening. In the meantime I suggest you wrap the directory path in $(host | directory "..."), like this:

with-directory "${WORKDIR}" $(host | directory "./../../extensions/database/ext-data/src/main/resources/schema")

[viral furnace]

Thanks.

BTW, I haven't tested with other components, like with-file, but I'm assuming they will behave the same as with-directory. So while you're at it, please look into those too 🙂

[rugged otter]

It's the same thing. It's based on Directory and File argument flags, not specific to those API calls. The $(host) command cirvumvents that flag.

[wintry basin]

Thanks for looking into it @rugged otter !

[viral furnace]

P.S. @wintry basin @rugged otter I'm one of those annoying pricks who ALWAYS attract all the bugs others never run into in their lifetimes 🙂 Since I'm seriously considering using dagger extensively in my projects and perhaps eventually spreading it across other projects in our company, expect to hear more from me 😄

I already have a lengthy list of issues/inconsistencies noted down (collected them while reading docs; still halfway through, so not posting them yet) 😉
I'm not trying to be annoying. But I expect my tools to be reliable, intuitive and work as expected and I hope I'm helping the dagger project at least a bit by reporting all the issues I find instead of keeping them to myself and/or ditching dagger altogether 🙂

[rugged otter]

It is helpful, thanks for reporting! 🙏

[viral furnace]

hopefully one day I'll be proficient enough in golang to contribute with code too, not just bug reports 🙂

[rugged otter]

I have a fix, just need to add a test before pushing.

[mellow marsh]

@rugged otter my question to you yesterday was coming from here. Not sure what changed which required a fix since your reply was that the standard way of fetching the cwd files was via host | directory

[rugged otter]

I didn't have full context and thought . resolved to repo root because you .cd'ed into it. Both should work in this case.

If you start the shell in a subdir and don't change it . should point to that same subdir.

[mellow marsh]

right.. that was the intention with my question when I meant "not the CWD where the shell was started". Apologies for not being clear. 🙏

[rugged otter]

For example:

> dagger -m modules/alpine

In this case, . should point to /modules/alpine while $(host | directory .) should point to /.

Does it make sense?

[mellow marsh]

wait, but if I do cd modules/alpine && dagger then . is /modules/alpine ?

that's confusing

since . doesn't effectively represent the CWD

[rugged otter]

Yes, and $(host | directory .) should also point to /modules/alpine in that case.

Since your initial context matches the directory where you started dagger with.

When you use -m you change to that directory initially as well, but only within the shell.

[mellow marsh]

yeah I understand.. it's confusing that there's no consistent way to retrieve the CWD where the shell started using -m with relative paths

[rugged otter]

There is, but you need $(host) if what you want is to be relative to where you launched it.

Otherwise, simply . is relative to the workdir within shell, which you can change with .cd.

[mellow marsh]

gotcha.. it's confusing that we have multiple path configurations:

• git repo root
• shell workdir
• dagger cwd
• module root

[rugged otter]

Notice that it also works within a git remote:

> dagger -m github.com/dagger/dagger/modules/alpine@v0.18.3

Now:

• . is relative to /modules/alpine in that git repo, at tag v0.18.3
• $(host | directory .) is still relative to your local host's files where you launched dagger

[rugged otter]

https://github.com/dagger/dagger/pull/10169

[mellow marsh]

✅ . seems to be complaining about some linting rules 🙏

[rugged otter]

Fixed 👍

[mellow marsh]

🚢

[viral furnace]

@rugged otter since there are several points of reference in dagger, would it maybe make sense to export them as env variables? We have ${PWD] for the current cwd, maybe some other cwd-ilike envs with clear meaning would help to distinguish what's where in which scope? Just a thought. I know bash has several envs that it updates after each command/event (PWD, COLUMNS, LINES, etc.). Smth like GIT_ROOT, MODULE_ROOT, etc.

And a way to print those envs in shell would help greatly with troubleshooting too! I see there's no env / .env yet.

[rugged otter]

Yeah, maybe. It depends, but we can bikeshed (in an issue). For example $PWD is updated by interpreter built-in _cd. We have more control with commands anyway, so I suggest sticking to $(.pwd) in this case. If you need to debug you can use dagger --debug and it will show more information. As for a .env command, all the host's env vars would show up. But I'm ok with adding a CONTEXT_ROOT, MODULE_ROOT, and MODULE_SOURCE.