dagger call <...> with multiple directory exports? | Dagger | Page 1

tough comet Mar 27, 2025, 5:03 PM

#

Hi.
So I have a usecase -- build a codebase and run tests in a preassembled container and collect resulting artifacts. Normally I would be either setting up my workstation by installing all the required tools locally, OR do docker run -v .:/src/ <...> and run my commands there, so that all my created artifacts would automagically be persisted in my cwd.

Thing is, build/test commands produce several output directories: ./target/, ./build/ . I need them both. Based on my previous experience with dagger, dagger does not reference/link/mount directories into containers -- instead it copies them inside. And then I have to explicitly export them at the end of the pipeline.

For some reason my directory --path /src/target/ --output ./target/ directory --path ../build/ --output ./build2/ directory --path ../.m2 --output ./.m2 stopped working for the 'build' directory (it still exports 'target' and '.m2' though...). Then I tried specifying full paths inside the container (i.e. /src/target, /src/build, etc.), but this, to my surprise, didn't work as expected -- even though I specified full paths, they still were treated as relative ones... And appending on top of /src/target (/src/target/src/build/).

So I figured I'll upgrade 0.14→0.17.... With no luck. I still can't get ./build/ exported.

I also tried directory --path /src/target/ export --path ./target/ , but I can't find a way to chain multiple directory exports this way.

Then I tried with-mounted-directory --path /src --source ./ hoping it will mount my src directory just like docker does, but... still, no artifacts appearing in ./build/. And I can't find docs about this function.

Help, please 🙂

#

Full command:

dagger --progress=plain -m github.com/dagger/dagger/modules/wolfi call container                 from --address maven:3.9.6-eclipse-temurin-17-focal                 with-directory --path /src --directory ./     with-workdir --path /src           with-env-variable --name c --value ,      with-exec --args sh --args -xc --args 'mvn clean verify -Dmaven.repo.local=./.m2 -Duser.home= -s devops/maven/ci-settings.xml -Dspring.profiles.include=core${c}base${c}override${c}prprd -Dspring.profiles.active=prprd' with-exec --args ls,-la with-exec --args ls,-la,build/               directory --path /src/target/ export --path ./target/ directory --path /src/build/ export --path ./build2/ directory --path /src/.m2 export --path ./.m2

frosty lichen Mar 27, 2025, 5:15 PM

#

tough comet Full command: ``` dagger --progress=plain -m github.com/dagger/dagger/modules/wo...

Hello! I'm looking into it.

My first question is: are you sure this is how you want to use the wolfi module? In this example, you get a wolfi container, then immediately reset its state with from and download an unrelated image. So you don't really need the wolfi module

#

@tough comet this should work using the new dagger shell feature:

#!/usr/bin/env dagger

# Configure the container
ctr=$(
  container |
  from maven:3.9.6-eclipse-temurin-17-focal |
  with-directory /src . |
  with-workdir /src |
  with-env-variable c , |
  with-exec --expand -- mvn clean verify -Dmaven.repo.local=./.m2 -Duser.home= -s devops/maven/ci-settings.xml '-Dspring.profiles.include=core${c}base${c}override${c}prprd' -Dspring.profiles.active=prprd |
  with-exec -- ls -la |
  with-exec -- ls -la build/
)

# First export
$ctr | directory /src/target/ | export ./target/
# Second export
$ctr | directory /src/build/ | export ./build2/
# Third export
$ctr | directory /src/.m2 | export ./.m2

tough comet Mar 28, 2025, 8:49 AM

#

Thank you @frosty lichen . No, I'm not sure this is how I want to use the wolfi module 🙂 It was a result of a monkey approach -- do random things until you see results close to what you want 🙂 Tanks for pointing it out.

Is dagger shell the best-way-of-using dagger CLI you are curently promoting? Or is it just what fits my use case the best?

I couldn't find docs through a google search for 'with-mounted-directory'. Could you point me to the right direction?

I'm guessing I no longer need to work-around the CSV issue with ${c} (I need the command args to contain commas) any more with v0.17 and this shell approach?

Also, I see you are using variables and evaluations in your dagger script. However, I cannot find them in the docs https://docs.dagger.io/features/shell . I cant find syntax (variables, $() evaluation, etc.) usage or examples nor anything else apart from piping commands. Can you point me in the right direction? I'd like to learn more.

P.S. while looking for shell docs I stumbled upon llm. HATS OFF!!!! This is super cool!!

Interactive Shell | Dagger

Iterate faster with familiar Bash-like syntax and autocomplete

ember tiger Mar 28, 2025, 2:28 PM

#

@tough comet there is an example of using variables in the blog post (multi stage build) at https://dagger.io/blog/dagger-shell... But you're right that we should also have that in the docs, I'll open a PR to add it asap

tough comet Mar 28, 2025, 2:31 PM

#

@frosty lichen do dagger scripts have an ability to accept command-line arguments? I can't find this in docs.

Also, since you used shebang in the script, it is assumed that the script should be launchable by ./run_tests.dagger . But it's not... or maybe I'm doing it wrong.

$ dagger version
dagger v0.17.2 (docker-image://registry.dagger.io/engine:v0.17.2) linux/amd64
↑16:14:44 [0]
$ ./run_tests.dagger 
Error: unknown command "./run_tests.dagger" for "dagger"

The shell redirection works though

dagger --progress plain <run_tests.dagger

but now my tests fail to complete -- ClassNotFoundException. They worked with the wolfi approach. I'll try to find out why

tough comet Mar 28, 2025, 2:38 PM

#

ember tiger <@307609644245712896> there is an example of using variables in the blog post (m...

Thank you.

While at it, what other features do dagger scripts have? Are comments, variables and pipes all of the current syntax elements?

Should dagger scripts work with shebang? Should shebanged scripts accept additional dagger parameters (e.g. #!/usr/bin/env dagger --progress plain)? Because I'm given a help page if I try that.

Do dagger scripts accept command-line arguments?

tough comet Mar 28, 2025, 3:29 PM

#

What's that mean...?

Error: tried to access non-existent state "R2MDGQFTJ74TAAXBA26SBGHZXF"

#

↑↑ got this when trying to run @frosty lichen 's script

frosty lichen Mar 28, 2025, 3:36 PM

#

tough comet What's that mean...? ``` Error: tried to access non-existent state "R2MDGQFTJ74...

I've never seen this request before, maybe a maven error? or otel-related @lyric heart ?

#

That error message isn't in the dagger source code that I can find

lyric heart Mar 28, 2025, 3:37 PM

#

doesn't seem otel related thinkspin don't recognize it either

tough comet Mar 28, 2025, 3:37 PM

#

I can see it in dagger's sources (in a way?)

https://www.google.com/search?q="tried+to+access+non-existent+state"&sca_esv=2e5b74018e0ac27e&sxsrf=AHTn8zpbCXN1puVNp1SQOp1ZhLVu7RDsww%3A1743175802795&ei=esDmZ82mMI-LwPAP8vvi0AE&ved=0ahUKEwjNy4b0i62MAxWPBRAIHfK9GBoQ4dUDCBA&uact=5&oq="tried+to+access+non-existent+state"&gs_lp=Egxnd3Mtd2l6LXNlcnAiJCJ0cmllZCB0byBhY2Nlc3Mgbm9uLWV4aXN0ZW50IHN0YXRlIjIFEAAY7wUyBRAAGO8FMgUQABjvBTIFEAAY7wUyCBAAGIAEGKIESIoLUKkCWJcIcAF4AZABAJgBf6AB0gGqAQMxLjG4AQPIAQD4AQGYAgOgAtwBwgIKEAAYsAMY1gQYR8ICBRAhGKABmAMAiAYBkAYEkgcDMi4xoAe8CbIHAzEuMbgH2AE&sclient=gws-wiz-serp

lyric heart Mar 28, 2025, 3:37 PM

#

ah yep: https://github.com/dagger/dagger/blob/903471d9232f6e1c8bfa027af93139bd20d06a82/cmd/dagger/shell_state.go#L117

#

cc @tawdry wren

tawdry wren Mar 28, 2025, 3:40 PM

#

tough comet What's that mean...? ``` Error: tried to access non-existent state "R2MDGQFTJ74...

Ah, good to know. How can I reproduce it?

tough comet Mar 28, 2025, 3:42 PM

#

I was running this

#!/home/<my_user>/.local/bin/dagger

# Configure the container
ctr=$(
  container |
  from maven:3.9.6-eclipse-temurin-17-focal |
  with-directory /src . |
  with-workdir /src |
  with-env-variable c , |
  with-exec -- du -sh .* * |
  with-exec --expand -- mvn clean verify -Dmaven.repo.local=./.m2 -Duser.home= -s devops/maven/ci-settings.xml '-Dspring.profiles.include=core${c}base${c}override${c}prprd' -Dspring.profiles.active=prprd |
  with-exec -- ls -la |
  with-exec -- ls -la build/
)

#$ctr | terminal
# First export
$ctr | directory /src/target/ | export ./target/
# Second export
$ctr | directory /src/build/ | export ./build2/
# Third export
$ctr | directory /src/.m2 | export ./.m2

IDK what triggered it, but since it printed the .../target/ directory right before the error, I'm guessing it might have smth to do with it.

And ./target should exist in both container and host, ar I'm doing with-directory /src ., and it's already present on the host.
IDK ow to repro it tho..

tawdry wren Mar 28, 2025, 3:43 PM

#

Let me check

tough comet Mar 28, 2025, 3:43 PM

#

reuploading the pic (missed one spot when masking data)

#

IDK if it's relevant, but the mvn command was failing (non-zero exit code). And for some reason the with-exec were not run either. Could it be that a container with non-zero exit code prevented further pipeline execution and failed to assign the result to a variable?

#

↑↑ UPDATE unlikely: I've just ran the same script with a passing maven goal and progress=plain, subsequent with-exec also passed, and I still got the same error

#

@tawdry wren seems like it's failing on a second directory export. If there's just one dir export -- it works. If there are second or more -- it fails

#!/home/<my_user>/.local/bin/dagger

# Configure the container
ctr=$(
  container |
  from maven:3.9.6-eclipse-temurin-17-focal |
  with-directory /src . |
  with-workdir /src |
  with-exec -- mkdir -p /src/target /src/build /src/.m2 |
  with-exec -- ls -la
)

# $ctr | terminal
# First export
$ctr | directory /src/target/ | export ./target/
# Second export
# $ctr | directory /src/build/ | export ./build2/
# Third export
# $ctr | directory /src/.m2 | export ./.m2

#

just like it fails for me if I use CLI approach (OP)

tawdry wren Mar 28, 2025, 5:20 PM

#

tough comet <@768585883120173076> seems like it's failing on a second directory export. If ...

It's not the export per se, the problem is using the same $ctr in multiple lines. Here's a smaller repro:

dir=$(directory)
$dir | entries
$dir | name

This only happens when running in non-interactive mode. I'll fix it.

frosty lichen Mar 29, 2025, 3:29 AM

#

thank you!

tough comet Mar 29, 2025, 5:24 AM

#

@tawdry wren what is a non-interactive mode? Is it progress=plain? If so, I first got this error w/o 'plain' -- see screenshot

tawdry wren Mar 29, 2025, 6:06 PM

#

tough comet <@768585883120173076> what is a non-interactive mode? Is it progress=plain? If s...

No, see Modes of execution in the shell feature doc.

#dagger call <...> with multiple directory exports?