general

[chilly arch]

cc @modest pier @cloud canyon

[chilly arch]

Starting now! Join us here - https://dagger.io/deep-dives/agentic-ci

[modest pier]

Come hang out in #container-use - unfortunately right now we don’t have support for multiple checked in env configurations, so you should create one big image. In some ways that’s sometimes better in that your agent will be able to work on cross-language features without changing containers.

[mild coral]

Hi there. Apologies for my [spamming of this channel with the thread about dependency injection and interfaces](#general message).
The original issue is resolved now (worked around it)... I have new one when testing my demo project against the dagger main branch.

[topaz blade]

"In Dagger, each function is executed in isolation. You can't store state in a object and pass it to another dagger function"
Is this statement correct?

[winter linden]

I think it should be can

[mystic mantle]

Has anyone expressed interest in having fine grained authorization for Dagger Functions?
eg, only executing this function with some authorization token showing we have the right scope allows us to run it?

[warm temple]

I have the most basic version of that in some of my functions. basically there's an optional token argument, and if token was passed in, do these extra steps. Curious to hear what you're thinking of for fine grained authorization! https://github.com/kpenfound/greetings-api/blob/main/.dagger/main.go#L67-L70

[mystic mantle]

I'm evaluating a distributed CI system to try to integrate Dagger in it.

The way it works is:

• it has a task definition, which basically runs some command on a worker
• each task can potentially create more tasks
• and before each execution, they will check the scope of the task

As it is hard to handle those dependencies between tasks manually, there was a system that created a DAG of them.

So, the way the authorization works is:

• if a task is allowed to run another task
• if it is allowed to create this task on this specific set of runner (runner for mac os, or iOS, which have a different price point than runners for linux for example)
• or if it is a specific kind of task (e2e, regression, etc)

I was thinking of something like that, but by using context.Context values, but it's very Go specific, and I don't think it would be portable among other SDK.
So I like your idea, even if it's a little more heavy on the signature side.

[uncut karma]

Hello! Can anyone share a few repos using Dagger? I find the ones on https://docs.dagger.io/examples a bit too simple (I'm interested in setting up multiple environments for CI/CD and don't know what's the prettiest way of doing it, I don't want a huge main.go)

[winter linden]

If you want a wide sample, you can search Github for path:**/dagger.json. Any directory with a dagger.json is a dagger module

(I'm sure Gitlab a similar search you can use)

For a big and complicated monorepo, you can check out Dagger's own CI. Obviously we use Dagger to build Dagger 🙂

[cunning jolt]

I also utilize https://daggerverse.dev for inspiration/examples of dagger modules and usage.

[loud briar]

Whenever I need more examples or something to copy paste I use https://github.com/sagikazarmark. Mark is a Dagger Commander and has a lot of Dagger modules

[late kite]

Is it possible to list the names of current caches? I launched a terminal from an adhoc dagger pipeline on the shell, and can't remember what I named the cache, to resume the session 😅

[late kite]

How possible would it be to expose the name here: https://docs.dagger.io/api/reference/#definition-EngineCacheEntry ?

[late kite]

ah, poking at the bolt cache.db, it looks like cache's are keyed by a hash of their name, in which case, currently the original name wouldn't be available?

[stuck wyvern]

Really nice CLI improvements after this
Trace UI is amazing.

[winter linden]

Ha ha thanks! That's a nice chunk of improvements all at once 🙂

cc @elfin frigate 👆

[jagged flicker]

I'd love to get feedback on an idea I have in mind.

TL;DR: the ability to describe multiple "tasks", using dagger shell syntax, in a single file.

Longer description is I used to describe my builds and local tools using various tools, and recently using task (https://taskfile.dev). What's interesting here is I can describe my tasks and some variable in a single file, and then I can just run task <my-task>.
So I started to explore a way to do the same thing, using only Dagger.
Modules are super powerful, but the idea here is to be as light as possible (and in a real version it could use modules as dependencies).

So I created a small dagger.yaml (yes, yaml...) that exposes multiple tasks and a few variables. Each task body is a dagger shell command, and variables are also dagger shell commands.

From there, it's possible to dagger task <my-task> and it will run the corresponding task using the available variables.

A quick example to help understand:

vars:
  go_version: 1.24.4
  ctr: >-
    $(container |
    from golang:$go_version-alpine |
    with-exec go install golang.org/x/tools/cmd/goimports@latest |
    with-exec go install mvdan.cc/gofumpt@latest |
    with-workdir /go/src |
    with-mounted-file /go/src/go.mod go.mod |
    with-mounted-file /go/src/go.sum go.sum |
    with-mounted-cache /root/.cache $(cache-volume go-cache) |
    with-mounted-cache /go/pkg/mod $(cache-volume go-pkg-mod) |
    with-exec go mod download |
    with-mounted-directory /go/src .)

tasks:
  terminal: >
    $ctr | terminal

  go:test: >
    $ctr |
    with-exec -- go test -coverpkg=./... -coverprofile=.coverage -shuffle=on ./... |
    stdout

The var ctr is my base working container, that will be used by the different tasks. And then the go:test task for instance will run my tests inside this container.
I just have to dagger task go:test and that's it.

Let me know if that's something that could be interesting to investigate more 🙏

---

Try it: there's a draft PR here https://github.com/dagger/dagger/pull/10752 with a very minimal implementation and you can try it with this project and dagger file https://github.com/eunomie/docker-runx/blob/daggerfile/dagger.yaml

[winter linden]

1. As you know I love anything that helps reduce the initial complexity of adopting Dagger 🙂

2. How would I pass arguments to my tasks?

3. How do you feel about a variation where each task (script) would be in its own file? That way it's executable standalone, and easier to get syntax support in my editor

[jagged flicker]

I haven't thought about arguments. I think one way could be to define empty vars that are ovewritten on the CLI. Like dagger test go:test --go_version=1.24.5 This will only change the var value, but it doesn't add any other kind of interpolation/control/...

Regarding multi files: to be honnest I don't know. I can understand why, but I wonder if that doesn't break the idea of simplicity. And it probably means that each file will not be runnable itself, because of the variables. In my example, if I extract the body of the go:test in a file I need to be able to say it depends on ctr var.

That said, there's probably ways to solve that. For instance:

includes:
  - ./vars.yaml # could contain both vars and tasks sections

tasks:
  terminal: "$ctr | terminal"

  go:test:
    shell: ./go-test # meaning the go:test task is defined by the file 'go-test' that is a standalone dagger shell file

[late kite]

I've a nagging itch there's something in this direction too: this is a hacky pattern I have: literally cat a .dag file and then str append more '|''s https://github.com/cablehead/vibenv.dag/blob/main/vibenv/mod.nu#L57-L63

[late kite]

as i've time, i want to try creating a set of nushell commands you can pipe together, that generate the graphql to feed to dagger.. -- that'd make parameterizing things convenient

[late kite]

what's behind the itch is container | from alpine | terminal is just.. really really nice. so can we just stay in that world?

[winter linden]

The nice thing about dagger shell is that it already uses a bash-compatible parser, and that includes functions, arguments, variables etc. I feel like we could try and take advantage of that

Also, the library we use (sh by @radiant crane ) is very good at static analysis. So for example, we could easily analyse a script, and find all references to an uppercase variable for example. Then expose those as arguments to the dagger function

For type inference, maybe we need some sort of annotation, OR maybe we can infer the type from static analysis? 🙂

[jagged flicker]

yes, that was also one of my first ideas, to use it in shell mod only. And for instance to automatically expose all functions (or using comments or any other convention) as "tasks". Maybe that could be a better solution, by not adding one more layer on top

[late kite]

oh, I didn't appreciate that! I thought it was a custom (limited to just dagger commands) shell.. is there an example of defining functions or passing arguments to dagger's shell?

[winter linden]

I think it would be worth trying first. It's worth make the best possible effort to avoid wrapping shell in yaml IMO 🙂

(mm infering type from static analysis is maybe too ambitious. Maybe just a comment with the type name?)

We could also provide special funtions like directory_arg(), secret_arg but then it wouldn't 'just work' in dagger shell

@jagged flicker not 100% sure if it's relevant, but I'm actually in the process of porting the CLI's object loading to the API... Which means you can call it explicitly from a dagger shell 🙂 Maybe a bit too verbose but..

[vast mica]

I'm actually working on something similar to this with enact. https://enactprotocol.com You can execute commands in dagger from yaml. I haven't added the cool piping syntax yet.

[late kite]

root@a4320fa67962:/workspace/vibenv# dagger -c 'foo() { container; }  ; foo | from alpine'
▶ connect 0.0s
▶ detect module: . 0.0s
● loading type definitions 0.4s

$ container: Container! 0.0s CACHED
$ .from(address: "alpine"): Container! 0.0s CACHED

Container@xxh3:54d07fd66f1f211c

[late kite]

@winter linden this is incredible!

[winter linden]

#!/usr/bin/env dagger

go_version=1.24.4

ctr() {
  container |
  from golang:$go_version-alpine |
  with-exec go install golang.org/x/tools/cmd/goimports@latest |
  with-exec go install mvdan.cc/gofumpt@latest |
  with-workdir /go/src |
  with-mounted-file /go/src/go.mod go.mod |
  with-mounted-file /go/src/go.sum go.sum |
  with-mounted-cache /root/.cache $(cache-volume go-cache) |
  with-mounted-cache /go/pkg/mod $(cache-volume go-pkg-mod) |
  with-exec go mod download |
  with-mounted-directory /go/src .
}

terminal() {
  ctr | terminal
}

go_test() {
  ctr |
  with-exec -- go test -coverpkg=./... -coverprofile=.coverage -shuffle=on ./... |
  stdout
}

[late kite]

i totally missed this

[winter linden]

Well to be fair we don't document it

And also we don't take advantage of it in the DX (ie. we don't expose dagger shell functions as dagger functions etc).

But I'm taking advantage of @jagged flicker 's interest in the topic to bring it all back on the table 🙂

[late kite]

this replaces the need for my hack above at least

[winter linden]

What's missing is how to inject arguments 👆

[late kite]

ah right.. the shell doesn't get $1 etc?

[winter linden]

not at the moment. It's been requested often (including by me 🙂

We could easily add a basic $0 etc. but then that wouldn't map perfectly to dagger function arguments: no arg names, no arg types..

[jagged flicker]

that looks really close to the first version I had in mind. With a little bit of tooling on top (how to call go_test from the CLI?) and that could replace my yaml hack.
Very interesting, and it stays closer to the dagger builtin primitives

[loud briar]

Do env vars like $go_version-alpine here come from the host as if they'd been passed in as args?

[winter linden]

No, at the moment the shell interpreter is fully sandboxed. The only way to access the host is via the dagger API (eg. you can call host | file to get a file from the host, and so on)

We intentionally left it this way, to keep the door open to a clean design for arguments

[candid remnant]

Hello all, fresh dagger user here. Was trying to run the quickstart "Build a CI Pipeline" but faced some issues. Is this just me or are the docs not up to date? The docs state that a .dagger folder should be present but there is none. Also the publish task is failing on npm install.

Happy to help with updating the docs if needed

[winter linden]

It looks like maybe the quickstart is missing dagger init' ? You're supposed to run that command first, to initialize your dagger module. Normally it will create a dagger.json and '.dagger

[uncut karma]

Does dagger not respect entrypoints? docker run -v ./:/path ghcr.io/gitleaks/gitleaks:v8.23.1 detect --source="/path" --log-opts="HEAD~20..HEAD" -v works locally (the image has entrypoint gitleaks), however, I had to put it explicitly for it to work in Dagger

    gitleaksImage := dag.Container().
        From("ghcr.io/gitleaks/gitleaks:v8.23.1").
        WithMountedDirectory("/path", source).
        WithWorkdir("/path").
        WithExec([]string{
            "gitleaks",
            "detect",
            "--source=/path",
            "--log-opts=HEAD~20..HEAD",
            "-v",
        })

[sharp marsh]

it is there. Seems like a missed step in the tutorial @candid remnant 🙏 https://docs.dagger.io/quickstart/ci#initialize-a-dagger-module

[candid remnant]

Thanks for the help. I indeed missed a step but it was not the init, it was the git clone. When you do a dapper init on a clean directory it apparently doesn't create the .dapper folder. Thanks I can continue now

[topaz blade]

Are some docs or links which explains how dagger works internally?
How it caches stuff, the logic behind it. A async parallel working, etc

[placid magnet]

maybe someone could help
i want to integrate dagger on my company
we have a task to pull images from ECR and push them after retagging to our on premise harbor, we would have to trigger it once a week
do you think dagger will fit here?
i could do it pretty fast with skopeo or other tools on our jenkins, but i'm willing to give dagger a try : )
thanks

[cunning jolt]

I don't see a reason not. You can even use skopeo within dagger if that's a familiar workflow. You'd have to authenticate to both ECR and Harbor in dagger.

[chilly arch]

Happy Monday everyone!

Don't forget to mark your calendars for Dagger's episode with Containers from the Couch tomorrow with @knotty solar @warm temple @slender star !

Watch live here:
https://www.youtube.com/watch?v=S9BNa4Hn4-0&list=PLyHqb4A5ee1u5LrsbalfVkBRsrbjDsnN5

[uncut karma]

How to load image from tar?

[placid magnet]

would this be considred a good dagger approach ?

// Copy an image from one registry to another using skopeo
func (m *Mes) CopyWithAuthfile(
    ctx context.Context,
    srcRef string, 
    dstRef string, 
    authFile *dagger.Secret, 
) *dagger.Container {

    const mountPath = "/run/secrets/auth.json" 
    return dag.
        Container().
        From("quay.io/skopeo/stable:latest").
        WithMountedSecret(mountPath, authFile). 
        WithExec([]string{
            "skopeo", "copy",
            "--src-authfile=" + mountPath,
            "--dest-authfile=" + mountPath,
            srcRef, dstRef,
        })
    
}

[chilly arch]

https://tenor.com/view/coffee-overflow-pour-coffee-cup-pouring-coffee-gif-26440676

The time is now! Tune into the livestream with @knotty solar @slender star and @warm temple.

Fill up your coffee and join here: https://www.youtube.com/watch?v=S9BNa4Hn4-0&list=PLyHqb4A5ee1u5LrsbalfVkBRsrbjDsnN5

[tranquil tendon]

Hey everyone, does dagger have an eventing system for git repos? Looking to set up a CI server using it and I’m not too fond of the actions route

[winter linden]

At the moment Dagger provides the execution layer, but not the event system. Typically you continue using CI for that, but you gradually "gut" the legacy CI logic in favor of calls to dagger.

We're working on a solution for the event layer also... 🤫

Think of it as a first incremental step towards getting rid of actions 🙂

The other missing feature is scale-out. We also rely on CI for that. And we're also working on our own solution 😛

[tranquil tendon]

Awesome, I may just set up a small go server to intercept webhooks from gitea for now.

[winter linden]

Then you should talk to @sharp marsh and @rigid pebble about pocket CI 😛

[sharp marsh]

you might wanna check pocketci (https://github.com/franela/pocketci). It's "prototype" idea but it has several of the ideas and components you might need for that

[tranquil tendon]

Interesting, looks like it’s in the direction of what Im looking for. Thanks!

[winter linden]

We snuck in a new feature in the latest release 🤫 blueprints. If you're a platform engineer trying to roll out Dagger in a large codebase, and want to avoid excessive boilerplate, this is for you.

dagger init --blueprint=MODULE initializes a dagger module with no boilerplate code: just a reference to a blueprint. When you load your module, it's as if the blueprint were copy-pasted into it at runtime.

This is perfect for codebases with many components based on the same cookie cutter stack (for example, a php laravel app with mysql db; or a go grpc service...). Develop the blueprint module once; then reference it from any number of dagger modules, without boilerplate.

Here's the issue: https://github.com/dagger/dagger/issues/10464

Next: docs, and a few official blueprints that you can use 🙂

If you have a go app you want to daggerize, try this at the root of your go module (next to go.mod):

dagger init --blueprint=github.com/shykes/x/go

[cunning jolt]

Hey @winter linden , I am getting a 404 for - https://github.com/shykes/x/go. Is it not public yet?

[halcyon sphinx]

In a Dagger custom application (executed as dagger run uv run ./app.py) is there a way to instruct Dagger to always print the output of a given .with_exec() call to stdout (no matter whether the call actually got executed or hit the cache)? I don't want to always click on the dagger.cloud link just to see some command output. I suppose I could await the .with_exec() call and print() the output myself but maybe there's an easier way (especially if I have multiple consecutive calls, e.g. .with_exec("foo").with_exec("bar"))?

[winter linden]

It's a subdirectory of the github repo shykes/x. Dagger can load that address as a module, but it's not a valid http URL

[cunning jolt]

oh! my bad. Should have looked.

[winter linden]

You'll see that the module has nothing special. The main difference is that, if you load it directly, it won't do anything useful - because it uses default directories to access the context. Except when used as a blueprint, that will be the caller's context

In other words, if you use github.com/shykes/x/go as a blueprint in your go project, it's like I daggerized your project on your behalf

[cunning jolt]

Right.. It can be used as a non-blueprint module too but you have to explicitly pass in the source

[winter linden]

Exactly

[cunning jolt]

Me likey

[slender charm]

In a Dagger custom application (executed

[chilly arch]

Hey Daggernauts!

We are looking for Daggernauts with office space to share in the following cities in August for a meetup.

August 5 – Denver, Colorado
August 12 or 13 – Las Vegas
August 26 or 27 – Amsterdam

If you can sponsor a venue at any of these places on these dates, please DM me!

[tribal burrow]

"say when"
"no."

[winter linden]

For those of you interested in default secrets (a popular topic...). A new version of my POC is available. It's still rough, but testable. I would love to get some eyeballs on it!

https://github.com/dagger/dagger/pull/10697

--> There is a basic test command in the PR description, to get you started

[loud briar]

Getting endless "Loading..." in Dagger Cloud today, failed to decode snapshots: gob: wrong type (map[string]interface {}) for received field Spa snapshot.ExtraAttributes

[slender star]

Getting endless "Loading..." in Dagger

[wise kettle]

Hello, a question about Dagger - is dagger mainly helping in regards to just building and pushing and test images? Can I do stuff like maybe let’s say getting API responses from a package repository manager and then building a manifest file with just url links and then retrieving that file that I can just have locally or upload it somewhere else? Or is it just going to build me a container?

[slender star]

Hello, a question about Dagger - is

[loud briar]

Question about calling remote modules: what scenarios, if any, does dagger pull the module again vs using a cached copy? If I call a module's main branch for the first time the engine has to fetch that module, but if I call the main branch again does it fetch again? What if I call a branch ref, or tag/release ref? What about a commit hash?

Context: reducing time spent on calling a module from Gitlab, if possible

[wise kettle]

Hello just a general question, if both a Dockerfile and let’s say Dagger having an underlying BuildKit working - how is the caching system being used any differently? Is Dagger making use of BuildKit in another way?

[sharp marsh]

both Docker and Dagger use buildkit behind the scenes but each interact with their own instance so that's why the cache is not shared. Dagger also has the ability to build Dockerfiles (https://docs.dagger.io/cookbook/#build-image-from-dockerfile) so if you use Dagger to also build Dockerfiles, in that case you'll get maxium caching

[placid magnet]

Hi guys
Has anyone managed to run the Dagger engine in Jenkins without privileged permissions. for example, without mounting /var/run/docker.sock?
Our CI jobs run in Jenkins on ephemeral Kubernetes pods that can’t be given elevated privileges (we build Docker images with Kaniko for this reason).

[wise kettle]

Hello is there any guideline or documentation for getting a dagger engine as a pod on a k8s cluster? And how to set up the caching?

[sharp marsh]

yes, here https://docs.dagger.io/ci/integrations/kubernetes/

[sharp marsh]

hi there, it's currently not possible. Dagger needs elevated privileges to run (https://docs.dagger.io/faq/#why-does-the-dagger-engine-need-to-run-in-a-privileged-container)

[wise kettle]

Can I set up caching and a dagger engine in a minikube single node cluster?

[candid remnant]

Hello all, is there anyone who can help me on how to setup dagger in an air-gapped environment. I would like to use and develop dagger modules in our airgapped environment but are hitting a lot of rocks. For example a dagger init --sdk=go is trying to download stuff from the internet. Are there already past experiences or stances from the dagger community on how to work with dagger in airgapped environments?

[placid magnet]

I hope you will support it one day
I still don't understand it, if you support buildkit and i can run buildkit on kubernetes unprivileged and daemonless, why it wouldn't work ?

[sharp marsh]

I hope you will support it one day

[sharp marsh]

yes, you can totally do that. You can follow the Kubernetes integration guide in our docs to get an idea how that looks like 🙏

https://docs.dagger.io/ci/integrations/kubernetes/

[chilly arch]

**Denver Daggernauts, we're coming for you! **

We'll be co-hosting a meetup on August 5th with the Goose team and @warm temple. Come build with us!

https://lu.ma/tylz1e9o

[candid remnant]

I think I'm almost there by the information I found in this issue: https://github.com/dagger/dagger/issues/6599
The challange I'm facing now is that my GOPROXY requires authentication. Locally I can fix this by adding a .netrc file with my credentials. When I try to mount this file to several places I can't get it to work... Any experiences or ideas on how I can tackle this?

[cunning jolt]

Proxy authentication for Go (GOPROXY)

[raven stag]

Hey, random question.
As a maintainer of modules at my $org, is there an "dagger" way to raise a deprecated input?
Say I want to deprecate a parameter in favour of another one, and I want consumers of the module to have a prompt stating that a parameter that's being used will be removed in future releases of the module

Is there any functioality that would help me? (other than coding my own "check for input being used -> spit something to the logs")

[winter linden]

yes you can annotate an argument as deprecated. eg +deprecated in Go.

[winter linden]

actually I'm wondering if I imagined that 😅 it's definitely a thing when developing the core engine. Can't remember if it was made available to modules also.

[narrow nymph]

it's not available in modules

should be easy to add though

[sharp marsh]

yep, was about to mention that. We're not storing the deprecated attribute at the TypeDef level: https://github.com/dagger/dagger/blob/57403d72fcbcbcb55fcb616422e0c382f67bd396/core/typedef.go?plain=1#L315

[narrow nymph]

tracking issue: https://github.com/dagger/dagger/issues/10818

[raven stag]

damn... haha, i spent the last 10 min checking how that was supposed to be raised (CLI / dagger gen code / autocompletion)... and no trace of the // +deprecated)

Thanks for the issue!

[timber sphinx]

can i just say the LLM integration is so freaking fun?

I've built since the beginning, but no SDK I use has agentic built in. So I'm used to take content, output, write it, etc.
The ability to plug in a simple "go migrate this nondeterministic file" then pop it into a PR in my automation is such as freaking amazing experience. Still wrapping my head around correct handling/passing of content and haven't even touched the shell but it's such a great experience 🏆

Between the new Crush tool from charmbracelet and figuring out my first semi advanced dagger module, with LLM integration that is completely compliant with being self hosted "BYOK" and containerized.... it's a good week 👍

[halcyon sphinx]

Just spitballing here, but what if the Kubernetes cluster used something like krun as OCI runtime? In that case, running regular (rootful) Docker inside the pod might work? (I have long been looking for a good way to nest containers actually…)

[sharp marsh]

yes, that should be ok. It's also not necessary to run Docker since the engine pod will run in an isolated VM

[halcyon sphinx]

yes, that should be ok. It's also not

[wispy steppe]

What's the best resource for someone who wants to fiddle with creating an sdk for a new programming language? I've had a look in the other sdks, and tried to read some of Gemini's musings about it, and looks like the first actionable step is to have a basic way of sending graphql queries to the dagger api. But there's so much more around that that are unknown unknowns to me

Or, nevermind most of my previous question - I have many of the resources that I can use, like the docs and the other sdks, but what I feel I'm lacking is a bit of direction. If you guys got a "this specific api is a good place to start", that would be great 😎

[modest pier]

what language?

[wispy steppe]

Vlang (https://vlang.io/)

[modest pier]

I might try to emulate the rust sdk bc it’s compiled, and our go sdk is the “weird” one because it bootstraps the other sdks

[wispy steppe]

I might try to emulate the rust sdk bc

[loud briar]

Is there a way to see Dagger operations in traces? Trying to find out why Dagger seems to pause for nearly a minute for an operation that takes 3s locally

[high meteor]

Hello

Is there a way to check the login status of the user?

I like to create simple scripts that calls specific Dagger functions locally, and I would also like to be able to check if the user is currently logged in, and attempt to log them in if they are not already.

[slender star]

Hello

[cunning jolt]

Is there a sample agent instructions file (agent.md, copilot-instructions.md) file that would help AI Agents understand Dagger better? Whenever I try to get Claude (via copilot) to modify my dagger code it struggles.

[wise kettle]

Just a quick question - how do you rename a module name - do I just change the name in the dagger.json and do dagger develop?

[brisk anchor]

I guess you would have to rename the module struct as well ? You could use a tool like scooter to do search and replace. https://github.com/thomasschafer/scooter

Does any one know if it's possible to pipe a custom dagger function that returns a *dagger.Container to another custom dagger function the takes a *dagger.Container as an argument.

dagger -c 'some-build-function | some-push-function --push-secret'

[loud briar]

Piping *dagger.Container into functions

[raven stag]

Random question:
Is there a way to retrieve the path of a directory that was passed as an input parameter (*dagger.Directory)?

[brisk anchor]

Has anyone successfully been able to push an OCI image to ghcr.io from a Github workflow using the secrets.GITHUB_TOKEN as authentication with the .WithRegistryAuth() function ?
My conclusion is that it is not supported but I'am hoping for someone to prove me wrong.

[warm temple]

Has anyone successfully been able to

[late kite]

is it possible to use a secret with with-exec's --stdin? (i'm suspecting it's not, as it's listed as taking a string in the go sdk docs)

[cunning jolt]

This will enable it from next release. It was merged today - https://github.com/dagger/dagger/pull/10800

[stuck relic]

AI Agent HackNight with MCP ft. Goose, T...

[hidden dirge]

Hey folks, is there an easy way to distribute dagger shell scripts with a dagger module?

TL;DR - I have a few dagger shell scripts to easy calling dagger functions locally, so that consumers of our CI modules don't need to pass a bunch of flags to the constructor. Ideally, we'd maintain these files centrally and somehow make them available for repos which install this module from git.

[sharp marsh]

Hey folks, is there an easy way to

[midnight pewter]

Hi daggerists, I discovered by accident this project and am a little bit curious about it.

Would this project be compatible with: A program that needs to trigger some jobs. We could imagine let say a 3d printing factory with multiple 3d printers.

Then my program will schedule some jobs on different machines and then actually spawn some "dagger" agents with specific workflows to perform having 1 workflow per 3d printer machine and thing to print?

TLDR; can it be used as an "executor" for parallelism in a bigger application? can it be controlled via rest API or something similar?

Thank you

[junior ibex]

Hey folks I'm starting to collect dagger traces over OTEL. I'm collecting them fine, but i notice that the operation spans lack crucial detail - like dagop.ctr withExec doesn't seem to include span attribute for the command executed. Is there something I'm missing?

[still olive]

hey y'all 👋 Any CircleCI users using Dagger? I took a stab at making an Dagger orb based on the Dagger GitHub action, and its now live: https://circleci.com/developer/orbs/orb/cci-labs/dagger-for-circleci

Would love any feedback from the community on how to improve it! I wanted to make using Dagger on CircleCI even easier for folks.

[slender charm]

This is dope, i'll check it out today!

[still olive]

@slender charm I am currently caching the CLI, to maybe save some time (a couple seconds each time), but I was thinking if we could cache more. Modules? Docker layers? But my knowledge of Dagger is limited, but I am open to ideas.

CircleCI does have Docker layer caching (DLC), so we might be able to cache some additional things. The Act orb, which allows you to run GitHub actions on CircleCI, does a similar thing. Where we cache docker images that the Act CLI uses to speed up overall builds.

[slender charm]

I think this will remain a gap for the forseable future, because DLC does not help Dagger at the moement. We're exploring a few different approaches but caching for ephemeral runners is tricky.

[still olive]

Would saving the volumes, https://docs.dagger.io/api/cache-volumes?sdk=Dagger+CLI, speed up the builds on next run? Are these docker volumes under the hood?

Could we tar them up, then restore on the next run of the pipeline?

[sharp marsh]

hey there, that information is encoded in the parent span of the dagop.Ctr span under the dagger.io/dag.call attribute which a is a base64 encoded protobuf definition of (https://github.com/dagger/dagger/blob/main/dagql/call/callpbv1/call.proto)

[junior ibex]

hrm, I guess this'll make it pretty tricky to visualise in other tooling, right? Thanks for the reply! something to look intpo

[sharp marsh]

sure, np! we had to encode it that way to achieve good performance and UX in Dagger Cloud's telemetry, though the side effect is that it does make it more difficult for other general purpose tools to parse and display this data correctly.

[late kite]

I'd like to "export" the contents of a cache volume. It doesn't look like you can container | directory "/a-cache-volume, though?

~ $ dagger -c 'container | from alpine | with-mounted-cache /foo gpt-more | with-workdir /foo | directory static.html'
▶ connect 0.1s
▶ detect module: . 0.0s
● loading type definitions 0.2s
✘ directory 0.1s ERROR
! /foo/static.html: cannot retrieve path from cache

$ container: Container! 0.0s CACHED
○ .from(address: "alpine"): Container! 0.1s
● .withMountedCache(
  ┆ path: "/foo"
  ┆ cache: cacheVolume(key: "gpt-more"): CacheVolume!
  ): Container! 0.0s
● .withWorkdir(path: "/foo"): Container! 0.0s
✘ .directory(path: "static.html"): Directory! 0.0s ERROR
! /foo/static.html: cannot retrieve path from cache

[viscid vortex]

Guys, hello everyone, is TypeScript SDK up to date (in reality)? I get an error that dag.host() method does not exist on Client. I just start learning Dagger.

[jagged flicker]

It should be up to date yes.
In which context are you calling dag.host()? If it's inside a module that's execpted, dag.host() is not available in this situation and the module has no access to the host (filesystem for instance)

[viscid vortex]

Yes, I was exactly trying to call it inside the module I just created via init command. Why it appears then in TS reference? Tbh, I experience struggles in learning Dagger since docs do not explain fundamentals in incremental composable way (method by method a la lodash). I wish to create my entire CI pipeline with Dagger: from building affected (using Lerna) apps images (instead of doing it via Dockerfile in my monorepo) to pushing these images into registry.

Can you recommend me a beginner friendly path to learn how to remap/translate multistage cache optimized Dockerfile into Dagger's code?

Example of the Dockerfile:

# Use ARG to define the application we want to build. Default to 'web' if not provided.
ARG APP_NAME=web
ARG APP_PATH=apps/${APP_NAME}

FROM node:22-slim AS builder-base
ENV PNPM_HOME="/pnpm"
ENV PATH="$PNPM_HOME:$PATH"
RUN corepack enable pnpm

FROM builder-base AS deps
COPY pnpm-lock.yaml pnpm-workspace.yaml ./
RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm fetch

FROM builder-base AS builder
ARG APP_NAME
WORKDIR /athrio
COPY . .

RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --filter "${APP_NAME}..." --offline
RUN pnpm --filter "${APP_NAME}..." build

FROM gcr.io/distroless/nodejs22-debian12:nonroot AS nextjs-runner
ARG APP_NAME
ARG APP_PATH
WORKDIR /athrio/${APP_PATH}

ENV NODE_ENV=production
ENV PORT=3000
ENV HOSTNAME="0.0.0.0"

COPY --from=builder --chown=nonroot:nonroot /athrio/${APP_PATH}/.next/standalone ./
COPY --from=builder --chown=nonroot:nonroot /athrio/${APP_PATH}/.next/static ./.next/static
COPY --from=builder --chown=nonroot:nonroot /athrio/${APP_PATH}/public ./public

EXPOSE 3000

CMD ["server.js"]

[jagged flicker]

It's in the reference because it's available in the SDK to create custom applications. It's when you want to use the dagger SDK from inside an existing code. In that case you need to access the host.
When using modules the host part is done on the caller side. For instance after a dagger init you have a grepDir function taking a directory as an argument and it's translated automatically from your host to the module.
Basically the module is a container, there's no direct code access to the "host" (your machine) from inside the module.

[jagged flicker]

Did you saw the build section of the cookbook? It can help to see what's available: https://docs.dagger.io/cookbook#builds

[viscid vortex]

I had a look already and will learn it again. Basically, my stage is now — figuring out how this works in general for CI/TypeScript: what I should and can do, and why it is so. I like Dagger's approach a lot. Just docs could contain a section which explains how to compose functions in the best practice way, in API description manner. Now, I see fragmented examples and SDK reference. So, I need to decipher it myself. I would contribute to the documentation, if I knew Dagger. 🙂

Just as a feedback from a beginner.

[jagged flicker]

I haven't tried it, so maybe there's some changes to do, but this kind of Dockerfile could look like that in Dagger.
(I'm using Go here but the beauty is it's very close in Typescript, the translation should be easy)

func (m *MyModule) Build(
    ctx context.Context,
    // +defaultPath="."
    src *dagger.Directory,
    // +default="web"
    appName string,
    // +default="apps/"
    appPathPrefix string,
) *dagger.Container {
    appPath := appPathPrefix + appName

    builderBase := dag.Container().
        From("node:22-slim").
        WithEnvVariable("PNPM_HOME", "/pnpm").
        WithEnvVariable("PATH", "$PNPM_HOME:$PATH").
        WithExec([]string{"corepack", "enable", "pnpm"})

    pnpmCache := dag.CacheVolume("pnpm")

    deps := builderBase.
        WithFile("pnpm-lock.yaml", src.File("pnpm-lock.yaml")).
        WithFile("pnpm-workspace.yaml", src.File("pnpm-workspace.yaml")).
        WithMountedCache("/pnpm/store", pnpmCache).
        WithExec([]string{"pnpm", "fetch"})

    builder := deps.
        WithWorkdir("/athrio").
        WithDirectory(".", src).
        WithExec([]string{"pnpm", "install", "--filter", appName + "...", "--offline"}).
        WithExec([]string{"pnpm", "--filter", appName + "...", "build"})

    nextjsRunner := dag.Container().
        From("gcr.io/distroless/nodejs22-debian12:nonroot").
        WithExposedPort(3000).
        WithWorkdir("/athrio/" + appPath).
        WithEnvVariable("NODE_ENV", "production").
        WithEnvVariable("PORT", "3000").
        WithEnvVariable("HOSTNAME", "0.0.0.0").
        WithMountedDirectory(".", builder.Directory(filepath.Join("/athrio", appPath, ".next", "standalone"))).
        WithMountedDirectory("./.next/static", builder.Directory(filepath.Join("/athrio", appPath, ".next", "static"))).
        WithDefaultArgs([]string{"node", "server.js"})
    
    return nextjsRunner
}

To call the build function should give a container that can then be run, pushed, etc

[viscid vortex]

Wow, you just wrote my Dockerfile! 😄 Thanks! I will work on it and tell you if I managed to create the working TS pipeline basing on this.

[jagged flicker]

no problem 🙂 if that's in a module you should be able to do something like:
dagger -c 'build | as-service | up --ports=3000:3000' and your app should run on localhost:3000
Or dagger -c 'build | publish my-registry/username/repo:tag to push it somewhere.
If you haven't done it yet, I encourage you to have a look at the CI quickstart for instance

[loud briar]

How do you pass local directories in Dagger scripts? Using dagger call with file:~/.filename works, but in Dagger shell or scripts it doesn't find the file - it looks in the present working dir suffixed with the path, e.g. in /this/is/my/dagger/module it looks for the file at /this/is/my/dagger/module/Users/myname/.filename

[jagged flicker]

In dagger shell you have access to a host object. For instance:

dagger -c 'host | file README.md | contents'

And then you can do things like:

dagger -c 'container | from alpine | with-file /src/readme.md $(host | file README.md) | directory /src | entries'
readme.md

[chilly arch]

Hey Daggernauts 👋

We’ve got something fun coming up in August that blends two awesome worlds, Dagger + LocalStack.

@neon warren from our team will be showing how to treat your CI pipelines like real software components you can run locally, test easily, and reuse anywhere. Then @neon crescent from LocalStack will join to walk us through running a full AWS environment right inside your Dagger pipelines, complete with Cloud Pods and Ephemeral Instances.

It’s a chance to see how two powerful tools fit together to make your workflows faster, more reliable, and way more enjoyable to work on.

LocalStack will be hosting this live from YouTube, so you can join from anywhere! You will need to register to get the link.

📅 August 27th - 7 am PT
🌎 Virtual - Livestream on YouTube
🔗 RSVP here for YouTube link - https://www.meetup.com/localstack-community/events/310385779/?eventOrigin=group_events_list

[halcyon heart]

Whats the minimal required specs for a shared VPS machine that will host a Dagger engine??

[sharp marsh]

There's no minimal specs but take into account that the smallest the server, the slower the pipeliens

In some cases they might not even finish due to resource starvation depending on what you're running there

Dagger is quite disk intensive so if possible we always recommend using the fastest drives you can afford

[halcyon heart]

Thanks @sharp marsh
So NVMe/SSD is the way to go then if I got your point and maybe 4vCPUs and above....

[true trench]

would dagger be an efficient replacement for "docker-compose" ? or is it more "used with docker-compose"? eg I need some platform to monitor my deployments, and create containers on the spot, do a few actions with them configure networks, etc

should I rather go with Pulumi/docker for this?

[winter linden]

Dagger didn't start as docker-compose alternative, but it's definitely encroaching more and more (based on how I see the community use it).

Depending on the tasks or workflows you have in mind, I can give you a more detailed recommendation.

cc @jagged flicker @narrow nymph

[true trench]

would love to!

Basically for now I'm using Pulumi as a better docker-compose, creating my containers in typescript, and doing some light orchestration, but I have a feeling that Dagger could be better suited to this, and have Pulumi just handle the "infra" side of stuff (creating Service Accounts, storing keys in 1Password, creating buckets, etc), and have dagger "run the services" on top of the infra

[true trench]

how would you suggest to do this? is there a kind of documentation / template which I could follow to do a first "docker-compose equivalent" with dagger?

basically nothing really fancy, just orchestating a few containers, their network so they can easily speak to each other, and possibly doing a few "pre-launch" operations, like running a few commands inside the container to change a few settings / reset db / push seed data (some people push images that don't let you specify everything through env vars unfortunately)

basically looking for something "more programmatic" than raw Docker-compose files, but not as bulky as a Kubernetes distrib... something in the middle haha

[winter linden]

do you plan on running these workflows in both local dev and CI environments?

will you integrate it into wider workflows, for example across teams and diverse tool chains?

[true trench]

exactly, local for dev, and as a light prod setup

[true trench]

no for now it's for a limited team, so it will be mostly me doing dev and maintenance

or perhaps dagger is suited only for workflows, rather than "running and monitoring containers" like docker-compose/swarm?

[winter linden]

Dagger is a good choice for portability & repeatability across environments (ie. dev, CI, prod)

[true trench]

I'd like the minimal thing in prod, like watch if the process crashes and restart it, without doing too much fancy Kubernetes-like stuff for scaling for now as I don't need it, and don't want to pay for the Kubernetes complexity upfront

[winter linden]

You can definitely run containers, wire them together etc with dagger. As long as they're ephemeral. I wouldn't run an actual prod service in Dagger.

[true trench]

okay I see, so it's more for workflows than to watch "prod-like" stuff

like builds, and tests, etc

I'm basically looking for a "swarm" like system, a middle ground between compose and Kubernetes, but it feels like swarm is not maintained (or is it?)

and something that I can create programmatically, not just with dumb yaml files who always fall short

[winter linden]

Sounds like Dagger might be a good fit, there's a little more upfront complexity compared to docker-compose (not too bad imo), but in return you get a real API and much more programmable control over the workflow and the container orchestration

[true trench]

indeed, but as you say it wouldn't fit running "services" in prod, right?

so it would be a fit more for my CICD to do build/tests, but not to run stuff?

[winter linden]

Right, at least not yet 😇

The sweet spot for running services in Dagger is to spin up ephemeral dependencies for integration tests, QA etc

[true trench]

I would basically get containers at the end, and orchestrate those with Swarm/mini-kube/etc

okay I see! thanks 🙂

[winter linden]

or for running a local dev instance

[true trench]

okay

I'd like to be as isomorphic as possible between local dev and prod though... even if this means removing some scaling options/etc

that's why Pulumi was interesting: it works quite nicely in dev, and could possibly (but don't know if it's complex) run for prod too

[winter linden]

There's an interesting crossover with AI agents, because those have a very different application architecture which is basically made of workflows.

[true trench]

indeed

although I guess you still need some central base to orchestrate those workflows, which is a long-running service somewhere

[winter linden]

So if you're developing an agent, you could use Dagger both to orchestrate the building and testing of the agent, and also as the workflow runtime for the agent itself

[true trench]

indeed, I see... although (beyond the current AI hype) I'm not sure what's specific to AI there... could be as useful for any workflow that's just using deterministic processes

[winter linden]

yes. Typically you would combine a durable execution system (off the shelf or custom) with Dagger as an ephemeral execution system. They work well together.

[true trench]

speaking of this, do you have recommendations? I'm looking at DBOS and Inngest

[winter linden]

Yes absolutely. It's just that AI agents are popular, so as a result more developers are architecting their applications as workflows

[true trench]

actually a dagger-native durable execution engine could make sense perhaps

or just integrating it tightly with a local postgres running DBOS

[winter linden]

we will get there 😁

[true trench]

haha

[winter linden]

The decoupling is nice

[true trench]

I agree

better have a strong interface that can fit multiple providers and adapt to many situations

yet the "all-in-one-toolbox" / "I install and it works" is quite nice to have a new tech piece getting traction

[winter linden]

It allows for simple local execution without having to bootstrap a giant local replica of an event system and machine orchestrator

[karmic zephyr]

Hello everyone, quick question about “Module Dependencies.” The examples show GitHub URLs. Is this compatible with Bitbucket?

[slender charm]

Hey! Yes it should work with any git SCM.

We also support SSH and some provider-specific auth: https://docs.dagger.io/api/remote-repositories/#ssh-authentication if you have private modules/repos

[brisk anchor]

How do people organize their repo with dagger modules and version tagging ?
If you have multiple modules in the same repo, in a folder structure, you would have to version tag with the module name, kinda like my-module-v0.1.0 for every module you have. Or is there a better way?

[chilly arch]

**Anyone going to PyCon India? **

Looks like there will be a Dagger talk there! https://www.linkedin.com/posts/urvashi-choubey_pyconindia-techcommunity-dagger-activity-7364026872772374530-kNED?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAbjwqUBPZEp__AD2xcvfBAb5pzs_ir11nM

[brisk anchor]

Is it possible to use the existing entry point of an image and only supply the command. Using .WithExec() ignores the entry point and you will need to include the executable you would like to run.
Like: .WithExec([]string{"--help"}) instead of .WithExec([]string{"/path/to/executable", "--help"})

[loud briar]

https://pkg.go.dev/dagger.io/dagger#ContainerWithExecOpts UseEntrypoint

[brisk anchor]

Explain it like Im 5....

I dont want to set the entry point.. I want to use the one that comes with the image, and only provide the arguments for the existing entry point.

[loud briar]

@brisk anchor UseEntrypoint: true

Uses the existing entrypoint

It requires an explicit setting because previously it was implicit and not clear that the image entrypoint was being preprended without telling you

[brisk anchor]

Cant seem to find it in the docs

The search bar couldn't find anything on it, but Google could.
https://docs.dagger.io/api/services/

Like this? dagger.Container{UseEntrypoint: true} ?

Nope.. Apprently not

dagger.ContainerWithExecOpts{UseEntrypoint: true}, so far so good, but this is not a dagger.Container.

Got it...

dag.Container().From("some-image").WithExec([]string{"--help"}, dagger.ContainerWithExecOpts{UseEntrypoint: true})

[jagged flicker]

Yes that's it 👍
WithExec is a bit like a RUN inside a Dockerfile, by default it doesn't use the entrypoint.

[loud briar]

There's quite a lot of this pattern in the Go SDK. dagger.<Thing><Operation>Opts e.g. dagger.DirectoryWithDirectoryOpts Dagger.ContainerWithEnvVariableOpts etc, once I figured it out I've grown to like it, simple to use

[torpid maple]

Has anyone setup Dagger caching using using container registry alone? (assume I don't want to use Dagger Cloud, and I'm running my CI workloads on GitLab via EKS runners).

[torpid maple]

Also rather than having to run docker/dind as a service in a GitLab CI pipeline here as documented here.
https://docs.dagger.io/ci/integrations/gitlab

Could we not instead run a dagger-engine service and use that?

[slender star]

Yes, you should be able to do something like https://dagger.io/blog/argo-cd-kubernetes, but for GitLab runners that would take advantage of the local cache on each EKS node.

[karmic zephyr]

Thanks for the quick reply.
I'm trying, but I can't get it to work. At my company, we use port 443 for SSH. It works locally (git clone), but when I go through the engine by doing “dagger install ssh://myhostname:443/xxx,” it doesn't work. I get an error with port 22.

[cursive bridge]

How do I modify the PATH env to include a dir off the $HOME of the user executing the container? This is the kind of thing I'm trying to do:

homedir, _ := ctr.WithExec([]string{"realpath", "~"}).Stdout(ctx)
ctr.WithEnvVariable("PATH", "$PATH:"+strings.TrimSuffix(homedir, "\n")+"/go/bin", dagger.ContainerWithEnvVariableOpts{Expand: true}).

I've also tried referencing $HOME directly, but no dice. In both cases, the homedir/$HOME is empty. I suspect because when dagger is executing the WithEnvVariable or even WithExec there isn't a login shell (although withExec surprises me).

I found this bug posting, which may be related/same thing:
https://github.com/dagger/dagger/issues/10881

Is there a way to get the container user's homedir?

[jagged flicker]

One way, at least as a temporary solution, would be to get the output of echo $HOME.
You can try this quick function, I get the $HOME directory, create a file inside, add it to the $PATH and run it:

func (m *MyModule) RunWithEnv(ctx context.Context) (string, error) {
    ctr := dag.Container().From("alpine")
    home, err := ctr.WithExec([]string{"sh", "-c", "echo $HOME"}).Stdout(ctx)
    if err != nil {
        return "", err
    }
    home = strings.TrimSpace(home)
    return ctr.
        WithEnvVariable("PATH", fmt.Sprintf("$PATH:%s/go/bin", home), dagger.ContainerWithEnvVariableOpts{
            Expand: true,
        }).
        WithNewFile(fmt.Sprintf("%s/go/bin/foo", home), `#!/bin/sh
echo Hello World`, dagger.ContainerWithNewFileOpts{Permissions: 777}).
        WithExec([]string{"foo"}).
        Stdout(ctx)
}

[cursive bridge]

Hrm, I see why that would work. Didn't think to try invoking sh in the WithExec. Let me try that.

[karmic zephyr]

Does anyone have any ideas to help me? Thanks in advance to anyone who takes the time to help me.

[cursive bridge]

@jagged flicker That suggestion does in fact work!

TY!

[cursive bridge]

When I do a container.WithEnvVariable and pass dagger.ContainerWithEnvVariableOpts{Expand: true}, how is that expanstion performed?

I have a situation where the same code works on one image, but fails on another. This is the full line attempted:

ctr.WithEnvVariable("PATH", "$PATH:"+strings.TrimSuffix(homedir, "\n")+"/go/bin", dagger.ContainerWithEnvVariableOpts{Expand: true})

With one container image the PATH env is set correctly, in another it completely hoses PATH and obviously causes everything to fail. I've isolated the failure to this code snipet, but I don't see why just this expansion would differ so much.

Also, the homedir variable is correct for both images (different values though).

[cursive bridge]

I found the answer. It would appear that the value for echo $PATH has a \n in one instance and not in another. When using a string that ends with a \n in trying to sset the env for PATH, that silently fails and breaks the setting of the PATH env

[cursive bridge]

A bit more information. In both images, the path actually comes back the same, with a trailing \n. However, when I set the PATH using the expansion option, it works for one image and not the other. I can only guess something is stripping the trailing \n in one instance and not the other. If I grab the path via:

path, _ := ctr.WithExec([]string{"sh", "-c", "echo $PATH"}).Stdout(ctx)

And then trim the trailing \n, and add the path I want, it works for both images.

[slender star]

Did you try echo -n

[cursive bridge]

I haven't tried that, but can do it. I have it working with the echo, but in my original code I didn't use echo. I did:

WithEnvVariable("PATH", "$PATH:"+strings.TrimSuffix(homedir, "\n")+"/go/bin", dagger.ContainerWithEnvVariableOpts{Expand: true})

And ^^ works on some images and not others. I'm taking a guess as to why based upon the errors I saw. The \n seems to kill the PATH definition, and can reproduce that via the echo approach. I don't have a clue why the expansion functionality would be different on the images, but the error state matches not having the \n removed from the existing path definition when trying to append.

In order to get it to work on both images, I do:

path, _ := ctr.WithExec([]string{"sh", "-c", "echo $PATH"}).Stdout(ctx)
path = strings.TrimSuffix(path, "\n") + ":" + homedir + "/go/bin"
    ctr.WithEnvVariable("PATH", path)

So I am not using the expansion functionality and am able to add the path. I'm explicitly setting the new value instead of relying on a $PATH:<new_path> expansion method

[torpid maple]

I feel like Dagger is expecting you never to have Git repositories deeper that one sub-path (GitHub centric view of things)?

$ dagger -m gitlab.com/jshbrntt/sandbox/pipelines call grep-dir --directory-arg . --pattern pipelines
▶ connect 0.5s
▼ load module: gitlab.com/jshbrntt/sandbox/pipelines 5.8s ERROR
! failed to get configured module: failed to resolve git src: failed to resolve git src: select: cannot resolve "https://gitlab.com/jshbrntt/sandbox": git
  error: exit status 128
  stderr:
  remote: The project you were looking for could not be found or you don't have permission to view it.
  fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found
# ...snip
        stderr:
        remote: The project you were looking for could not be found or you don't have permission to view it.
        fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found
      ╰─▼ git ls-remote --symref https://gitlab.com/jshbrntt/sandbox HEAD HEAD^{} 0.6s ERROR
        ┃ remote: The project you were looking for could not be found or you don't have permission to view it.                                             
        ┃ fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found
        ! git error: exit status 128
          stderr:
          remote: The project you were looking for could not be found or you don't have permission to view it.
          fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found

Error logs:

▼ git ls-remote --symref https://gitlab.com/jshbrntt/sandbox HEAD HEAD^{} 0.6s ERROR
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found

! git error: exit status 128
  stderr:
  remote: The project you were looking for could not be found or you don't have permission to view it.
  fatal: repository 'https://gitlab.com/jshbrntt/sandbox/' not found

[slender charm]

I feel like Dagger is expecting you

[torpid maple]

Is there anyway to stop dagger from breaking links in it's output?

├─● Container.withEnvVariable(name: "CACHEBUSTER", value: "2025-08-27 13:52:42.355457"): Container! 0.0s
╰─▼ .withExec(args: ["aws", "sso", "login", "--profile", "dev"]): Container! 38.0s
  ┃ Attempting to automatically open the SSO authorization page in your default browser.                           
  ┃ If the browser does not open or you wish to use a different device to authorize this request, open the followin
  ┃ L:                                                                                                             
  ┃                                                                                                                
  ┃ https://oidc.us-east-1.amazonaws.com/authorize?response_type=code&client_id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&
  ┃ rect_uri=http%3A%2F%2F127.0.0.1%3A38921%2Foauth%2Fcallback&state=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&code_chal
  ┃ e_method=S256&scopes=sso%3Aaccount%3Aaccess&code_challenge=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

It's very annoying not to be able to click this link in the terminal.

[cursive bridge]

I'm having a caching issue I don't know how to solve. I'm using the golangci-lint dagger module and trying to run using a container who's user is not root. The module uses a cache dir, which I can figure, but it would appear the mounted directory is always owned by the root user.

The golangci-lint module allows me to pass in a cache dir, but does not expose any configuration for ownership of the mounted directory. It appears dagger has options for defining ownership via WithMountedCache, but before I head off an suggest/PR changes to that module is that the right path? I would imagine this cache/dir ownership issue would pop up in other containers. Should dagger ensure that mounted directores are mounted rw for the default user of that container?

[tall python]

the dagger repository using git-gallery.com

[timber sphinx]

i see docs changed as I'm working ✨
Looking forward to seeing the changes. 💪

[slender star]

https://tenor.com/view/never-ending-story-gif-24792258

Thanks! Love to hear your feedback.

[unborn pendant]

you can pass some variable to your dagger function call, let's use banana for the argument:
--banana cmd://"aws configure export-credentials --profile whatever_profile --format process"

and then:
.with_mounted_secret("/aws.json", banana)

and use this option in the profile:
credential_process = cat /aws.json

works in my machine(tm)

for a github action we'll need to use roles with openid connect... unsure how it will work then – i'm counting on magic here

[!IMPORTANT] it took me a long long time to figure this out

[timber sphinx]

as long as I can still get it with context7 I'm good. 🙂
But for the review now it's a lot less overwhelming to find the details, so good reorg

[torpid maple]

Sorry but I think you've misunderstood, this command is running locally when I developer attempts to run the pipeline they don't have the credentials from the CI environment.

Instead they have to authenticate as themselves in order to assume the same role that the CI environment would when it runs the pipeline.

In order to do this they have to go through the aws sso login flow which requires they click a link, authenticate in the browser, then the CLI will be authorized with a token obtained from this process.

My issue is that when the developer runs the pipeline with the dagger TUI it will wrap the link and break it so it cannot be clicked therefore breaking this method of authentication.

[torpid maple]

Has anyone got an example of using GitLab CI cache to store and restore Dagger's cache?

[loud briar]

Are you running Dagger as a Gitlab service? If so I'm fairly certain you have no access to it, unless it's changed recently Gitlab don't allow you to add/access volumes on services

I'm also using Gitlab but with a dedicated engine hosted and accessible from the runners, I'd recommend this approach over the in-Gitlab options

[torpid maple]

Yes I'm running the Dagger Engine as a service.

  services:
    - name: registry.dagger.io/engine:${DAGGER_VERSION}
      alias: dagger-engine
      command:
        - --addr
        - tcp://0.0.0.0:1234

[loud briar]

I'm fairly certain what you're asking for isn't possible then. Gitlab services are quite limited in terms of volumes

[torpid maple]

So there's no way to export the EngineCache?

The only way is to directly access the volume of the Dagger engine container (running as a service on GitLab CI).

[loud briar]

I don't even think you can get to it

[torpid maple]

🤔

[loud briar]

https://docs.gitlab.com/ci/services/#accessing-the-services

[torpid maple]

So prior to running it as a service I was starting a dind service, then initializing the dagger engine on that docker host for use in the job. That worked fine and I could access the volumes of the running containers using the Docker CLI.

[loud briar]

Yeah, but with services you have no access to the Docker instance that starts the service

[torpid maple]

So I could go back to that approach it just means there's a little more overhead as you're having to run the dind service, then the dagger engine container on that.

[loud briar]

Is running a dedicated engine not an option?

It is remarkably simple to do and has performance benefits

[torpid maple]

It could be done in future, I'd just have to coordinate with the internal team who managed our self-hosted GitLab and runners.

However, I kind of want to prove Dagger out a bit more before I bother getting other teams involved.

What does your dedicated engine approach look like in practice?

Have you got a GitLab runner specifically for Dagger jobs, with the dedicated engine bound to a specific host?

[loud briar]

A large EC2 with a large NVMe drive accessible to a specific runner tag

Engine is exposed on a port, network load balancer between Dagger and runners

[torpid maple]

So our GitLab runners are using EKS, to 'scale' out in order to meet flucating demand.

[loud briar]

NLB url is saved in Gitlab CICD vars at the top level, so jobs just run dagger ... and it works

[torpid maple]

So not sure how that would look, probably would have to use a PVC where the engine data is stored.

[loud briar]

Yep

I think there's some docs about this on docs.dagger.io

[torpid maple]

Don't even know if that would work having multiple things simultaneously access said volume.

[warm temple]

yeah there's a bunch about using dagger with your CI runners on kubernetes here https://docs.dagger.io/reference/container-runtimes/kubernetes
There's a few approaches to optimizing cache in that setup. Typically you create a volume for /var/lib/dagger. Works great if you run the engine as a daemonset. If you run as a sidecar (not sure exactly how gitlab services get executed) you'll want a copy-on-read volume because you don't want multiple engines writing to that data at a time. Lots more on this in #kubernetes

[neon warren]

@here The team launched a new documentation site this week! 🎉

Based on your feedback, we’ve restructured the docs to make it easier to find the right information depending on your focus whether you’re a new user, a module developer, or working in operations/support.

We’d love your feedback as you explore the new site. Hope you enjoy it and let us know if you have any questions!

[slender star]

I'm having a caching issue I don't know

[torpid maple]

I know it's not advisable but it would be great to see what sort of hacks people have used for cache, like exporting it to be cached by a CI's default caching facilitor, or exporting it to a container image registry?

With this environment variable _EXPERIMENTAL_DAGGER_CACHE_CONFIG does anyone know what that is specifically used for is it just for container step caching as opposed to the cache volumes?

[winter linden]

yes it's literally a pass through to buildkit layer cache export

[unborn pendant]

I use the same aws sso command, but I do it before running automation. Then I just pass the credentials to dagger as I've wrote.

[mild coral]

Hi there guys. Just wanted to know what this operation does asModule getModuleDef:

In some of our traces, we are seeing its taking a long time...

more context:

finding that module loading is taking a while...

it's likely an issue on our end, but trying to understand this particular op as its the slow one.. might help us track down the issue

can DM the full trace if that helps

[mild coral]

More on this... it seems our disk cache has blown up the 2TB which is the size of the disk itself...

Not sure how this happened and didn't think what we were doing consumed this much disk... but when we get to the bottom of it, will let you know

Realise I need to prefix this with "we are using self hosted runners"

[tired moth]

Dagger or container-use is going to support? https://agents.md/

[warm temple]

With agents in dagger you can pass in any prompt file you want, so yes its supported. I've been using this convention. For container-use its not relevant because its an MCP server and not an agent

[tired moth]

then we should get dagger listed on agents.md, what you think?

[warm temple]

It looks like they list out agent products which makes sense because those are hard coded to look for AGENTS.md, but they dont list out agent toolkits or SDKs because you have to build your own agent to be compatible with AGENTS.md. I think dagger agents fall into that second bucket

[tired moth]

but can we give optional flag based support of agents.md in dagger agent and then we can get listed dagger there, because like MCP this is also going to be big. Just my feedback

[white zephyr]

tongue in cheek slogan "Dagger: finally the Golang use case beyond kubernetes you were waiting for"

[mild coral]

Hi @winter linden I noticed that this PR is open right now for .env support. Just wanted to double check, is this feature coming? Was in discussions with the team this morning and it would be really useful to have something like this. We are finding our dagger modules are rapidly exploding with many secret parameters that need to be injected.

A few questions from me on this:

1. will non secret types be supported ( say a parameter is a string, can I have a default )
2. Am I able to use any secret provider?
3. If a module ships a .env file and is installed as a blueprint module in other repo, can it pull secrets?

[drifting berry]

Greetings.

Just finished a deep security audit of container-use and honestly? This is the ONLY Claude Code tool I've seen that actually takes isolation seriously. Not "pattern matching theater" like other tools - actual containers with real sandboxing. Respect.

The Good (aka why you're killing it)

• Real container isolation - While everyone else is playing with bash hooks and hoping for the best, you're actually sandboxing agents. Each agent gets its own container + git branch. Chef's kiss.
• Git worktrees are genius here - complete audit trail, easy rollback, human review before merge. This is how you do state management.
• BuildKit caching - Parallel agents without destroying performance? Yes please.
• cu watch + intervention capability - Being able to drop into an agent's terminal when it goes rogue? This is what control looks like.

The Not-So-Good (security nerd concerns)

• That curl | bash install... (checksums?)
• MCP protocol vulnerabilities are inherited (see recent GitHub MCP exploit CVE-2024-MCP)
• No encryption layer (SOPS or similar come to mind)
• Git worktrees share .git directory - potential cross-contamination vector
• Early dev status disclaimer is... concerning for production use

Quick Hardening Suggestions

# Instead of curl | bash
git clone && audit && ./install.sh

# Max security mode
--security-opt=no-new-privileges \
--cap-drop=ALL \
--network=none \
--read-only

The Bottom Line

This is the first tool where I'm not screaming "WHAT ABOUT ISOLATION?!" - because you actually built it. It's not perfect (what is?), but the foundation is rock solid. Solomon clearly knows what he's doing (shocking, I know 😏).

For anyone running AI agents on real code, container-use should be your default. It's literally the only option with actual sandboxing vs security theater.

Would love to contribute some security PRs if you're interested - thinking signature verification for install, secrets management integration, maybe some container escape detection?
Also, that Dagger + BuildKit stack? Chef's kiss - this is what happens when people who understand containers build container tools.

Keep shipping! 🚀

P.S. - Running 10+ parallel agents without conflicts is just... beautiful. My inner chaos engineer is satisfied.

[winter linden]

Thanks for the kind words! Credits for container-use go to @cloud canyon @modest pier @swift inlet @coarse hatch . Dagger is an even longer credit list 🙂

We would love any contribution to CU or core Dagger! Thank you 🙏

[chilly arch]

The latest Dagger story is out!

Big thanks to @late kite for sharing how he cut CI/CD feedback loops from 30 minutes down to seconds with Dagger, and why he’ll never go back to traditional GitHub workflows.

🎥 Check out his story here: https://www.youtube.com/watch?v=0XArNGsQmF0

[tender orchid]

Hi folks! I'm pretty deep into our journey with dagger and really loving it so far (though discord is blocked on my work computer 👎). QQ: Can I define a default source of a secret in a constructor? e.g., in go:

struct MyModule {
    EcrLogin *dagger.Secrt
}

func New(
    // +default="cmd://aws ecr get-login-password --region us-east-1"
    EcrLogin *dagger.Secret,
) *MyModule {
    return &MyModule{
        EcrLogin: EcrLogin,
    }
}

[raven stag]

0.18.17 - Allow entering terminals from the TUI for Containers, Directorys and running Services by @narrow nymph

👏 👏

[narrow nymph]

hehe thought you might like that one 😁

had you in mind when working on it!

[fossil halo]

maybe i'm blind -- but i'm not finding any docs on how to interact with a list of objects in dagger shell -- say i have

▶ $my_env | outputs | .help
LIST OF OBJECTS
  []Binding

how do i interact with these outputs/inputs without knowing what they are? -- is there any way to enumerate/inspect them somehow?

[fresh pike]

The latest release broke my module constructor, which is loading a *dagger.File with the default name .args. That is in the .gitignore to prevent adding it accidentaly to git. This module is not loading anymore: failed to get value for argument "args". How to overcome of this? https://github.com/dagger/dagger/pull/10883

[wispy tapir]

Is your module only used locally in that case?

[fresh pike]

Yes, it's a local only module.

[wispy tapir]

Hmmm right now the .gitignore is applied by default, we assumed that we could rely on it but it seems we were wrong for that kind of use case

[fresh depot]

https://tenor.com/view/im-thinking-im-back-john-wick-keanu-reeves-gif-18424861702889754229

Hey ! It's been a long time !
New job here and i can finally use dagger professionally !

[sharp marsh]

welcome back!

[cunning jolt]

This is awesome! If they don't let you use dagger, change jobs 😄

[elfin frigate]

You can continue piping and the rest of the pipe will be called for each value iirc

[sharp marsh]

yep, xargs like behavior 🙏

[winter linden]

https://tenor.com/view/wwe-kevin-owens-randomly-flings-chair-are-you-kidding-me-seriously-gif-22199704

What an AMAZING FEATURE!!!

[fossil halo]

thanks, i need to try that, didn’t seem like it initially but awesome if that’s the case 😁

[late kite]

1/ this is really cool! 🙌 2/ i'm still hoping for something like | terminal | # continue pipeline with the mutated container .. although i also appreciate this may be a terrible idea and could make reasoning about caching weird

[narrow nymph]

yeah that's exactly why it's not supported today 😢

curious what your use case is though!

it would be good to solve the use case even if we don't make that API

[cunning jolt]

This is possible with code right?

withExec().Terminal().withExec() works

[late kite]

I think the output of the Terminal(). command will be its input container though? e.g. if you touch a file during the interactive Terminal, will it be available to the downstream .withExec()?

[cunning jolt]

Ah... maybe not. That would be cool!

[late kite]

yeah!... potentially. a core dev told me the original name for Terminal was BreakPoint.. or something like that, so for inspecting things.

[cunning jolt]

I do use Terminal a lot. But when I want changes/mutations I just add a withExec again before it. Having the ability to persist mutations within Terminal would save me a bit of time

[late kite]

I'm experimenting with using dagger to generate just in-time environments for "vibenvs" (ie Cloud Development Environments) .. the idea is to pull together just the right environment for a given task for a given operator, and then use | terminal to drop them into the environment to work on the task. when the terminal completes it'd be nice to take care of preserving their session etc .. although, volumes could work well for this too .. 🤔

[jagged flicker]

Yep, the touched files will not be available once terminal is closed. I wanted to have something similar so that the pipeline can ask the user for some inputs and continue. One way (one hack) would be to use a volume cache to persiste the file, as the volume can be used by the container but is external. And after the terminal is closed, the file should be available.

[late kite]

gotcha.. yeah i think that covers my use case

[loud briar]

Dagger behind NLB 🧵

[cerulean token]

anyone know of a dagger module that can execute a single GitHub Action e.g. actions/setup-go@v5

(I know that this trivial example that would be better done by a simple golang container, I'm targeting a more complex action in reality)

[sharp marsh]

hey there! we tried different prototypes on making Dagger being able to "natively" run Github actions but it in the end the surface to cover is so big that we have paused pursuing it since the benefits don't really justify the effort.

What has worked better for some other Dagger users it to incrementally start retrofitting Dagger in different steps / jobs withing that action pipeline and slowly start migrating from the things that benefit the most to the rest of the pipeline.

[cerulean token]

I've had a project of my own for running individual GitHub Actions via Docker that I've been wanting to swap to using Dagger (and in doing so also turn it into a Dagger module), but have been waiting on the functionality now provided by | terminal. If nothing exists as-is, I may go ahead in doing that. Hopefully the code I've already written covers a lot of that big surface area that you're describing 🙂

Thanks for the insight!

[winter linden]

Just to make sure we're talking about the same thing. You want actual runtime compatibility with the Github Actions platform? Similar to Nektos Act?

Or, you want to replace Github Actions job with equivalent Dagger functions, without exact compatibility?

[cerulean token]

Nektos' act runs entire GitHub Actions Workflows, I just want to be able to run individual Actions e.g. actions/setup-go

I imagine the interface being like

dagger -m github.com/my/module -c 'use actions/setup-go@v5 | toolcache'

dagger -m github.com/my/module -c 'use actions/checkout@v5 | workspace'

[winter linden]

You can try https://daggerverse.dev/mod/github.com/aweris/daggerverse/gale

But as @sharp marsh mentioned it's not actively maintained

[cerulean token]

Thanks! This doesn't provide the level of granularity that I need since it, like act, appears to only run entire workflows, but it will provide as an excellent reference.

Follow-up: does a Go Dagger Module have to be package main?

[sharp marsh]

Follow-up: does a Go Dagger Module have to be package main?

yes since the way it's designed today, all modules need to be callable. Is that impeding you from achieving what you need somehow?

[cerulean token]

Not at all, it was a totally inane question in hindsight. Thanks for answering anyway 👋

[winter linden]

Note that it's a limitation of the official Go SDK, not the engine. So an alternative Go SDK (or a fork of the official one) could totally have different rules and not require package main.

[torpid tulip]

Hi all! Currently trying to get some metrics going on our self hosted Dagger engine in K8s, specifically from this PR: https://github.com/dagger/dagger/pull/10555

However, it seems like there's no easy way (as far as I can tell) to add an additional container port to the container manifest via the existing Engine Helm chart. There is only .engine.port, which would override and prevent the metrics server from starting up on that port.

Wanted to get some thoughts on this, and if it would be feasible/possible to add potentially an 'additionalPorts' or similar config to the values.yaml?

Thanks! CC @mild coral

[loud briar]

@warm temple Any chance the proxy module can be updated to latest 🙏

[narrow nymph]

would absolutely be open to a PR that does that, for sure 🎉

[sharp marsh]

I can take care of this one. It shouldn't be that much work 🙏

[sharp marsh]

https://github.com/dagger/dagger/pull/11022

[warm temple]

updated! https://daggerverse.dev/mod/github.com/kpenfound/dagger-modules/proxy@428ffe16f0a2e3c9e4d0d9debf6420db29d3cb64

[cerulean token]

Note that it's a limitation of the

[tender orchid]

Does dagger run things in parallel by default (where possible), or do I need to use the language-specific concurrency (e.g., errgroup for go)?

[winter linden]

The engine always runs everything concurrently whenever possible. You just need to be careful that your client code sends requests concurrently when it makes sense.

Here's a cookbook article about that: https://docs.dagger.io/cookbook/builds/#execute-functions-concurrently

[tender orchid]

Ahh, ok. I did read that but maybe there was something I missed. I thought it only covered errgroup et al in a custom module, and not the default operation of dagger itself.

[winter linden]

Yeah that page could be more explicit. It's strictly about making your client requests concurrent, to take full advantage of engine concurrency.

[torpid tulip]

Thanks a bunch!

[grizzled fox]

Best updated youtube tutorial on dagger?

[warm temple]

what kind of thing are you interested in learning about? like github actions, go, typescript, python, etc? For the full picture, this one is great though its quite long https://www.youtube.com/watch?v=sviQX3SzpGc

[grizzled fox]

as I am going thought he tutorial I can notice that everything is called "dagger" 😄 I have hard time breaking it down on what is what, a dagger command that creates a dagger.json file that creates a dagger folder that is a dagger module

[winter linden]

Ha ha welcome to the dagger life 🙂

Are you saying the docs themselves use the word "dagger" too often which makes them less readable?

Or, are you saying that the product has too many concepts, regardless of docs?

[grizzled fox]

the product itself

[winter linden]

Ah. Well basically there is:

1. a CLI
2. a module system (configured by dagger.json)
3. an API which you can call from various SDKs

And that's about it

But yes, there are definitely many moving parts... You don't have to learn them all right away. It's like a lego system with many different bricks, you can incorporate more bricks as you progress

[grizzled fox]

here I am advising on small choices for new starters as I have a fresh mind

[winter linden]

Yes please!

[grizzled fox]

$ mkdir test
$ cd test && dagger new --name demo --sdk go 
$ ls -l  .  
.dagger/modules/demo 

something like that I would have expected now that I know the components

[winter linden]

ok. well it works in a way that is very similar

init instead of new

and each module is mapped to a single directory.

similar to git init

(or go modules)

[grizzled fox]

yeah I feel like is the go module requirement that is forcing the hierarchy

[winter linden]

Go modules are not a requirement, just a source of inspiration (along with git). Modules work the same way regardless of your choice of SDK

If you're using Go, your dagger module doesn't even need to map 1-1 to a go module. By default the SDK will set it up that way, but you can customize it to fit whatever works best for your Go codebase.

Same idea for other SDKs as well

[grizzled fox]

I arrived at dagger though the use container, multi agent video on youtube. I am just trying to understand all the components to generate these dependency graphs,

[winter linden]

Basically you can add context to any directory, in the form of functions. The source code and configuration for those functions is called a module. Modules can depend on each other, like a packaging system - but cross-language. So functions from module A can call functions from module B, etc.

Typically the context you add in a dagger module are things like:

• How to build this directory
• How to test
• How to deploy
• How to do codegen

This becomes exponentially more useful as your codebase grows and your toolchain is fragmented across siloes.

As long as each component in your codebase has a dagger module, you can always integrate them with a common system, instead of writing glue scripts

[grizzled fox]

so in theory devops should only worry about the cluster and the build kit provider, no more terraform for each of the project, if the build / deploy fails go fix the dagger code

[undone parcel]

I'm trying to run a dagger module which has a dependency on a private module in github.
I used following approach in my github-actions workflow https://docs.dagger.io/extending/remote-repositories#credential-manager-configuration

So basically in my gh actions workflow I do following

echo "token-with-access-on-other-repo" | gh auth login --with-token

Then when I run the dagger module I have following issue.

 git ls-remote --symref https://github.com/my-org/daggerverse HEAD HEAD^{} ERROR [0.2s]
15  : ┆ ┆ ┆ ! git error: git authentication failed

Does anyone know how to resolve this on a github actions workflow?

[rigid pebble]

Remote Repositories | Dagger

[tender orchid]

This is an interesting issue. Our company uses gh auth to set up authentication against GitHub repositories over SSH. When I try to include a dagger dependency on a module over SSH, it acts as if there's no valid key presented by the ssh-agent (so the dependency fails to download). I have to manually add ssh-add ~/.ssh/id-rsa to get it to work. I think the credential-helper osx-keychain does this for me when I clone on the host. That would explain why dagger doesn't know about this. Does that make sense? Any ideas on how to solve this?

[grizzled fox]

is the macos channel on discord active?

[winter linden]

I accidentally archived it this week... Just restored it, sorry!

We might change the docs in the future, to be less specific about which channel to ask.

[grizzled fox]

I am currently very interested in that setup, spinning containers inside mac-only environments

[winter linden]

@grizzled fox to be clear, dagger will spin up one native Apple container, running its own engine. Then from there, the engine will run all other containers the regular way, within that micro-VM

It has the benefit of not requiring a third-party installation of a VM-based container system, like Docker Desktop, Podman Machine, Colima etc. But, it's not orchestrating apple containers via the dagger API either.

[grizzled fox]

@winter linden like this ?

[winter linden]

almost, the dagger modules are run by the engine. the dagger CLI only does 2 things: 1) run the engine, 2) call the engine

[grizzled fox]

how is the module sent to the engine? the dagger engine compiles it in the vm from the dagger.json spec ?

[winter linden]

Dagger modules are a collection of dagger functions and types. They are all runnable by the dagger engine

So, yes

[tender orchid]

I have a module that has a dependency on another module, and my client project uses both of these modules. If the versions of the "shared/dependent" module differ, I get a runtime error that the pain "already exists with a different source". Sorry I don't have copy/paste, discord is blocked on my work laptop. Does this issue make sense? Any ideas or workarounds?

[winter linden]

Blueprints + local defaults is going to be an awesome combo

I'm playing with it in dev, and genuinely annoyed that it's not available in my stable release yet 🙂

[gaunt nova]

Hello, any chance to get some eyes on this? https://canary.discord.com/channels/707636530424053791/1417861965841436783

[torn wren]

I am loving the Dagger Shell and I am looking for some good examples to learn from. I found one section in the help shown below. Also, any plans for adding flow control aka if-statements and/or for-loops?

Shell variables: container=$(container | from alpine)
Shell functions: container() { container | from alpine; }
Job control: frontend | test & backend | test & .wait
Quoting: single quotes and double quotes have the same meaning as in Bash

[winter linden]

you can already use all the flow control features of the bash syntax 🙂

it's an actual bash syntax parser

[hidden dirge]

Hey folks, heard there was some discussion previously of the File.Contents() method. Looking to review and understand behaviour which prints the returned string to stdout. This is annoying when retrieving the contents of large files as it results in truncated logs.

Did a search on GitHub + Discord but didn't turn anything super relevant up

Closest being https://github.com/dagger/dagger/issues/10604

[fossil pine]

Don't know if it makes sense for Dagger, but I found this: https://secretspec.dev/

[elfin frigate]

what version of Dagger are you on? this was addressed recently already i believe

[mild coral]

0.18.16 atm

Morgan is with Nine 🙂

[elfin frigate]

hmm and where are you seeing the output show up? CLI output in CI?

the fix was in these two PRs, shipped in v0.18.13:

• https://github.com/dagger/dagger/pull/10617
• https://github.com/dagger/dagger/pull/10707

[torn wren]

I can't seem to get it to work, can you give am an example of a Dagger Shell script using flow control? Maybe I can figure out what I am doing wrong if I see a correct example.

[sharp marsh]

👋 here's a very simple example: dagger -c 'for i in {1..10}; do container | from alpine | with-exec sh,-c, -- echo hello $i | stdout; done'

[cunning jolt]

This is an interesting tool. IDK if it makes sense for Dagger though with the default secrets but it's cool!

[fossil pine]

Well, either as an integration or as a concept, it might be interesting to look at.

[torn wren]

Very good that works but I am still having trouble with if-statements. It always evaluates to false. The following was ran in the interactive dagger shell. What I am I doing wrong?

▶ i="foo";
if [ "foo" == "foo" ]; then .echo isTrue; else .echo isFalse; fi
if [ 1 ]; then .echo isTrue; else .echo isFalse; fi
if [ 0 ]; then .echo isTrue; else .echo isFalse; fi
if [ true ]; then .echo isTrue; else .echo isFalse; fi 0.1s
isFalse                                                                                                                       
isFalse                                                                                                                       
isFalse                                                                                                                       
isFalse  

[winter linden]

@torn wren I think if [ ... ] will need a little work from us, because that's actually a hack by the original bourne shell developers - if is part of the syntax but [ is not, it's just a builtin command. So we would need to expose that command

[torn wren]

Ok so avoid using if [ ...] for now but it looks like if <function> works

▶ if container ; then .echo isTrue; else .echo isFalse; fi 0.0s
Container@xxh3:766ce0fa9282de7c                                                                                                                                                                                                   
isTrue   

[torn wren]

Also the case statement works!

#!/usr/bin/env dagger

for i in {1..10}; do
  case "$i" in
    8)
      container |
      from alpine |
      with-exec sh,-c, -- echo hello $i |
      stdout
      ;;
    *)
      .echo "$i skipped"
      ;;
  esac
done

[winter linden]

Ha ha I realized there is also $RANDOM 🙂 Very handy for cache busting

[torn wren]

Please redirect me to and existing thread if this as already been discussed...

What's the best way to handle traditional CI "rules" in Dagger, like the conditional job execution found in GitLab?
I've been creating small Dagger functions and using a bash script to call them, but this gets complicated with dynamic conditions, such as running a different set of jobs based on the branch name.
My goal is to create a set of reusable Dagger functions, like "Lego blocks" that can be assembled based on the necessary conditions, rather than having one large, monolithic function. Is this the recommended approach?

[cunning jolt]

My goal is to create a set of reusable Dagger functions, like "Lego blocks" that can be assembled based on the necessary conditions, rather than having one large, monolithic function. Is this the recommended approach?
This is the approach I take. The branch based conditional logic we delegate to the CI orchestrator (Jenkins in our case).

You could have a monolithic function that wires everything together for local use if you want.

[cerulean token]

Trying to use https://github.com/vito/daggerverse/blob/main/docker/compose.go with a compose project that uses services.*.env_file, running into:

Failed to load /scratch/my_env_file

I see that it passes os.Getwd() to the loader. Any idea how I can shove my env files there so that it loads them?

Alternatively, is there another docker compose module that's preferable? I searched around but vito's looked best 😃

EDIT: forked and I think I've got this figured out now.

[winter linden]

Hi all, fair warning, there are several sleeper features that we snuck in the last few releases. We will talk about them in the 0.19 release blog post, but they are already live...

[winter linden]

Changeset

There is now a native type to describe the difference between 2 directories, at full resolution.

• Directory.changes() will compare 2 directories and return the changes between them
• You can convert a changeset to a git-compatible path
• You can apply a changeset to another directory
• You can programmatically inspect all the changes
• If you dagger call a function that returns a Changeset, the CLI will prompt you apply the changes to the current directory 🤯 . This makes Dagger 10x better for codegen applications (because it handles file removals etc)

Hey wouldn't it be cool if there was a script to compare 2 container images in diff format, instantly?

#!/usr/bin/env dagger

rootfs1() {
 container | from golang:1.22 | rootfs
}

rootfs2() {
 container | from golang:1.21 | rootfs
}

rootfs1 | changes $(rootfs2) | as-patch

@elfin frigate so dope

[hidden dirge]

Yep, cli output in non-interactive runs

Do we need to run with --plain frontend?

[elfin frigate]

it should be automatically doing plain mode in CI assuming it's running without a TTY - but actually, some CI systems do configure that, so maybe there's a gap in handling it. would you mind sharing a screenshot or something so i can confirm? either way, yeah you can try --progress=dots or --progress=plain (dots is the newer one, much quieter, I prefer it for CI)

[hidden dirge]

With

   -m workflow \
   --bao-addr=http://openbao.service-secret-manager-production-v1:8200 \
   --bao-token=env://VAULT_TOKEN \
   --ci=true \
   --github-token=env://GITHUB_TOKEN \
   --irsa-token=file://${AWS_WEB_IDENTITY_TOKEN_FILE} \
   --src="." \
   --ssh=/tmp/ssh-XXXXXXeWkkyw/agent.188 \
    call check-pull-request \
   --head-branch=XXX \
   --pr-id=114```

File in question is a fairly large JSON file

In GitHub Actions for context

[elfin frigate]

ohhh - that's not the output of File.contents, it's a gigantic argument value, yeah those don't have any special handling at the moment. it used to, but we removed it

if you set DAGGER_PROGRESS=dots or --progress=dots it'll go away, at least

[hidden dirge]

My bad! That should work for us for now though. Thank you ☺️

[elfin frigate]

might also be a good idea to pass those in as a file somehow instead of string args, especially since some of that looks like binary?

but i guess if that were already an option you wouldn't be turning it into a File 😛

[hidden dirge]

As a dagger.File or via native IO mechanisms?

Should just be utf-8 JSON afaik

[elfin frigate]

the JSON is fine, but that gpickle file looks spooky

within the module, you could write it to the current workdir, and then do (I think) dag.CurrentModule().Workdir().File("path/to/file")

[hidden dirge]

Oh yeah, we're supposed to be ignoring that

But guess it's not working there 😅

[elfin frigate]

https://docs.dagger.io/getting-started/types/#currentmodule has docs for Workdir, there's also WorkdirFile which seems preferable

[hidden dirge]

Yeah, still trying to wrap my head around how best to work with the abstractions provided by the Dagger API. Bit fuzzy sometimes on what return types make sense when trying to pass info between modules / Dagger functions

In this case we have a Dagger module to retrieve a file from GCS, that we need to do stuff with in the calling module

If I export to host from Module B, then I assume this is not accessible in Module A? I don't have a good intuition for how GraphQL API queries get mapped onto containers

[elfin frigate]

modules can't export to the host themselves - generally they return a File or Directory, and when you want to export you do that on the outside. (or return a Changeset for functions whose intent is more clearly to be exported, i.e. a generate function that runs codegen/etc)

my general rule of thumb is once you have a string or value that can be dynamically sized or become quite large, put it in a File so it can be passed around more efficiently

putting files in the function call's Workdir is a great way to do that, because then the file/directory can be passed around "by reference"

each function call gets its own little scratch space workdir for that exact situation

(sorry not 100% sure if that's what you were asking so tried to cover a lot of ground broadly 😅)

[hidden dirge]

Just looking at where this is happening , seems to be when we do
dir = dag.directory().with_files(path=".", sources=files)

Was probably a fairly vague question to begin with on my part!

So, function is returning dagger.Directory

[elfin frigate]

the root of the problem should be wherever that file("graph.gpickle", foo_content) call happens

could be some other function upstream

the reason it's an issue is that the TUI always represents object values by showing how they were constructed, so once you have a giant arg value, that arg value will show up everywhere that object shows up

[hidden dirge]

Right

Problem is here

   contents = blob.download_as_text()
   file = dag.file(contents=contents, name=os.path.basename(blob_name))
   return file

Is there an alternative way to create a new instance of File so we can return it to the calling module?

iirc there's a file from http function available in the core API

[winter linden]

yes you can do:

return dag.http(url=MY_URL)

[elfin frigate]

Either the http API if you can just fetch it that way, or write it to the local directory (like literally create a file with your languages stdlib) and use currentModule.workdirFile to reference it

Would be nice if there was a gcs module

[cunning jolt]

If you dagger call a function that returns a Changeset, the CLI will prompt you apply the changes to the current directory 🤯 . This makes Dagger 10x better for codegen applications (because it handles file removals etc)

Whoa, can I use this to save stuff like linter reports or test reports to local? Is there a flag to say, "don't prompt me"? Today I use --export for that and only return the relevant Directory/Files

[cursive bridge]

Is it possible to change functions from different modules together? Something like:

dagger -m module1 func1 | -m module2 func2

[elfin frigate]

have you tried out shell? you can reference multiple modules in one unix style pipe: https://docs.dagger.io/features/shell/#modules

[elfin frigate]

you can do -o . still, which should skip the prompt

[cunning jolt]

That still does the same thing as Changeset?

i.e. takes care of removals and additions?

[cursive bridge]

I haven't tried that. How would that fit into an external orchestrator like jenkins or gitlab? I'm thinking of this as the orchestrator is maybe forming the command or series of commands to execute.

[elfin frigate]

yep! (for a Changeset return value)

[cunning jolt]

ah!!! return Changeset instead and do -o nice!

[elfin frigate]

it's just another way to call functions, so it'd work wherever you're otherwise using dagger call. 🙂 it might look something like dagger -c 'module 2 | func2 $(module1 | func1)' (or you can put it in a script with a shebang, like #general message)

[cunning jolt]

You could create an orchestrator module and use that too. Which is what I do.

[cursive bridge]

interesting. So, in that scenario, how is data passed from module2.func2 to module1.func1?

[cursive bridge]

An orchestrator module would have to be updated for each new scenario, right?

What I'm trying to address is pluggable modules I guess. IE, I have one project that needs to build go + nodejs and produce a container. Another project is maybe c++ and go. Rather than have functions unique for those combinations, be able to have a go build, node build, and g++ build function and just chain them together

[sharp marsh]

it's the other way around. Data gets passed from module1 | func1 to module 2 | func2 since $(module 1 | func 1) gets resolved before calling func2. It's pretty much the same as using any shell and doing something like cat $(ls) (this will print the contents of all files in the current dir)

[cursive bridge]

Sure, but same question. How is the data actually passed. Positional args? files on shared storage? In memory?

[sharp marsh]

it's passed the same way as if you'd call those functions via code like module2.Func2(module1.Func1()). You'd need to make sure that func1 and func2 arguments and return types match

it's using the same underlying RPC mechanism Dagger uses internally for cross-module function calling

[sharp marsh]

yep this is the way. I'd suggest starting by separating the concerns within scoped functions in a module and slowly move them to specialized modules depending on their concenrs

so eventually you could have a Go module, NodeJs module, c++ module, etc

and then mix and match them in whatever way you want (shell, orchestrator module, etc) to build higher level pipelines

[cursive bridge]

That would be the plan for sure. It was how to link those together since you'd need build functionality from multiple modules. Creating an orchestrator would require functions for each combination, so it'd be more ideal to be able to link them together on the fly.

[winter linden]

@elfin frigate is the "grand fragmentation event" upon us? With build functions either returning a Directory or Changeset and devs asking which they should use?

[elfin frigate]

seems too early to tell imo, we're at the "devs realizing it exists at all" phase 😛

which one you use just depends on what behavior you want, it's not clear to me at this point that folks will overwhelmingly define both

Maybe it's just one big nerd snipe and not a real problem

[cerulean token]

Has anyone mentioned the use-case of a process inside of a Dagger container wanting to do an OIDC login flow before? e.g. az login which opens a browser for the user to login through.

I guess most CLIs that do that also output a link that you can go to manually, so you'd just need to expose whatever port the CLI is listening on for the redirect to the host?

[raven stag]

We do have this workflow for many of our modules and have not found yet a good devex around it... We haven't manage to do the oidc flow inside the dagger container in a seamless way

[sharp marsh]

@cerulean token I'd assume this is mostly for users running pipelines in their local machine given that it requires some level of interactivity in the login-process. Have you thought about using secret providers for this? i.e: you can do something like this today dagger call myfunc --token cmd://"az account get-access-token" which will pass the current azure access token from the user's config

in any case, secret providers is the way to go here so if you have custom login process within your org, coming up with a secret provider for that would be the best approach IMO.

[cerulean token]

That makes sense, thanks. It's a bit of a bummer that it can't be containerized (e.g. az has to be installed on the host), but I can't fathom an alternative

[winter linden]

we could ship a native AZ auth integration in dagger, like we did for 1password and Vault

just need to find a clean way to express it as a URI

[cerulean token]

Now that you say that, my actual use-case is vault (az was just an example that I came up with), so I guess this is solved for me.

I wonder if there's space for a plugin system so that these don't all need to be in-tree?

[sharp marsh]

not currently but we can make it happen. The current secret providers are pretty decoupled from Dagger core as they are today

[winter linden]

We decided to start in-tree because realistically, there are not that many providers needed to cover the majority

But we can always add it later, if there's overwhelming user demand for it (which at the moment there is not 🙂

@warm temple did prototype an external provider system built on dagger modules 😇

[sharp marsh]

the thing is that the az login flow is that is not desgined to be executed in a container. The fact that the az login could potentially listen an any available port in your machine makes it very difficult to containerize. That means that you'd have to run your containers with --net host or equivalent which is 😬

In any case I wouldn't recommend running any credential fetching commands within a container and/or Dagger mostly because of the caching. Whatever returns from that command will be stored in the cache without encryption

that's the reason why we've developed our Secrets and Secret Providers primitives to handle these cases

[cerulean token]

This makes a lot of sense. Thanks for the discussion guys.

If you guys do look into a native az Secret provider, another thing that would be nice is az acr login being able to map directly to a withRegistryAuth (same for aws ecr get-login-password)

As-is the best I could come up with was cmd://az acr login then getting the contents and parsing the plaintext contents as JSON, which outputs the token in the logs

[fossil pine]

Not sure if it's been mentioned here, but Dagger works well with GitHub Models:

OPENAI_BASE_URL=https://models.github.ai/inference
OPENAI_API_KEY=
OPENAI_MODEL=gpt-4o

Might be worth adding to the docs here: https://docs.dagger.io/reference/configuration/llm/

[winter linden]

oh nice! didn't even know those existed!

[fossil pine]

It's really cool, because you already have a token in GHA. Haven't tried it yet though.

[mild coral]

Hey guys. If I want to get the generated code for a module which is in the dependency tree of the module I'm currently in, is there a way to do this through the module api? On our mono repo we run go checks and unit tests that need the complete go project available. We are currently achieving it with dagger develop being run in experimentalPrivillegedNesting mode, but the performance isnt great. My thought is, the genned code has to already be here since the module has been loaded... how can I access this? Thanks guys!!

[winter linden]

Mmmm it's probably accessible from the API, you want to get the module's codegen as a dagger directory, to run operations against it directly in dagger, instead of running dagger develop then reading the exported files? Basically avoiding an export round-trip from/to dagger. Correct?

I believe you can do this with ModuleSource type, which has a generatedContextDirectory function

You can get a ModuleSource in a few ways:

1. Directly from a module ref (local or remote address). eg. dag.ModuleSource("github.com/my/module")

2. From a dynamically loaded directory, eg. myDirectory.AsModuleSource()

@mild coral can you clarify how you want to address the dependency module from your upstream module?

• By absolute path in the git repo of your module? (eg. /modules/foo/bar) ?
• By relative path in the git repo, from your module directory (eg. ../../modules/foo/bar)?
• By remote address? (eg. github.com/my/module/foo/bar)
• Or by name of the dependency in your current module's dagger.json? (eg bar)

[mild coral]

Hi Solomon that is correct. Okay the GeneratedContextDirectory is what I need most likely. What I was thinking of doing is evaluating all local dependencies by using CurrentModule.Dependencies and evaluating all dependencies of the module I am in. Basically all modules in our monorepo are in some way a dependency of the monorepos main module, so this will work for discovery.

Failing this, I would probably try a manual method of reading the dagger json myself and surfing the dependency tree. It would end up resulting in a bunch of absolute module paths which I would use to fetch directories for.

They would absolute in the context of the git repo of our mono repo.

If I succeeded evaluating all the modules, would GeneratedContextDirectory give both the generated code and the normal code and the SDK? Tried it the othrr day and I only got the generated code (dagger.gen.go, go.mod, go.sum) and no internal folder...

Could be doing something wrong or have a bad memory but just wanted to clarify.

[magic gorge]

I wanted to report this, as it seems like the Service Dashboard is green but when trying to go to Dagger Cloud, I get a WASM error

[sharp marsh]

Hi, which broswer is this? Can you try hard refreshing?

I can't seem to repro

[magic gorge]

Sure, it is Safari

[sharp marsh]

Mind trying if that happens in a different browser?

[magic gorge]

I will try it, also just getting developer tools on safari, newer computer.

Here is the safari wasm error

[magic gorge]

Confirmed it is localized to Safari browser, Google Chrome works

[sharp marsh]

Thx for sharing. Will check ASAP

[fossil pine]

Is there a way I can manipulate how Dagger resolves DNS?

I'm trying to get a module from a locally running git repo:

dagger call -m ssh://[e-mail]/laborant/test.git

gitea.local is in the hosts file

[mild coral]

Got it working @winter linden , thanks for the response 🙂

I was doing something weird which is why I had directories disappearing

[mild coral]

Just wondering would it be possible to somehow expose the already generated code from the running module to the module itself? Whilst GeneratedContextDirectory works, I know that the codegen was already generated when the module itself was loaded... so we are paying twice for it. I was surfing the dagger code and saw that CurrentModule wraps Module. Was wondering if it would be possible to expose the underlying module object?

A bit more context... I am wondering if it would be possible to use the Module object to get the generated context directory for any dependencies modules that are local in the chain. This is why I think it would be useful to access the Module of the CurrentModule.

I noticed that when you have a Module object, you can get the module dependencies mod.Dependencies. From there you can recursively drill down into any other dependencies they have. Along the way you would be able to hit GeneratedContextDirectory for any of these modules. Right now doing this just causes codegen to be run again on all modules. Feels weird since I know its already there as a result of the module being loaded. If it were possible to access the module object of the current module, would I instead get cached results for GeneratedContextDirectory?

Let me provide a bit more context.... for those wondering.

Our pull request check for our main dagger mono repo takes about 3m 30s - 5m when run locally and about 4m when run on CI. I have tasked myself with making it faster. Our pull request check spends a lot of time running dagger develop right now for each module. I was trying to see if I could cut the time down by avoiding running dagger in dagger and switching to GeneratedContextDirectory. It seems that there is not a huge increase in performance switching to GeneratedContextDirectory... hence looking for another solution.

Thanks for reading!

[cunning jolt]

mono repo takes about 3m 30s - 5m when run locally and about 4m
Is this with cache? How long would it take without dagger? (just curious)

[tender orchid]

My team and I are interested in incorporating our Go code from our platform product into a Dagger function. We spoke briefly about building the binary, then using dagger to build a container from that binary and run it. But since it's a Go project, what would it mean if we imported that code as a library in go.mod? Where does that code get run? Is there a difference between it running our custom code (referenced via go.mod) in a dagger function written in Go, versus writing a custom Go application and running it with dagger run?

[mild coral]

Yes that is with cache. The operations we are doing are:

1. running go test on every dagger module in the mono repo
2. running every dagger module's "module tests" - that is another dagger module which tests each individual dagger module
3. running the go linter on every dagger module

To do steps 1 & 3, we require the codegen to be present. To ensure consistency, we run dagger develop from within dagger at the start ( step 0 ) on every module. This takes more time than doing steps 1 & 3. This is what I am trying to optimise at the moment !

[winter linden]

Calling my go code from a dagger function

[cerulean token]

FYI dagger with the following env file:

OPENAI_BASE_URL=https://models.github.ai/inference
OPENAI_MODEL=claude-sonnet-4

Tries to talk to https://api.anthropic.com/v1/messages instead of https://models.github.ai/inference, while this env file:

OPENAI_BASE_URL=https://models.github.ai/inference
OPENAI_MODEL=gpt-4o

Behaves as expected.

EDIT: I incorrectly assumed that claude models would be made available thru GitHub since VSCode exposes them

[cunning jolt]

Time taken to run dagger module CI

[narrow nymph]

that's because claude-sonnet-4 is an anthropic model - you should use the ANTHROPIC_ env vars to configure, not the OPENAI_ ones

agreed that the behavior is confusing though

[eternal stag]

Hi folks - Name's Vlad 🙂 Im the organizer of a pretty cool and unique ctf called pwnctf.ro in romania. I was currious who i can reach for potential partnership 🙂 . thx

[cursive bridge]

Does dagger understand how to use .netrc to authenticate to repos? It would appear that it doesn't, or maybe I just don't have the syntax correct. On my local machine, I can auth to our git repos with creds in .netrc. However, dagger seems to get a 128 error code on the git output

[narrow nymph]

it should

which version of dagger are you on?

[cursive bridge]

dagger v0.18.19 (image://registry.dagger.io/engine:v0.18.19) darwin/arm64/v8

[narrow nymph]

https://github.com/dagger/dagger/pull/10957 added support for loading netrc

[cursive bridge]

I'm now trying to build a container image from a local Dockerfile and export that to my local container registry. I'm trying to use ExportImage via dir.DockerBuild().ExportImage(ctx, imageName), but I keep getting a "client has no supported api for loading image" error. Am I missing a config step somewhere?

[winter linden]

This is for exporting to a local container engine (eg. docker engine, podman etc)

For pushing to a registry, try Container.publish() instead

[cursive bridge]

Maybe I'm using wrong terminology here. I want the image to be available in my local docker instance on my host. I guess that is the engine? IE, I can do docker images and the dagger built container will show up

[winter linden]

Ah. Then in that case, yes, ExportImage should work. But, you're calling that from a module? So, the issue is that the module is sandboxed, and you're trying to access a docker engine running inside the sandbox

[cursive bridge]

I was thinking about that. How would that function be used? I wouldn't think the docker socket would be exposed in dagger.

[winter linden]

What you need to do, is instead have your function return the container. Then the caller (ie. you on the CLI) can do whatever they want with that container. In your case, chain a call to ExportImage

So, in your function:

func (m *MyMod) DoTheThing(dir *dagger.Directory) *dagger.Container {
  return dir.DockerBuild()
}

Then when calling it:

# Syntax 1: dagger call
$ dagger call do-the-thing export-image --name foobar

# Syntax 2: dagger shell
$ dagger -c 'do-the-thing | export-image foobar'

# Check that it worked
$ docker image list | grep foobar

[cursive bridge]

That worked

Interesting lesson for me. Not all object calls are intended to be used within a module. 😄

[winter linden]

Well, not commonly anyway. But there's nothing stopping you from running a nested docker engine in your sandbox container, and then you can access it from your module 🙂 Useful for bridging dagger code and "native" code.

The docker engine export is kind of a stretch, but local filesystem export is a more realistic example. For example if you module includes a native library that must interact with the local filesystem

That said, your confusion is pretty common, I would love to find a better way to avoid it.

[tribal zealot]

Do dagger modules set their own version of dagger? I noticed they come packaged with dagger.json files. In that case, could incompatibilities arise with the user's version?

[winter linden]

Yes a module sets the engine version it expects. The engine handles this to detect incompatibilities and warn the user. Also, the engine has a compat mode, to handle breaking changes. If module expects an older engine version, newer engines can "simulate it", so you are not forced to update your module to the latest engine version

Compat mode is explained in more detail in the 0.12 release post: https://dagger.io/blog/dagger-0-12

[tribal zealot]

Good to know, because I am having trouble using @warm temple 's proxy with dagger 0.18.18+, getting errors like this:

! start 493m98vuho62k (aliased as proxy): start dependent services: start 6ngnl8kmkofos (aliased as
  frontend): lookup 1673hev6tkf1e for hosts file: lookup 1673hev6tkf1e on 10.87.0.1:53: no such
  host
  lookup 1673hev6tkf1e.juh3n4fafi8lc.igehl49v9vcla.dagger.local on 10.87.0.1:53: no such host
  lookup 1673hev6tkf1e.igehl49v9vcla.dagger.local on 10.87.0.1:53: no such host

This was the first time I encountered version-specific behavior from a module.

[winter linden]

Looks like https://discord.com/channels/707636530424053791/1417948998454476840

[sharp marsh]

@tribal zealot this has been fixed recently in main. Should be out in the next release

[winter linden]

FYI @raven stag @rancid spire since I remember you've been asking for it for a while... @jagged flicker has been working on self-calls 🙂 Shipping as an experimental opt-in feature in the next release

[cunning jolt]

Count me also in that camp! Self calls would save me a ton of jankiness

[plush monolith]

I finished my first pass at migrating the Headway[^1] build stack from Earthly to Dagger.

Headway is an open source self-hosted map stack built on OpenStreetMap and other open data.

The build system outputs several artifacts (maptiles, routing, geosearch) and the service containers that consume them.

The migration wasn't exactly straight-forward. It took about 100 working hours over the course of 2 months, but I'm fairly happy with the end results. The folks here were super helpful. I couldn't have done it without you! ❤️

Some challenges that stand out:

1. I don't know golang, but I was too afraid to try the "experimental" rust bindings. I wanted to stick with something well tested.

2. The Headway build is kind of weird — primarily the long running nature of our build process (can be several days!). I ran into challenges and edge cases unique to our setup when using Earthly too.

3. Using an existing docker-compose system "magically" worked with Earthly, but it was more trouble than it was worth with dagger.

Some nice things:

1. Having a "normal" programming language (rather than Earthfile syntax) is great!

2. WithMountedFile/WithMountedDir is a nice speedup since I'm frequently passing around 100GB files (like planet.osm.pbf). I think in Earthly, I would often end up with multiple layers caching the same huge file, which was slow to copy and also caused my build process to balloon to terabytes of disk space.

3. (repeating myself) this support forum has been really wonderful!

[^1]: See a demo at https://maps.earth

[winter linden]

That's great, thank you for sharing your progress! It's always motivating to see real world usage 🙂

I'm curious about the docker-compose part, and if we could improve this in the future. It looks like it boils down to a missing compat layer?

[plush monolith]

One of the artifacts I produce is an elasticsearch database for my geocoder ("pelias"). The upstream project provides a docker compose file which encapsulates the various services you need to populate the geocoder, and some invocations that utilize that docker-compose.yml.

It was simple to translate their "run this in your shell" instructions to earthly, because I could reference this docker-compose file directly from my Earthfile.

However, with dagger, I ran into some trouble. Ultimately, I got rid of the docker-compose file and reimplemented much of the logic in golang/dagger directly. The end result isn't so bad, I just wish I hadn't wasted so much time trying to make it work the old way. It seemed close.

One thing I'm still missing with the dagger version are these declarations you can easily do in docker-compose:

    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add: [ "IPC_LOCK" ]

[cerulean token]

One of the artifacts I produce is an

[plain grail]

hi, can someone help me with Services? I would like to run 2 services and get ports on my host for each one but without success 🤔
#1421034561269141504 message

[sharp marsh]

hey there! just replied in the issue

[noble ingot]

Hey guys. I'm having trouble trying to configure a module as a remote dependency but I think I'm not following the docs or maybe doing something wrong..
I am trying to authenticate to github using a github app essentially as a service account so that it isn't tied to a specific user's account or PAT.
When trying to set this up locally, following the info on the site, I authenticate to github using gh auth login and HTTPS auth is supposed to just work as far as I understand it. If I run dagger install github.com/org-name/private-repo though, I see an exit status 128 error and this message:
remote: Invalid username or token. Password authentication is not supported for Git operations.
Any clue what I might be doing wrong?

[cunning jolt]

Dagger module in private git repo

[balmy cedar]

Hi, are interactive commands (sops edit in my case) supported in a dagger shell ? with-exec hangs and crashes, like https://github.com/dagger/dagger/issues/11095.

The cache + terminal hack (https://discord.com/channels/707636530424053791/1310441918630592662) helps but it's not ideal.

Thanks

[golden patrol]

@moderators the user @gabrielcrusk83 posted some spam across several help thread advertising his linkedin/fiverr etc

[winter linden]

Thanks, we're on it but having issues with the discord UI

(moderator tools not working)

Should be done now

Please let me know if I missed one of their messages! (had to manually purge it)

[golden patrol]

@winter linden thanks for the quick response, while i have you here 😅 any way to get access to that mentioned tech preview? 🙂 currently running our own build agents on azure but currently migrating to run them in our own cluster completly

[sharp marsh]

fixed 🙏

[narrow nymph]

hey @whole sparrow! do you have an example of what kind of thing you're running?

20+GB does sound like a lot to me, but also toolchains (e.g. gcc, clang, etc) do have a tendency to be quite large

[jagged flicker]

You can use dagger core engine local-cache prune to prune the cache, and dagger core engine local-cache keep-bytes or dagger core engine local-cache max-used-space to control the maximum bytes to keep in the cache.
It doesn't solve your issue but at least it allows better control on the cache and to deal with it without to delete the container

[cerulean token]

Is there any way to pass the executing dagger engine into a container as a service binding for dagger-in-dagger? something along the lines of

container.WithServiceBinding("dagger", dag.Engine())

if not, any suggestion on how to run a separate dagger engine as a service and access it?

[winter linden]

It depends what you're trying to do. Are you simply trying to do "dagger in dagger", in the simplest and most performant way possible?

[cerulean token]

yeah, pretty much. just want a Dagger Engine attached to my container and the env var to configure dagger to use the aforementioned Engine

[winter linden]

So, Dagger supports nesting natively, so it's even easier than that: just set the argument experimentalPrivilegedNesting to true when executing your dagger client (whether the Dagger CLI or another client). And the client will automatically connect to the nested session.

(We are actually about to deprecate that argument, to make nesting the default).

Once that's done, there will be literally nothing to do to get nested dagger-in-dagger

[cerulean token]

That's super nice, thanks!

[winter linden]

Yeah it's a very powerful feature, I wish we did a better job explaining it. Lack of clean nesting has always been an issue that plagued us in the early days of Docker. This time we did it right 🙂

[pliant summit]

when running a container-use for codex/claude/etc. does it actually spin up a sandboxed container or is it just a worktree?

for example does it have access to my OS / system tools, or only the image, and if the agent installs utils / tools is it sandboxed from doing anything on my local machine?

my idea is i could make an image that has all the tools i develop with and then let the agents run in that with container use

[winter linden]

Fully sandboxed in a container

[pliant summit]

lastly i’m curious about the cpu/mem allocation to a container use—i basically run a bunch of rust projects in parallel with worktrees now (ie. compiler take a while lol), and i want to make sure the sandbox is giving the programs i use in them max resource for cpu/mem.

[warm temple]

what container runtime are you using? docker desktop?

[pliant summit]

yes, but open to switching.

is it whatever i specify in docker desktop settings and the container will take some portion from that? i don’t fully understand how docker does the resource alloc anyways haha, so maybe need to read on it

[warm temple]

the only resource allocation for container-use would be whatever docker desktop has (thats where the dagger engine is running), so if you go to settings -> resources you can see what the VM has configured. For alternatives, I'm not totally sure how podman handles it, and apple's beta container thing allocates resources per container, so you'd set specific allocations for the engine

[whole sparrow]

I think I narrowed down what mostly seems to cause it: Its primarily the export process.

But, as you said, its quite large so not much can do to avoid that beyond clearing the cache

I think what I'll do as a workaround is seperate it from a single cli app into dagger functions, and clear the cache between them

[cursive bridge]

@narrow nymph Did the work you did with netrc auth also include inheriting the host system's config similar to how git config is handled?

[winter linden]

We were just looking at that code with @swift inlet , so I know the answer is yes 🙂 Currently works out of the box for dag.Git(), and we are in the process of adding it to SDKs also, so they can use your host netrc to download dependencies

[cursive bridge]

Is there a GH issue or something I can watch?

[swift inlet]

https://github.com/dagger/dagger/discussions/10921

[torpid tulip]

Hi Marcos, just wanting to follow up on the status of this PR since I notice that the pipeline is failing on it and that it might get closed soon :)
Not a huge rush to get it through, but would be nice to see!

[sharp marsh]

Sorry. Seems like the reviewer forgot about it. I'll poke @rigid pebble about it tomorrow

[winter linden]

So this is a fun experiment 😇

https://github.com/dagger/dagger/pull/11156

An experimental lockfile for Dagger, for repeatable & offline builds

[tired moth]

Does dagger going to support lightweight container runtime or wasm based runtime like this? https://github.com/project-ocre/ocre-runtime

[narrow nymph]

not currently, do you have a specific use case in mind?

[tired moth]

to use containers and devops on low end devices like raspberry pi and edge/IOT devices

[narrow nymph]

dagger should work on the raspberry pi - just as normally as docker does

but we're very much tied into linux systems, super low-end devices aren't really the current target

[tired moth]

Linux lxc will be supported as such dagger added support for apple container which is based on Linux lxc only?
So if Linux lxc directly will be supported then also it can be a game changer for us.

[tired moth]

Any issue to track?

[sharp marsh]

merged @torpid tulip

[torpid tulip]

Thanks again!

[winter linden]

Hey everyone Dagger 0.19 is out! #announcements message

It has a ton of improvements, on all fronts.

If you're interested in the agentic features, definitely try the monster improvements in the "build-an-agent" APIs, you can now legit build your own Claude Code on top of Dagger's API, with very little code. Fully integrated with Dagger modules of course 🙂 So your agent will be the only one on the market that can actually run e2e tests reliably as it develops - that's the hot new thing to do

This ⏬ is 100% build on Dagger.

Also if you've been using Dagger for file generation workflows (codegen, docs generation, etc) then Dagger now makes it much easier.

There is a new Changeset type that represents a diff as an artifact. So your functions can programmatically construct diffs, and return them. When you dagger call a function and it returns a Changeset, the user is shown a diff, and prompted to apply it.

So you can do dagger call generate and get a really nice interactive UI that just applies the codegen.

We're in the process of incorporating that for our own needs

[cunning jolt]

Can we get a dark mode on the blog site please? 🙂 Just got blasted by light mode 😄

[winter linden]

https://tenor.com/view/light-blind-blinding-light-too-bright-open-curtains-gif-16971259

Our X post if you feel like amplifying: https://x.com/dagger_io/status/1973459838123585884

And a spicy hot take, just for fun https://x.com/solomonstre/status/1973462124006613393

Solomon Hykes (@solomonstre)

"Coding agents" aren't standalone products. Rather they are features to embed within a CONTEXT. The quality of the context determines performance.

Speaking of.. The new Dagger can run agents in the context of a perfectly repeatable e2e test environment.

May the best context win

X · Oct 1, 2025, 6:56 PM

[cerulean token]

👋 Does anyone think that the Dagger engine would be able to export just the manifest (or just a specific layer) of an image any faster than it is able to export the entire image and I am able to parse the manifest from it using something like go-containerregistry?

EDIT: and, if so, am I correct that this functionality is not currently exposed?

[pliant summit]

anyone able to help me with this error when invoking codex?

MCP client for `container-use` failed to start: request timed out

i followed the instructions here: https://container-use.com/agent-integrations#openai-codex

and my ~/.codex.config.toml is this:

model = "gpt-5-codex"
model_reasoning_effort = "high"

[mcp_servers.container-use]
command = "container-use"
args = ["stdio"]
env = {}

to be fair, when invoking container-use stdio i also just get a hang. no output. so i think this is the root cause.

[smoky smelt]

Hey,, might be noob question but is it possible to use this as a module where on top of that I can add custom logic like daggerverse modules?
https://github.com/samalba/agents/tree/main/dockerfile-optimizer

[steel sail]

What's the main difference between "TypeScript Custom Application" and the other way, whatever the proper name is (a module?).

Is there a more correct way to use Dagger?

What's the correct way to use it in CI?

I like the "TypeScript Custom Application" way, as you are installing @dagger.io/dagger from NPM pre-built, not having to codegen it. But I see it is also a slightly different syntax, where you need to wrap into a connection with a top level await.

If I am working on CI/CD and plugging it into GitHub Action, which way is the preferred and why?

Thanks.

[sharp marsh]

hey @steel sail ! using modules is generally preferred as they encompass a lot of goodies around runtime and DX aspects which are constantly being improved to deliver the best out of the box Dagger experience.

"custom applications" are generally best suited for more advanced use-cases where some teams or companies already have some custom tooling which they'd like to extend with some dagger capabilities by using their preferred SDK without leaking the Dagger specific DX into it.

LMK if that makes sense

[steel sail]

Thank you. Makes sense. I wish though there was a way to use modules with npm installed dagger package.

[sharp marsh]

You can. It's experimental and not documented yet.

https://github.com/dagger/dagger/pull/9531

[whole sparrow]

Question .Is it possible to disable cahcing such that anything created is cleaned up/. deleted completely after a client disconnects?

Or, possibly a way to set the cache timer, that after X amount of period, it gets cleaned up? Perhaps I'm over thinking it

[sharp marsh]

The way I'd suggest to achieve this is by using a cache buster at the very beginning of your pipeline as described here: https://docs.dagger.io/cookbook/builds/#invalidate-cache

Have you tried that?

[whole sparrow]

I have not

[steel sail]

@sharp marsh That PR sounds like the opposite of what I was talking about, or am I mis-understanding it?

Import the generated client and use it like if the library 🙂

I want to be able to install @dagger.io/dagger from NPM and not have to codegen anything.

[raven stag]

Having a pragma for functions, along the lines of this, would be

// My function
// +cacheTTL=0
func (m *Test)  NeverCached() string { return "hello daggernaughts" }

I was checking the issues and I couldn't find one. I think this is not the first time this has been raised.
I'm sure it is not a straight forward feature to implement (dagger has done a lot of good work on having a very aggressive caching system 😄 )

[cunning jolt]

This!! https://github.com/dagger/dagger/pull/10975

[raven stag]

Thanks @cunning jolt ! I knew it existed, i'm just bad a searching 😛 (we've discussed with dagger many times over this exact use case)!

[sharp marsh]

if you want to use module dependencies in "custom applications" you need some sort of codegen since your local code should be able to dynamically expose all the functions your dependency modules provide for the SDK you're using.

[cerulean token]

couldn't you craft the graphql queries yourself rather than codegenning?

[sharp marsh]

yes you could do that but it also requires managing the module registration in the engine yourself. It is possible, but generally not recommended

[cerulean token]

<deleted because I saw it was discussed above>

[cerulean token]

any chance the good first issue label referenced in #welcome message can be revived? 🙂

[whole sparrow]

Sweet. Hope to see it merged

Would alleviate some of my issues

[winter linden]

Yes we have been discussing it forever but it's finally happening

We have A LOT of good stuff brewing. They're bigger changes so take a while to review and polish. But it's coming 🙂

@whole sparrow @raven stag @cunning jolt FYI here's the leading candidate for cache control DX:

+cache=never|session|DURATION

Go Examples:

• Maximum caching: nothing to do (default)
• No caching: +cache=never
• Cache only for the current session +cache=session
• Cache for 1 hour :+cache=1h

[loud briar]

Sounds like the current caching is the maximum level of caching? Could e.g. +cache=session force caching where it otherwise wouldn't? Say if Dagger wouldn't cache a function, but you know it's safe to cache for the session?

[raven stag]

This looks amazing.
Would +cache=never avoid creating layers of caching I assume? We have many pipelines that generate GB of cache layers that are impossible to be re-used (from the nature of the pipeline), but they contribute to the ever growing size of the cache. This would be a massive improvement for us 🙂

Also +cache=DURATION - We have many hacks with env vars doing the BUSTING_CACHE type of workaround, when we would be fine caching for a few minutes, of for the same session.

Super excited with the new features brewing!

(fyi: @rancid spire )

[steel sail]

Sorry, I cannot parse this. I don't have enough context/intuition yet around this.

Here's my goal, in other words:

I want to be able to "git checkout" and then "tsc --no-emit" check the code for the Dagger project to validate it. Also ESLint check it in CI.

As it is right now, I have to first run dagger develop which isn't desireable, as that means I have to adjust my entire build process just to get the types for Dagger in, which are already technically published in the npm package.

I'm still confused why I need to do this.

[sharp marsh]

I want to be able to "git checkout" and then "tsc --no-emit" check the code for the Dagger project to validate it. Also ESLint check it in CI.

There's multiple ways you can achieve this.

1. You could commit your generated code. This is something some users do since they're expecting a specific flow to be possible with the dagger generated code to be present
2. Make that tsc --no-emit a dagger call target. Ideally it's desired to have this things in Dagger so you can have strong guarantees of the environment that's running these kind of checks

I'm still confused why I need to do this.

are you using modules or "custom applications"?

when you're using "custom applications" you only need to generate code if you're expecting to call some third party modules. If that's not the case, you can use the dagger SDK directly via the @dagger.io/dagger package and it should be ok

[steel sail]

Using modules

I was considering committing the codegened code but it seemed quite verbose. And I was/is cobfused why I need it if I can npm install it 😀

[sharp marsh]

yes, that only only works for the core types. The moment that you need some additional types from a third-party module (modules are designed to be composable), then you'll need code generation. So having a solution that only works as long as you're just using the core types doesn't make too much sense from the design perspective

[winter linden]

Codegen questions

[whole sparrow]

I like this. perfectly would fit the use case

[winter linden]

MERGED! https://github.com/dagger/dagger/pull/11034

[winter linden]

and, released. 0.19.1 is available with user defaults 😁

[tender orchid]

Hi folks! I have a coworker that's showing some weird behaviors when running dagger-engine v0.18.19 when running behind a corporate VPN. We have a tool that wraps dagger and copies certificates to ~/.config/dagger/ca-certificates. On my machine dagger is able to pull images, but on his machine it fails with an x509 "certificate signed by unknown authority". I'm not quite sure what else to check. Any thoughts?

[winter linden]

Hi! One possible cause is that the certs are not kept in sync at each execution of the dagger CLI. They're synced at provisioning of the engine by the CLI.

So it's possible that your coworker changed the cert configuration and since then the engine has not been re-provisioned

[cursive bridge]

I seem to be having some issues using non-root containers. Directories seem to default to being mounted with root ownership. While there are options for ownership, it doesn't seem like those are recusive. IE:

WithMountedCache("${GOPATH}/pkg/mod", moduleCache, dagger.ContainerWithMountedCacheOpts{Expand: true, Owner: user}).

Will result in ${GOPATH}/pkg being owned by root, but mod being owned by user. Am I missing something?

Ah, maybe I need to use WIthUser?

[stoic knot]

Are we an unusual user setup if we run the same container for dagger-engine for weeks to months on end?

[cursive bridge]

I seem to be having some issues using

[tepid forge]

Nice. However v0.19.1 seems to break +defaultPath when used with +ignore:

With v0.19.0:

func (dev *MyModule) MyFunc(
    // +optional
    // +defaultPath="/"
    // +ignore=["*", "!src"]
    source *dagger.Directory,
) error {
    ...
}

...
● myModule: MyModule! 0.1s
● .myfunc(
  ┆ source: Host.directory(path: "[redacted]/my-project", exclude: ["*", "!src"], noCache: true): Directory!
...

With v0.19.1:

...
● myModule: MyModule! 0.1s
● .myfunc(
  ┆ source: Host.directory(path: "[redacted]/random/dependent/module", include: [".env"]): Directory!
...

Seems like the source of a random dependent module is passed instead. Will open an issue asap.

[cursive bridge]

Maybe this is what caused 0.19.2?

[tepid forge]

The issue is also present in v0.19.2. The release notes only mention an issue with a crash when container sourced from private image.

[swift inlet]

Nice. However v0.19.1 seems to break `+

[candid gyro]

Just stumbled upon this while searching a bit before I'm about to do a "GH checks" abstraction in my dagger pipelines (pretty much exactly what Mark mentioned he wants). Curious what the take on this is now, a year later? Any relevant upstream issues I should check out maybe ?

[winter linden]

We're refactoring our own CI to incorporate all the best practices we've learned and observed in the wild. I'm working on it right now 🙂 There's definitely a top-level concept of "checks" emerging. Basically Dagger functions to be used as entrypoints for your CI

We also greatly improved the observability features in Dagger Cloud

[candid gyro]

Cool, I'm mostly trying to figure out a better way to communicate the top-level results of the pipeline, since the full Dagger log is often not very useful for people who don't know Dagger well.

The trace view in the cloud is better, and I like the improvements with the colors and emphasizing certain spans and hiding noisy spans on error etc. It's still quite verbose and IME people actually really like the high level view (i.e. is it the build, the test or the deploy that failed, etc).

What I would personally want for a "check" feature is maybe more of an in-engine/in-SDK abstraction (with ideally an integration to GHA by default, but maybe other platforms too), and an API where you can basically say e.g. dag.ReportResult(...) to reflect in the platform that e.g. the (abstract) "test" part of the pipeline is running / succeeded / failed, etc.

If each of these abstract results/goals/artifacts of the pipeline have to be run as a separate entry point, I would personally probably not find that very useful, since I'm more and more preferring to keep the GHA workflow as short as possible and instead make dedicated high-level entrypoints that basically trigger the full pipeline. I prefer that because I can orchestrate and parallelize much more easily than the equivalent multi-job multi-step GHA worfklow (calling different dagger functions) that I would have to maintain.

Maybe I'll try and have a look at your CI structure to get inspired 🙂

[winter linden]

Yes I agree. My thinking is to use our CI as an incubator for a first-class concept of "check" as a thin opinionated layer on top of functions.

Here are 2 relevant PRs:

• Simplify our CI: https://github.com/dagger/dagger/pull/11161 (WIP)
• A POC of a native dagger checks command : https://github.com/dagger/dagger/pull/11211

@candid gyro if you're going to look at our CI as an example, I recommend looking at that PR (11161) rather than main... We have a lot of legacy in there, since we started using Dagger very early, when it was much less polished. Then over time cruft accumulates, you know how it is..

[candid gyro]

I'll have a look, thanks 🙂 Not exactly related, but the util/parallel package looks really useful for what I'm doing right now too

[winter linden]

oh yeah, it simplifies our code so much.

We were just discussing how to make that a native feature too.

it's tricky because of callbacks and generic types, but might be doable with some pragmatic constraints

[candid gyro]

yes, it's what kinda brought me to look into reporting the GH checks, because it's a bit hard to properly wait for all the parallel runs and return both their errors and outputs in a way that's not super confusing.

I haven't tried the the custom span trick. That might improve the trace view in dagger cloud enough to just send people there and not have to bother with how the final lines of the GH log look like. (.. but reporting checks would be nice either way)

Another, very different approach, I was experimenting with is to basically not use dagger CLI in CI, but just bake it all into a custom CLI that uses generated clients and WithLogOutput(io.Discard) to just produce separate high level logs itself to keep the GH workflow output clean and for anything more detailed you go to the Dagger Cloud trace. Bit too much of a rewrite from the current state just for log aesthetics for now though.

[winter linden]

If all you need is to suppress the logs, you can do dagger -s

[candid gyro]

The issue is I do want logs, just at a way higher level (basically only ones that I print). I believe with dagger -s the only "logs" I can produce is a string result returned at the end of the execution. I guess having leveled logs with the ability to only silence lower levels would make that work

[winter linden]

ah I see

yeah it would be cool to have a hook to customize that

you could wrap the dagger CLI in a custom otel renderer. We honor the standard otel collector env variables

[tired moth]

git diff supported natively in dagger? if yes then anyone can please share docs link, i have tried but couldn't find it...

[digital tundra]

Can someone please generally advise if this is a supported pattern or not? I've tried many different combinations of APIs and it seems like the cache behavior ultimately depends on the context set by the Directory passed, and it can only be manipulated unidirectionally for the first argument decorator it finds. Any help is appreciated.

import { dag, Container, Directory, object, func, argument } from "@dagger.io/dagger"

@object()
export class DaggerRepro {
  /**
   * Base func needed by other functions. It only needs a small number of files
   * to run `pnpm install`, so we only copy those files to the container.
   * This should be cached based on the contents of those files only.
   */
  @func()
  base(
    @argument({ ignore: ["**", "!package.json", "!pnpm-lock.yaml"] })
    source: Directory
  ): Container {
    return dag
      .container()
      .from("node:22.17")
      .withExec(["corepack", "enable"])
      .withWorkdir("/app")
      .withDirectory("/app", source, {
        include: [
          "package.json",
          "pnpm-lock.yaml"
        ],
        exclude: ["**/node_modules/**", "dist"],
      })
      .withExec(["pnpm", "install", "--frozen-lockfile"])
  }

  /**
   * A sample function that reads a specific file from the source directory
   * and returns its contents as a string. It uses the base function to set up
   * the container environment. Should be cached based on the contents of the
   * relevant files only.
   */
  @func()
  async myFunc(source: Directory): Promise<string> {
    const base = this.base(source)
      .withDirectory("/app", source, {
        include: ["relevant-to-my-func/**"],
      });
    return base.withExec(["cat", "relevant-to-my-func/file.md"]).stdout();
  }
}

[sharp marsh]

Can someone please generally advise if

[torn wren]

is there a way to ignore certain directories in a dagger scritp? I see you have pre call filtering which is what I am looking for but I want to do the filtering in the dagger scrip, what are my options?
https://docs.dagger.io/getting-started/types/directory/#pre-call-filtering

[winter linden]

What do you mean by "in the dagger script"?

[torn wren]

in a dagger script eg ‘dagger -c <a dagger script>’

[winter linden]

ah I see.

yes you can call host | directory --exclude=foo,bar --include=bar,baz

So, something like:

#!/usr/bin/env dagger -m MYMODULE

filtered_source() {
  host | directory . --exclude=node_modules,build/
}

my-function --source=$(filtered_source)

[hardy umbra]

How are people handling the developer experience in github actions?

We are stuck either showing the developer all the dagger output which they do not understand/need or using --silent and showing them nothing. Preferably we want our dagger step to fail with an error but that means we have very few options in regards to how to present the output to the developers.

We are considering outputting a string that contains an exitcode and parsing that in another step but this seems brittle.

Is there some pattern we are missing here?

[raven stag]

How are people handling the developer

[torn wren]

my .env file has export FOO=bar and all is working, then I upgrade and now dagger gives me this error?

▶ connect 0.3s
▼ detect module: . 0.2s ERROR
! load user defaults: Evaluate env file: parse error: "export FOO=bar": not a bare assignment
╰╴✘ moduleSource(refString: "."): ModuleSource! 0.1s ERROR
  ! load user defaults: Evaluate env file: parse error: "export FOO=bar": not a bare assignment
Error: find module ".": load user defaults: Evaluate env file: parse error: "export FOO=bar": not a bare assignment

I know I could remove export to make dagger happy but I want to export the variable so that it is available in my bash script.

I might be confused but looks like dagger now "owns" my .env file 🙂
What do you suggest I do in this case?

[cunning jolt]

Isn't export an .envrc thing? I don't think i've see export before in a .env file. It's always KEY=VALUE

[winter linden]

@torn wren sorry that's a little aggressive... We should find a more graceful way to handle it.

But also, yes, it appears your .env is actually incorrect in this case

[torn wren]

ok if my .env is incorrect then I will adjust. It just caught me by surprise. Thanks for the quick responses.

[winter linden]

Sorry about that.. I didn't anticipate such a jarring "takeover" experience, but now it makes sense...

[torn wren]

That worked! That is very useful.

[winter linden]

What I worry about, is a situation where an actual correct .env causes this kind of error with dagger.

(There is no formal spec for .env so each tool does what it wants, within the bounds of a loosely defined convention)

[narrow nymph]

imo, we should allow export. it's pretty common, since it allows loading it into your current shell session using source (that's what i often do)

[winter linden]

does any .env parser allow this? I don't think I'v ever seen it

[narrow nymph]

dunno? but it's valid shell syntax 🙂

but as you mention there's no spec

[winter linden]

lol

great

maybe we should have stuck with our own json format

[cunning jolt]

Won't hate it. .envs are for the most part .gitignored too and a ton of hesitation enterprise wide, checking them in. A dagger specific "env" file would be totally acceptable to check in as it won't contain secrets. My current plan is to upload as a .env.example and let consumers rename.

[loud briar]

I'd probably prefer that. I'm currently not committing .env and essentially saying "this is a convenience factor for local usage" and our CI jobs still have all the arguments. It'd be too easy for me to take .env out of .gitignore then someone commits some actual secrets without even noticing

[winter linden]

well one reason we're using .env is specifically to piggyback on the convention that you probably won't commit it.

So if we add a custom json file, you should also not commit that file

[loud briar]

No-one is putting tokens/secrets/etc in a Dagger-specific json, I'd commit that to Git

[wheat mango]

I am trying to learn writing dagger module

import { dag, Container, Directory, object, func } from "@dagger.io/dagger"

@object()
export class CiDagger {
  /**
   * Builds the Go application and returns a production-ready container
   */
  @func()
  async build(src: Directory): Promise<Container> {
    // Build app
    const builder = dag
      .container()
      .from("golang:latest")
      .withDirectory("/src", src)
      .withWorkdir("/src")
      .withEnvVariable("CGO_ENABLED", "0")
      .withExec(["go", "build", "-o", "myapp"])

    // Create production image on alpine base
    const prodImage = dag
      .container()
      .from("alpine")
      .withFile("/bin/myapp", builder.file("/src/myapp"))
      .withEntrypoint(["/bin/myapp"])

    return prodImage
  }

  /**
   * Publishes a container to a registry
   */
  @func()
  async push(container: Container): Promise<string> {
    return await container.publish("ttl.sh/myapp:latest")
  }
}

I was trying invoke with

dagger call cidagger.build --src=https://github.com/golang/example\#master:hello push

but i got error ! unknown command "push" for "dagger call build"

[winter linden]

dagger call

[tired moth]

Can we connect ENV feature directly with github repo env to sync for devops with dagger while developing locally? if dagger can support this natively then it can help devs a lot

[cerulean token]

env sources is an interested concept--instead of only using one source (e.g. .env or dagger.env.json), you could configure the source you want in ~/.dagger or something

[tired moth]

so as of now also based on dagger current env feature and github cli we can create this small utility function which can sync/get and set ENV for dagger locally, right?

[sharp marsh]

Yep

[winter linden]

@tired moth be careful, Dagger doesn't allow loading secret values from .env (too insecure). Instead it will load secret references (eg. op://foo/bar, file://bar/baz, env://myvar). If you sync secret values directly to a local .env and then try to load thatf from dagger, it won't work

[tired moth]

we used to do this with dagger secret...

[winter linden]

If Github had an open API for their secret store, we could support that as an additional secret backend - eg. gh://my-secret but I believe they don't

[tired moth]

ohh ya right, so secrets we have stored in github repo ENV settings we should sync with dagger secret feature only?

[tired moth]

https://cli.github.com/manual/gh_secret_set this can help i think to make it more easy for devs

[tired moth]

this is supported i think, is should be natively supported in dagger? https://docs.github.com/en/rest/actions/secrets?apiVersion=2022-11-28

[tired moth]

https://docs.github.com/en/rest/actions/secrets?apiVersion=2022-11-28#create-or-update-an-environment-secret

[winter linden]

@tired moth that is an API for managing secrets, but not for consuming them from within a github actions job. For that, you need to inject them explicitly, one by one, into the environment.

[digital tundra]

Is there a way I can tell dagger to use a specific buildkit instance (not a custom runner)?

[winter linden]

No, Dagger is based on buildkit but it's a heavily modified version. It bundles its own instance.

What use case do you have in mind?

[digital tundra]

AWS provides managed BuildKit instances integrated with CodeBuild, so I was willing to leverage that.

[digital tundra]

Is there a way to make the --progress=plain output less verbose? Ideally I'd want to only log the output of a Dagger function (I say --progress=plain because this is for CI environment).

[cunning jolt]

Try --progress=dots?

[digital tundra]

Thank you!

[digital tundra]

I'm getting quite a few of these:

ExecError: failed to update rootfs: session "eo70bnuoy7m9560i5ptx8jbe4" not found

Is there an usual cause for this?

[wheat mango]

I have integrated dagger for CI with argo workflow and my dagger module for CI is written in type script, whenever i run my pipeline .
the "load module" times are very high like 20-30 seconds , what are the things I can do to optimize it ?

[hidden dirge]

Likely more of a GitHub actions question than a dagger question: is it possible to create multiple checks from a single workflow step, e.g. via checks API or similar? Context / use case is that we call a single dagger function in our workflow and then all the rest of the workflow happens inside dagger engine.

Ideally, we'd create multiple checks on the GitHub repo so that engineers could feeback faster on individual steps running inside Dagger engine

I guess the alternative would be to call multiple dagger functions in the workflow and export intermediate results to the GitHub actions runner so that they can be passed between different steps

[steady sparrow]

This is tripping me up. The file appears in the fs in terminal right before a .File() call to get the exact path, but the File call can't find it? Any ideas? ```
├╴✔ .withMountedCache(
│ ┆ path: "/var/cache/build"
│ ┆ cache: cacheVolume(key: "builds"): CacheVolume!
│ ): Container! 0.0s
├╴▼ .terminal: Container! 22.0s ◆ Disk Read: 4.1 kB ◆ Disk Write: 0 B ◆ IO Pressure: 85µs ◆ CPU Pressure (some): 752µs ◆ CPU Pressure (full): 624µs ◆ Memory Bytes (current): 184 kB ◆ Memory Bytes (peak): 6.1 MB ◆ Network Rx: 320 B ◆ Network Tx: 1.8 kB ✔ 1
│ ┃ dagger ~ $ ls /var/cache/build
│ ┃ main.wasm
│ ┃ dagger ~ $
│ ╰╴✔ exec sh 22.0s
╰╴✘ .file(path: "/var/cache/build/main.wasm"): File! 0.0s ERROR
! /var/cache/build/main.wasm: cannot retrieve path from cache

[steady sparrow]

Hmm, is it this? https://github.com/dagger/dagger/issues/10932

[winter linden]

Yes. "cache volumes" are special mounts which are only available during execution of a command. This is a limitation of underlying buildkit. So in practice if you need to get something out of a cache volume, you need to write it to a "regular" part of your container during execution, then you can get it out.

generally the final output of your build is not something you want in a cache volume anyway, since you won't benefit from persisting it across builds. it's ideal for persisting intermediate artifacts like package download cache etc

[steady sparrow]

Understood. In this case the actual compilation is very slow (tinygo) and caching the output seems potentially helpful. I should be mindful of the footguns.

[winter linden]

do you mean to avoid rebuilding if no inputs change?

[steady sparrow]

Using a stale build is probably not worth the time savings when it would result in testing something that's not reflective of the source code.

[steady sparrow]

I do wish to avoid rebuilding it if the input source hasn't changed, but I'd also love to be able to reuse the binary from a Build(ctx) function in, say, a SmokeTest(ctx) function

I was trying to figure out a good pattern to use for that. I thought cache volumes might be it, but ran into the above.

[mighty flax]

Hello everyone. Trying to understand why healthcheck aren't part of the Dagger API ? in the asService() for example or even part of the Container object. Is there a good reason I am missing ? Because implementing it ourselves is really not that straigthforward specially in PHP where we don't have async natively. (tryiing to mimic the depends_on multiple service health status)

[winter linden]

Hello - there is a builtin healthcheck, service bindings will block until the TCP port is available. If you need a custom application-level healthcheck, you should be able to implement it yourself - but I don't know the specific challenges of doing that in PHP. I'm happy to discuss them and see if we can find a solution.

[mighty flax]

Healthchecks

[mighty flax]

Other topic (migrating from docker compose + dockerfile to dagger) : it says dockerfile are supported but multiple context build are not am I right ?

[mighty flax]

Additionnal contexts

[candid gyro]

There's no such dagger primitive right now. Seems like they're experimenting with something like that, but it's unclear to me whether that will support the 1:N relationship between a dagger invocation / entrypoint and GH checks.

I have the same need (single function entry point doing many "checks") and I ended basically just doing it myself from within the function using the github API

[winter linden]

FYI We're working on a first-class concept of check 🙂 https://github.com/dagger/dagger/pull/11211

[tired moth]

What problems this feature will solve? i didn't find any info in pr or issue so asking here...

[winter linden]

I will write a summary soon.

Basically, there will be a first-class command dagger checks which will detect functions in your module that qualify as checks, and run them with zero configuration.

This way your project can expose tests, linters, format checks, etc. anything that returns green/red. As a standard entrypoint

This also provides a standard entrypoint for CI

[tired moth]

and this will be natively supported with github action checks features also?

[winter linden]

Yes it will give us a perfect entrypoint for integration with Github Actions checks 👍

Also a perfect building block for removing legacy CI completely, and running a fully Dagger-native CI stack 🙂

[tired moth]

Awesome let us know once it ready as experimental even and we would love to try and give feedback. Thanks a lot

[cursive bridge]

I'm getting confused about how flag conflicts work. For example, in this module:
https://github.com/sagikazarmark/daggerverse/blob/e6c14178bcaaff9e102e2a89173ca31479b80629/golangci-lint/main.go#L127

The verbose flag is allowed and there's no conflict. However, when I create a module and have a function that has a verbose flag I get a conflict error with core dagger. Both verbose args are optional. What am I missing?

[winter linden]

I'm getting confused about how flag

[digital tundra]

Are there any known network gotchas when running services with dagger ... up? I'm running into some pretty weird behaviors: https://github.com/dagger/dagger/issues/11239

[winter linden]

🐞 Service stops responding · Issue #112...

[digital tundra]

Is there a way to completely hide a Container.withExec output from the logs? I mean selectively, something like .withExec(...).noLog(). Maybe at the function level? It'd also work.

[candid gyro]

I don't think so but would love to be proven wrong. I think the only control you have over the logs is --progress / --quiet and --silent.

[steady sparrow]

Anyone have a thought on this error? I've got a bool field on my own type, rather than the dagger module, and it refuses to serialize it: ```
./dagger.gen.go:397:16: json.Unmarshal undefined (type bool has no field or method Unmarshal)

[steady sparrow]

ContainerWithExecOpts gives you a way to redirect stdout and stderr, which could be used to hide it. You can get it from the resulting container later.

[narrow nymph]

i'm guessing your field is called json?

that'll be the issue

[steady sparrow]

daggit!

[narrow nymph]

there's an issue here: https://github.com/dagger/dagger/issues/6186

[steady sparrow]

Thanks, that error message would not have clued me in. Saved me a lot of head scratching.

[cunning jolt]

I saw this today and wondered, "haven't we been able to do that all this time?" - https://github.com/dagger/dagger/pull/11238.

[candid gyro]

Don't think that hides it from the output/trace though. You have to redirect in the exec shell script itself if you want to hide it from the engine.

[steady sparrow]

You're right. I just tried it.

[stuck wyvern]

Howdy.
Trying to run my dagger build... I guess for the first time on my new(er) desktop.
The engine keeps crashing due to cni setup failing to initialize the nat chain.

Note: host uses nft

[cursive bridge]

Could this also be the source of the function param collision I was hitting?

[candid gyro]

Changesets are 🔥 guys gj

[tender orchid]

Hi folks! I'm curious about the expected behavior of the following lines in the dagger shell:

my-module | foo-returning-ctr | stderr &
my-module | bar-returning-ctr | stderr &
.wait

Should I expect this script to run the foo and bar functions in parallel? Or is there a different syntax I need to use? The examples I saw in the dagger docs only spoke to running dagger functions using native language features with Go/Python.

[tender orchid]

Answering my own question... It looks like this is definitely the right syntax.

[tender orchid]

Another question... We're running on a VPN, and a step that is as simple "WithExec" shell command (like sed on a small file) is taking up to 4m to complete. When I increase verbosity I see that there's around 6-7 HTTP GET requests, taking from 1s - 3m45s. What's up with the HTTP requests? It doesn't seem like it's necessary to run a command in a container.

[cunning jolt]

If you share a screenshot of the HTTP requests I could confirm but typically those "HTTP requests" are for each layer of a container image that's being pulled. So that would explain the timing variance.

[tender orchid]

This seems to happen across runs too in the same dagger engine, which to me indicates that it's not a docker layer download.

[cunning jolt]

It is, though, maybe some layer got updated. Does it happen for every run?

I see you've pinned to a SHA, so it shouldn't pull again. At this point I'm out of my depth of knowledge. Someone from the Dagger team may have to investigate.

[tender orchid]

Yep, every time.

[cunning jolt]

That is strange indeed. And shouldn't happen

Do you have enough free space on that machine? I know dagger prunes it's cache based on available disk space

[tender orchid]

Yep, plenty of storage space. It has to be VPN-related.

[cunning jolt]

https://docs.dagger.io/reference/configuration/cache/#garbage-collection (just in case). I can't imagine why it would be network related. Unless your previous pull failed.

[narrow nymph]

which issue was that?

[winter linden]

Re-downloading layers for pinned image?

[loud briar]

Have searched through discord and this has come up quite a lot: running go test on a module. Dagger's own tests appear able to connect to a client (e.g. https://github.com/dagger/dagger/blob/main/core/integration/container_test.go) but it's not clear where Middleware and connect come from, or if this method could be used within a module without clashing? Is there a summary of this anywhere? Seems like existing attempts are wrapped up in codegen for the module to be able to use it?

[cunning jolt]

SDK native test framework support

[winter linden]

@elfin frigate as discussed - I took a minute to file the issue https://github.com/dagger/dagger/issues/11258

[hidden dirge]

I was thinking of doing something very similar. Do you have an example of what the required API calls look like? I couldn't quite make sense of the GitHub Actions docs. I assume these checks need to be defined as a workflow in the repo?

[candid gyro]

I've done it purely with the Create and Update Check Run endpoints. The idea is:

1. Create Check Run with status queued (or in_progress if you're gonna immediately follow it by the actual operation)
2. Do you thing
   3 Update Check run as status completed and conclusion (e.g. failure / success / skipped etc) based on the result of #2

And you don't need to tie the checks to the workflow at all. You can have a single job workflow that runs e.g. dagger and then the code there can do what I described above arbitrarily many times. That's what I personally want and am doing, which then ends up looking like this.

[winter linden]

do you call the gh checks api from within your dagger function, or from a wrapper?

[candid gyro]

within

my use case here is building cookie-cutter platform pipelines, where it's just simpler for me to have a single function / entrypoint, where I can do a bunch of discovery and then orchestrate the actual sub-operations. Using GH checks and custom OTEL spans then helps "unblackbox" that single entrypoint

[winter linden]

how do you use the otel spans? dagger cloud?

[candid gyro]

yeah I stole that from your util/parallel pkg, so the above checks would also be visible in Dagger Cloud

[winter linden]

Nice. Basically we're going to make this portable. So you'll get the same result but without tying your workflow logic tona Github API (after all one of the promises of Dagger is portability)

[candid gyro]

yeah I think that would be great as a dagger primitive, especially if it's not necessarily scoped to just dagger call or a single function, but you can also use it as an API while doing stuff in a single function. That being said I personally also might end up sticking to direct Github API calls in some cases, depending on how much of the Checks UI we want to use (I expect a generic checks abstraction will have to have a more limited, common denominator of what you can output/report).

[cunning jolt]

Is the Changeset functionlity documented? I couldn't find any. My question is, I created a dag.Directory and added files+directories to it and I return that from the function. Now, I want to get the Changeset functionality of prompting the user to apply changes, however, I am not sure how to convert the Directory to a Changeset without comparing it with something. Do I have to create an empty directory just for comparison?

I kind of accomplished it this way. Given output is my Directory to export, I return output.Changes(dag.Directory()). I feel like it would be better UX to be output.AsChangeset(). Unless I'm missing an existing API

[cerulean token]

@cunning jolt I don't think that a Changeset generated against an empty Directory gives any benefit over just exporting the Directory directly. I'd guess that's why there's no Directory.AsChangeset()

I think Changesets are targeted at comparing a Directory before and after some generation step has ran so that you can export the just the additions and removals rather than the entire Directory

*Disclaimer: This is just my understanding as a user of the feature after playing around with it for maybe 1h

[winter linden]

I actually have felt the need for something like Changeset.withLayer(layer: Directory)

Not sure how niche of a need it was

[cunning jolt]

I thought that, but I have an actual use case. I'm building a function that scaffolds a project. The consumer of that function has an empty directory and I would like them to see the changes that are being applied to their directory. I don't want them to have to pass --export as a flag blindly

[winter linden]

Until Changeset, the standard way to represent a change to a directory was with a layer directory - it's what Directory.diff would give you. You couldn't express removals, but it was a subset of Changeset

[cunning jolt]

This would work too. If I can create an empty Changeset first and add stuff to it.

[winter linden]

Also we want Changeset.withChangeset to more easily merge Changesets without having to plumb the source directory around

[winter linden]

Yes correct, dag.Changeset() would be part of the package (unless @elfin frigate objects? seems like a must for the other stuff we discussed)

[cunning jolt]

Also, I noticed (by accident), that the UI (output of Changeset) in the TUI is not scrollable. For very big changesets, it actually hides the Accept/Discard buttons too.

[winter linden]

I ran into that too 🙂

[cunning jolt]

The way TUI works, doesn't let me access the scrollback of the terminal either

[elfin frigate]

hmm arguably it might still make sense to start from the empty host directory, just so that the Changeset knows exactly where to export to regardless of where the user called it from - see https://github.com/dagger/dagger/issues/11160

[elfin frigate]

if your terminal supports it, you can hold shift to bypass the scroll wheel interception, but yeah, that still won't let you see the rest of the list (just useful if you want to view output before you started the TUI)

the UI needs some polish, for sure - we're on iteration 1.1-ish 🙂

[winter linden]

@cunning jolt if you want a real-life example of using Changeset in a big workflow. We just merged a refactor of our own CI module 🙂

Try:

dagger -m github.com/dagger/dagger call generate

Also lots of new fancy custom spans, so it's easier to see what's going on

[cunning jolt]

I've been keeping an eye on it. I'd love to pull those custom spans into my modules.

[winter linden]

There's a tiny Go package which you can now import. It's an internal utility, not part of our SDK, but it's been very helpful.

github.com/dagger/dagger/util/parallel

Combines errgroups with custom spans

[cunning jolt]

I'll try pulling that in!

I've been favoring the sourcegraph/conc module for parallel stuff

[winter linden]

Ha, let me know if there are ideas from that package that we can steal!

[cunning jolt]

It basically gets rid of a ton of boilerplate and produces a clean interface. It looks like the util/parallel has similar goals 🙂

[cursive bridge]

Is there a way to get dagger to exit with a failure but also allow files to be returned and exported? The use case I'm looking at is that I am running a series commands that will produce output files. Those files will be produced even if there's a failure in one of the commands. I want the output files but also to designate that there was a failure in the pipeline.

For example, if I am running go test and using that to generate code coverage files. The fact that go test fails (because a unit test fails) may not be terminal for the use case (determining code coverage). So, I'd want to indicate that go test failed, but also be able to retrieve the code coverage files.

[winter linden]

Hi! There's been a pretty substantive discussion of this at some point, let me try to find it in the archives.

Basically, errors are meant to communicate that something went wrong at the time of execution. It's a fundamental property of the Dagger API (and of graphql, which it is built on) that a function either returns an error, or a value, but not both.

So, to achieve what you want, you have to catch the underlying exec error, and handle it yourself. Then return a non-error value that includes 1) whether the test failed or not, 2) additional information like code coverage

There's an argument in withExec called expect which does exactly that. You can set expect:ANY then check the container's exit code. And continue your logic from there

[cursive bridge]

Yep, I found expect, and I can retrieve the exit code for the command. What I can't figure out is how to have dagger indicate there was an error, but also return the files. AFAICT I have to follow the <cmd> | export to get files out, but if I return an error from the dagger cmd then there are no files (or it won't continue to the export function). Is this maybe where a custom return type would do the job? I'm not exactly sure how that would work though.

Maybe a custom return type that has functions that can chain to export then return the exit code? Ala

ReturnStruct.Export().ReturnCode()

So the files are exported but the final call is the exit code from the command

[winter linden]

Yes you can define custom object types, and return that

[loud briar]

Having a small problem with OTEL vars. Dagger sets these but if another image reads them they can cause errors. I'm not sure how to get around this short of deleting them all. WithoutEnvVariable isn't sufficient here either for the OTEL_ vars Dagger sets.

[cursive bridge]

Is the Daggerverse search down? Every query I try returns:

API error [403] Invalid Application-ID or API key

[winter linden]

Correct, we're on it! Thanks for the report.

[winter linden]

@cursive bridge fixed. Thank you @rigid pebble

[cerulean token]

Is there an issue for accessing stdin/out/err streams and exit codes of services? I feel like I've seen it mentioned a couple of places but I can't find them now

[winter linden]

cc @elfin frigate

aka the Service Master

[elfin frigate]

https://github.com/dagger/dagger/issues/6553

[cerulean token]

Really makes you wonder how I couldn't find this, eh? Thanks!

[loud briar]

Does 0.19.3 have the user default fix for directories with +ignore in it?

[winter linden]

yes, and also the fix for modules with dashes in their name 🙂 (FYI @raven stag )

[loud briar]

Then the only issue I have left is something to do with simultaneously accessing secrets

[raven stag]

Thank you!!!
Just had 10m. We have so many modules with dashed names 😅

[winter linden]

I realized that my own .env was broken (our main module is called dagger-dev...) 😅

[pseudo stream]

fix for at least one cause of that error here, should be in next release: https://github.com/dagger/dagger/pull/11295

[winter linden]

A lot of sweet features are getting merged next week... be prepared!

[winter linden]

Function cache control just got merged! by @pseudo stream

• All functions cached by default, new syntax to customize each function's caching
• +cache=never to disable
• +cache=<duration> to set a ttl: +cache=30s, +cache=2weeks, etc
• Opt-in for existing modules, for backwards compatibility
• Default for new modules (once we release)

https://github.com/dagger/dagger/pull/10975

[raven stag]

generalized function caching by sipsma ·...

[pine crag]

Hey Dagger team! I'm Arsh and I work as a DevRel engineer at MetalBear. Recently someone created a GitHub issue for our OSS tool (mirrord) about creating a mirrord dagger command. Before we proceeded to work on this I wanted to ask if you all would be open to co-marketing this together and helping support some of the dev effort (I'll be building this and am familiar with Dagger but would need help in some areas and would have questions)?

Thanks! Looking forward to the possibility of working together on this :)

[sharp marsh]

hey Arsh! not sure if you've already seen this thread (#1428375419387641907 message) but I think there's still some more clarification needed in order to fully understand what's the best possible integration while using Dagger and Mirrord together. Given that @gaunt nova had an initial requirement for this, It'd be awesome if we could all spend a bit more time understanding the scope and what's the expected outcome

[loud briar]

Could a module type's fields be split from functions in the dagger functions display? Module type fields needing to be public to use elsewhere in the module clutters up the output.

[cunning jolt]

Are you aware of // +private? Have you tried it?

[loud briar]

I am, yes, but there is a note here that suggests I can't use both // +private and use those fields in other functions in that module: https://docs.dagger.io/extending/constructors/#simple-constructor

[cunning jolt]

Are you referring to this? That's just referring to "public" in Go (Uppercase). Not the pragma. You can use // +private fields in the same module anywhere. You just can't use them in other modules.

[loud briar]

Ah so "other module functions" means "not this module" functions, not "other functions in this module"

That might help, I'll review with that in mind

[cunning jolt]

Yeah the wording there could be clearer, however, the subject of what it refers to is correct. It's referring to the following

Public

type Test struct {
  Field1 string
}

Private

type Test struct {
  field1 string
}

Defining private as ^^^ above can introduce weird behavior even in the same module.

Nothing to do with the pragma //+ private. That pragma was born AFAIR because there was a need to hide fields in the CLI and external modules even after defining them as public in Go.

[loud briar]

Great, that helps quite a bit!

[winter linden]

MERGED lazy module loading, by @wispy tapir (built on foundational work by @jagged flicker )

• dagger call no longer loads all module runtimes for all dependencies upfront
• Instead, it load only the runtimes it needs, when it needs it.
• In practice: dagger call will be faster. The more dependencies you have, the faster it gets
• dagger functions will also be noticeably faster

This is part of a big push to improve performance. More soon 🙂

[digital tundra]

Hey! We've recently switched the subscription to "Team" but in the Dagger Cloud UI we still can't add other people. Can someone from the team help?

[wispy tapir]

Hey!

@sharp marsh or @rigid pebble will take care of that soon, they'll be awake in few hours (UTC-3)

[rigid pebble]

Hey! We've recently switched the

[chrome moss]

Quick question for you folks. When you dagger init with an sdk, it brings in some example functions in main.go. Is there a way to replicate this for things like using dagger init with a blueprint and have the blueprint put example runs of whatever I have in the blueprint in the main.go when installed?

[cunning jolt]

This is not possible today but it's a cool idea!

[wispy tapir]

I think @warm temple worked on a feature called eject that basically do that 😄

[jagged flicker]

Link to the draft PR: https://github.com/dagger/dagger/pull/11166

[warm temple]

Yes thats exactly the idea 🙂

Its on hold for the moment while we figure out a WIP feature called Toolchains https://github.com/dagger/dagger/pull/11238

[cunning jolt]

I love the idea but IDK if I like eject. I can't think of a better alternative right now though, maybe depending on the outcome of toolchains 🙂

[warm temple]

Yeah we can workshop it when its time 😛 The cool thing is it uses the original blueprint as a dependency and creates a passthrough function to each of the blueprints functions, which means when you eject (or whatever we want to call it), you can do so in any sdk regardless of the blueprint's sdk

[cunning jolt]

I like it but IDK if I'd want all the public functions of the blueprint created as passthrough. Could be overwhelming. I think it would be useful if I can have an examples.go (or whatever name) where I can specify example functions (with comments) I want to distribute to the end user. Just a thought.

Or a pragma // +example 😄

[warm temple]

Yeah that makes sense. It currently just does the top level functions on the main object, so not actually everything. With toolchains though, you could potentially use the blueprint as a toolchain and a dependency and then only write code for the functions you actually want to modify

and unlike blueprints, toolchains can coexist with sdk code, other toolchains, a blueprint, dependencies, etc

I'm very excited about all of it though, because soon enough we'll have lots of great paths for using dagger without needing to write any code or graduate from Dagger university first 🤩

[cunning jolt]

I love where it's headed! But I also dread explaining the difference between module, toolchain, blueprint and sdk etc. to my end users.

[cerulean token]

I agree; I think that your suggestion about a pragma to signal that a particular function should be passed thru as a toolchain function is nice. assuming that I'm understanding the toolchains feature correctly 🙂

[warm temple]

100%, to us they're all "just modules" but that wont be obvious to new users. I started this docs page for toolchains, but we'll need a very high level one explaining exactly that https://github.com/dagger/dagger/blob/42637b5c849cf09e3095155786c00c8b454bb6b5/docs/current_docs/introduction/features/toolchains.mdx

[winter linden]

In the end you should only have to explain toolchains. Everything else will be for power users.

[cursive bridge]

Do functions defined in custom dagger modules support repetitive option invocations? IE

dagger call func --option arg1 --option arg2

I know some command line tools allow that kind of thing for defining multiple output formats or something. Is that allowed in dagger (ie each --option will call the Option function and allow building a list of passed args)?

[winter linden]

if the argument is an array, yes

[cursive bridge]

I've got a tool I'm building a module around that can take a local file, container on a registry, etc and the tool cmdline doesn't need any flags to differentiate the type. Rather than create a function for each type with almost identical args, I was trying to consolidate to a single function in my module. This is providing to be a bit of a challenge because dagger wants to operate on dagger.* objects.

My original idea as my function would take 2 args: artifact and artifactType, and perform the conversion internally, mount file/directory, etc, and call the tool. To do that though, artifact is effectively a string and I have to create the appropriate dagger object depending upon artifactType. For files and containers this seems doable, but I'm not sure how to create a dagger object with a contents from the host.

The other way I came up with is an arg for each type of artifact. That gets messy if I add more artifact types, but at least I operate with the dagger objects already created.

Is there another way to do this? Is there a way to create a dagger directory object with contents from the host inside a dagger module?

[balmy cedar]

Hi, it seems that dag.Git() is never cached (?). Is there a way to cache the result of the clone, and tie its lifecycle to the value passed to .Ref() ? Thanks

[winter linden]

The object downloads are cached. What you're seeing is probably the ref lookup, which is cached only for the duration of the session. This is necessary to get the latest commit for a given ref. If you want no lookup, you can pin the ref to a specific commit ID, then you should see that lookup go away.

Do you mind sharing a snippet of the code you're seeing issues with?

[jagged vault]

Hi, I was following Doc. on this step and it seems there's a lack of container().build(...) support ?
https://docs.dagger.io/getting-started/types/container#build-image-from-dockerfile-using-different-build-context

Thanks 🙂

[winter linden]

Hi, sorry about that! Container.build() has been deprecated in favor of Directory.dockerBuild() and it looks like we forgot to fix some docs snippets

(which is surprising since normally those snippets are tested in CI?)

[winter linden]

in that snippet you can do workspace.dockerBuild()...

[jagged vault]

Yep, that's the fix my IDE (copilot) proposed.

const ref = await workspace
  .dockerBuild({ dockerfile: "custom.Dockerfile" })
  .publish("ttl.sh/hello-dagger");

[balmy cedar]

Ah thanks ! It's the git ls-remote command that takes ~0.5s each time; the repo isn't re-cloned. What got me confused is that git() and .ref() are not marked as CACHED in the TUI.

Btw, specifying a commit hash doesn't seem to have an effect but I can live with that.

var repo = dag.Git("https://github.com/dagger/dagger")

func (*Mod) Example() *dagger.Directory {
    return dag.Directory().
        WithDirectory("/hash", repo.Ref("48241bd2fe5cf8125458d6a8d4f7fb019459b327").Tree()).
        WithDirectory("/tag", repo.Ref("v0.19.4").Tree()).
        WithFile("/http", dag.HTTP("https://github.com/dagger/dagger"))
}

[winter linden]

Ok I understand what's happening now.. This is actually a regression that occurs earlier in the flow, dag.Git() indeed used to be lazy. It recently changed to eagerly list remotes (there was a good reason but we underestimated the side effects). And indeed not only is that ls-remote not lazy, it's also not cached... We were just discussing the need to fix both.

[winter linden]

Here's the thread with all the details if you're curious: #1432773154752696521 message

[winter linden]

I know what you mean. We had the same situation for a vulnerability scanner. I think it's worth it to have different arguments or even different functions for this case, to take advantage of the type system. For example once you have a function that can take a Container type, you can pass any container, even one constructed on the fly and not available in any registry.

I think if you try to abstract this away you will create complications for yourself.

[winter linden]

if you do want to try the "dynamic casting" idea, you can use dag.Address() which is the API used by the dagger CLI to convert flag values to different types

[undone parcel]

Is there a way to have container-use, use the latest dagger engine? It seems quite outdated at the moment…

[sharp marsh]

this very likely works due to compat mode 😬 fixing

[sharp marsh]

fixed!

[winter linden]

Dagger 0.19.4 is out, with lots of goodies! 🧵

[winter linden]

Filesync details

When uploading local files to the engine, the TUI now shows real-time details on uploaded files & bytes transferred. Very convenient to catch that one directory you forgot to filter out 🙂

🧵

[winter linden]

Cache control!!

All functions are now cached by default, with a new syntax to customize each function's caching:

• +cache=never to disable
• +cache=<duration> to set a ttl: +cache=30s, +cache=2weeks, etc
• Opt-in for existing modules, for backwards compatibility
• Default for new modules (once we release)

[winter linden]

Lazy module loading

Instead of loading your entire module upfront (including dependencies), Dagger now loads only what it needs, as it needs it. As a result, calling your functions will feel snappier, and if your modules have many dependencies but only uses a small subset on any given call: your function calls will complete faster.

🧵

[winter linden]

GitRepository.uncommitted()

You can now access a git repository's uncommitted files, in the form of a Changeset, with the new API call GitRepository.uncommitted()

[winter linden]

Need for speed, part 1

We started a serious effort to make Dagger faster across the board.

Some of these improvements have landed in 0.19.4. For some operations you should start noticing Dagger being faster. This is only the beginning: we have many other performance improvements planned.

[digital tundra]

Hey guys! We're trying to deprecate our old CI stack in favor of a solution that uses Dagger, but we keep facing this issue every couple of days: https://github.com/dagger/dagger/issues/11293

I wonder if anyone else has ever faced this and found a workaround while the issue is not addressed.

[unborn pendant]

I hate all ci/cd tools... Except dagger – love it. Thanks for it!

[winter linden]

Thanks for the kind words! We're working on making it even better! More soon 🙂

[meager sentinel]

Where should i ask questions about Dagger? i'm thinking about implementing Dagger for our CICD, but i have a few questions before starting

[jagged flicker]

Here and #1030538312508776540 are fine, and then there's a bunch of topic-specific channels depending on your questions

[winter linden]

Dagger 0.19.5 is out, with another major performance improvement!

Need for speed, part 2

We tracked down a performance bottleneck in the engine, caused by excessive locking in the state database. All engine operations are now potentially faster. How much faster depends on your particular workfloads and hardware.

Generally, everything will feel more snappy. In some dramatic cases, your pipelines may experience 2x, 3x or even higher performance improvements. The gains are more dramatic on less powerful hardware. For example, local runs benefit more than production CI runs on modern machines.

See https://github.com/dagger/dagger/pull/11336

[loud briar]

@winter linden User defaults for enum parameters?

type Account string

const (
  AccountOne Account = "8741238742"
  AccountTwo Account = "9872387237"
)

...

func (m *Module) Example(account Account) {}

How do I provide a user default for that? I've tried a lot of variations of text, quoted/unquoted, single/double quotes. Error is failed to merge user defaults for "push": user defaults <module>.<function>(account=...): not valid JSON: 'AccountOne'

[jagged vault]

Hi, wondering if usefull to have the feature to manually set .*ignore files as arguments on Pre/Post-Call Filtering for "dagger-context" (as ignore files having one filter rule per line) : @argument({ ignoreFiles: [".gitignore", ".dockerignore",...]) 🤔

At the time of writing, Dagger does not read exclusion patterns from existing .dockerignore/.gitignore files. If you already use these files, you'll need to manually implement the same patterns in your Dagger Function.

[tired moth]

as of now can we replace grafana with dagger custom span traces and other otel features in dagger for devops otel?

[torpid maple]

Hey all, any news on self-hosted or offline trace visualization?

I would really like to produce a trace report as an artifact of my CI pipeline then visualize it locally through a self-hosted UI or static html files.

This would help me better demonstrate the value of dagger to my team internally.

I can't call out to the cloud hosted version of dagger cloud without it being sanctioned at my company, and I can better convince them to sanction it if I can demonstrate one of the key pieces of value it provides.

[winter linden]

Hello! At the moment there is no self-hosted edition of Dagger Cloud. But we are collecting requirements to figure out which kind of self-hosted exactly we should build. If that's OK I will DM you about your requirements.

Meanwhile, with the product we have: one approach is to setup a POC on multi-tenant Dagger Cloud, using no critical company assets (eg. an example app, maybe personal credentials, etc). Then demonstrate the value on that, in order to break the deadlock.

[torpid maple]

It's kind of a catch-22, because I wouldn't want to 'emulate' a company project in my personal account for the potential IP issues etc.

[winter linden]

From our standpoint, there's a chicken-and-egg also: building a self-hosted product is a big commitment. Supporting it, even more so. It's difficult to justify investing in that only for a POC that maybe, will result in a sales conversation in the future, which maybe will result in a sale... We already build plenty of stuff for free with the engine, CLI, SDKs etc

Looking at you @cunning jolt 😛

So for now we're focusing on making the product itself much better, so that the demo really speaks for itself - and everyone is more motivated to overcome those kinds of details

[torpid maple]

So as I understand it what would be good is the following use cases.

• Trace visualization solutions
  • Dagger cloud, receives a live stream of traces from the CI pipeline so you can use it as a live view of your pipeline's status.
  • Trace report artifact, produced at the end of a pipeline run as a static file. Allows you to visualize what 'happened' as opposed to what is 'happening right now'
    Thereby demoing the value of the trace feature, while still providing you some feature you can sell to customers (live trace visualization etc.)

[cunning jolt]

I do think some kind of a visualization feature locally would be beneficial overall because there is no other ci/cd tool that does that afaik. Could be the carrot that gets people to use cloud

[tender timber]

Hello guys, i have a dilemma what is better for a huge amount of pipelines? K8s daemon set which runs on huge nodes and orchestrates 10 - 15 workflows with 3 to 8 containers or gitlabs setup where all pipelines gets its own dind dagger setup but with mounted cache for the same amount? My workflows are greedy and use all available cpu capacity. It feels when running more resource greedy jobs even on huge machines dagger engine cant cope with containers it has to work and orchestrate. As well as load on privileged node seems to overwhelm dagger engine itself and starts giving all sort of random errors

[digital tundra]

Hi! Is there a way to have the traces in Dagger cloud (or even the output in any of the modes) to show colored output of the commands performed as if it was in a TTY?

[winter linden]

Hello! Are you asking specifically about the color? You already know how to stream TTY output within a trace in Dagger Cloud?

(Not sure about the color, will check)

[digital tundra]

Our functions show their stdout / stderr in dagger traces, but I don't remember doing anything special for that to happen.

[winter linden]

Are you seeing issues with colorized output in dagger cloud? What's the symptom?

[digital tundra]

Sorry, this was my bad. The issue is that the tooling disables colored output if it can't detect TTY on the process so I had to force it to output with colors. But talking about colors, there's this: #1436334941565747282 message

[torn wren]

Any suggestions on getting my IntelliSense in vscode working? I am using the python sdk and it works for the core objects like dag.container but it does not work for my dependent modules (things i installed via dagger install) I ran dagger develop; uv run code . with no luck.

[sharp marsh]

@torn wren do you have this setting enabled in your python vscode extension?

After enabling this and restarting vscode, it started to pickup dagger modules in intellisense without issues

[crisp cypress]

Aloha Dagger community - I am hoping someone can share some insight to help me decide if Dagger is a good fit for my use case.

I'm building AgentSystems - an open-source, self-hosted platform for discovering and running third-party AI agents.

The model:

• Federated (no gatekeepers - anyone can publish agents to a Git-based index)
• Agents run on user infrastructure with injected credentials
• All third-party agents treated as potentially malicious

I was looking at Kata/gVisor for isolation, but Dagger's sandbox looks very promising.

A couple specific questions:

1. Is the sandbox designed for potentially adversarial/untrusted code from arbitrary developers, or is it meant for isolating your own trusted modules from each other?
2. GPU passthrough? We need NVIDIA GPU access for local model inference (via Ollama/llama.cpp) I believe gVisor has issues with this

Thanks!

[torn wren]

That did not work for me. VSCode is still displaying a "Unable to import 'dagger'" error so maybe I have a different issue. My imports look like this:

import dagger
from dagger import dag, function, object_type, Doc

[torn wren]

I got the IntelliSense working by just navigating into the .dagger sub-directory of my project. Then run uv run code . and start the IDE from there and it works. Before I was running uv run code . from the root of my project. Not sure if that is what I was supposed to be doing all along or if this is just a work-around?

[hidden dirge]

Love the support added for the .env file for module constructor args. Is there an easy way today to load the ssh socket from $SSH_AUTH_SOCK?

On the cli we can use that environment variable which gets expanded to the address and properly created by Dagger

I've tried using a bunch of combinations of env:// unix:/// in the .env file but can't get this to expand properly before Dagger tries to create the socket

[winter linden]

You can't expand system env variables in the .env yet, we had the feature but had to cut it because of an implementation issue. The underlying issue should be fixed now, so we can try again. See #1413254056188579901 where we were just discussing this earlier today. Sorry for the delay!

[hidden dirge]

Thanks 🙂

Another question related to the .env file. How does this interact, if at all, with blueprint modules?

I've defined a .env file in module principally used as a blueprint module. Will these defaults also be applied when used as a blueprint module?

[winter linden]

They're orthogonal. Your .env should work the same whether the target module is based on a blueprint or not.

[hidden dirge]

Ok - thanks 🙂

[winter linden]

The only exception is a small convenience when using blueprints: when prefixing your env variables with the module name, you can use either the blueprint's name, or the target module's name

For example:

dagger init ./superapp --blueprint=./foo/bar/superblueprint

All these variables will match that module

$ cat .env
SUPERAPP_FOO=bar
SUPERBLUEPRINT_HELLO=world

[hidden dirge]

Also seems like default arguments may conflict in some way with values from .env?

If I have a constructor argument with default value None then the value from .env is not set as expected

Could be related to use of async constructor maybe, would need to repro better

    @classmethod
    async def create(
        cls,
        src: dagger.Directory,
        ssh: dagger.Socket,
        ci: bool = False,
        secret_with_default: dagger.Secret | None = None
        ...
    ):

If I try and set secret_with_default in the .env file I get None for secret_with_default

[winter linden]

If this turns out to be the case, it would be a bug. Normally a user default loaded from .env will overwrite the module's own default.

[hidden dirge]

Can reliably reproduce with example above

[winter linden]

I'm not a Typescript SDK expert, but I think this snippet from the docs is relevant: https://docs.dagger.io/extending/constructors/#default-values-for-complex-types

[hidden dirge]

Is resolved if I remove default value

[winter linden]

Specifically:

When assigning default values to complex types in TypeScript, it is necessary to use the ?? notation for this assignment. It is not possible to use the classic TypeScript notation for default arguments because the argument in this case is not a TypeScript primitive.

[hidden dirge]

This is python above, fwiw

[winter linden]

Ah it looks like my language selection isn't preserved in the shareable link, sorry about that

please manually click "typescript" 🙂

[hidden dirge]

Sorry bit confused 😅

Is this snippet relevant to Python as well? The module in question is authored in Python

[winter linden]

Oh sorry... I misunderstood. Thought your snippet was in Typescript. Long day...

[hidden dirge]

No problem, I feel you 😆

[winter linden]

I do think it's the same category of problem. Basically, for languages that have a native concept of default arguments, you have 2 ways to set a default: language-native, and with dagger decorators. It looks like in the specific case of the Python SDK + python-native defaults + secret arguments, .env defaults don't work. Would you mind opening a github issue for this one? 🙏

[hidden dirge]

Sure, can do 🙂