general

[ripe kestrel]

what's the recommended way to inject a secret variable into an interpolation in a file while building a container/context?

My Github Action i'm trying to replicate does this:

        poetry config system-git-client true
        echo "https://myorg:${{ secrets.GH_PAT_NEW }}@github.com" > $HOME/.git-credentials
        git config --global credential.helper store

in order to read from private repositories. Dagger is not allowing me to expand a secret variable like that in an echo with_exec, but my current solution (include a shell script as a file in the image that just emits that interpolation) is a little awkward. Any better recommendation?

[winter linden]

Actually withExec has an optional expand boolean argument for that purpose. So you can chain withSecretVariable("PAT", <value>).withExec(args: ["echo", "$ https://myorg:$[e-mail]"], expand: true, redirectStdout: "$HOME/.git-credentials")

[ripe kestrel]

Ah, but it defies me

│ ✘ .withExec(args: ["echo", "https://myorg:${GH_PAT}@github.com"], expand: true, redirectStdout: "/root/.git-credentials"): Container! 0.0s
│ ! expand cannot be used with secret env variable "GH_PAT"

[winter linden]

ah I guess it's trying to save you from accidentally leaking the secret. Try grabbing the secret's plaintext and setting it as a regular env variable instead? should do the trick

[sharp marsh]

@ripe kestrel I generally use envsubst for that in the container. The other option is doing sh -c "echo..." . That should also work I think.

I personally don't use the plaintext approach since it breaks the chain as you need to handle the err of the plaintext call

[sharp marsh]

also, I opened this earlier today: https://github.com/dagger/dagger/issues/9343

[round cradle]

I feel slow, but I don't see anything in the docs talking about the difference between withDirectory / withMountedDirectory, and they seem to be used in much the same context , e.g.

cookbook example

    return dag.Container().
        From("golang:1.21").
        WithDirectory("/src", source).
        WithWorkdir("/src").
        WithMountedCache("/go/pkg/mod", dag.CacheVolume("go-mod-121")).
        WithEnvVariable("GOMODCACHE", "/go/pkg/mod").
        WithMountedCache("/go/build-cache", dag.CacheVolume("go-build-121")).
        WithEnvVariable("GOCACHE", "/go/build-cache").
        WithExec([]string{"go", "build"})

vs https://docs.dagger.io/features/programmable-pipelines

return dag.Container().
        From("golang:1.21").
        WithMountedDirectory("/src", src).
        WithWorkdir("/src").
        WithEnvVariable("GOARCH", arch).
        WithEnvVariable("GOOS", os).
        WithEnvVariable("CGO_ENABLED", "0").
        WithExec([]string{"go", "build", "-o", "build/"})

Have I missed something obvious? I'm assuming it's persistence in the container if exported, but is there more to it than that? Layer caching impacts?

[slender star]

yes, I like to use that shell pattern together with withSecretVariable:

@func()
  test(token: Secret): Container {
    return dag
      .container()
      .from("alpine:latest")
      .withSecretVariable("TOKEN", token)
      .withExec(["sh", "-c", "echo $TOKEN"])
  }

dagger call test --token TOKEN stdout                                                                                                                                               
✔ connect 0.5s
✔ load module 24.4s
✔ setSecret(name: "448408b0970d5a6f8a68fa1db56c5e622daa103e6573a8d084834309e8d8672e"): Secret! 0.0s
✔ parsing command line arguments 0.0s
✔ eve: Eve! 0.0s

✔ .test(
│ │ token: ✔ setSecret(name: "448408b0970d5a6f8a68fa1db56c5e622daa103e6573a8d084834309e8d8672e"): Secret! 0.0s
│ ): Container! 3.6s
✔ .stdout: String! 0.1s

***

[turbid hornet]

I've got a Rust compilation workflow that all of a sudden is missing the cache on every build. Works remotely, just not locally. Anyone have any ideas about what could be going on? The mounted volume is in place. I'm really not sure how to debug cache stuff in Dagger!

[sharp marsh]

@turbid hornet if you use Terminal can you see the volume populated?

[turbid hornet]

yeah. It looks like I might have fixed it, actually… I have a postgres database running as part of this and the data is in a local (ignored) directory. Once I told Dagger to ignore that things got a lot faster and I started getting cache hits again.

just one of those things where asking the question here unlocked something that I had been stuck on for a while 🤷

[slender charm]

I've been meaning to add this to the docs forever, but here is the short answer:

The difference between withDirectory and withMountedDirectory is that withDirectory will include a directory in an image if you export it, and withMountedDirectory will not.

[slender charm]

Filed an issue to at least not let this one slip: https://github.com/dagger/dagger/issues/9349

[chilly arch]

Happy New Year, Daggernauts!

Thank you for your support, feedback, and contributions to Dagger. Check out our 2024 recap to see all that we’ve accomplished together. We’re excited to bring you even more in 2025!

https://dagger.io/blog/2024-highlights

[ivory cedar]

Hey guys, I'm building a CICD tool (not a CICD pipeline like Dagger - just a potential step to publish to the daggerverse). Is this channel good for asking for feedback on the idea or should I use something else?

[marsh timber]

https://docs.dagger.io/api/types/#container

Suggests the same thing.

EDIT: Sorry just saw how old your post was.

[stoic knot]

Are there any updates on calling out to modules from the SDK? We are currently using the Go SDK directly, and may have a situation where it makes sense to craft a Python module for our ML engineers

[modest pier]

calling out the modules from the SDK

[cunning jolt]

Hope everyone in SOCAL (near LA) are safe!!!

[chilly arch]

**Dagger Tutorial by your fellow Daggernauts: **

@paper grove and @tropic remnant translated their DevFest Nates presentation to English!

If you are interested in a Dagger tutorial from a fellow Daggernauts, check out the links below.

If you have any feedback, feel free to share it with them.

Slides - https://docs.google.com/presentation/d/1SvF3ZXCM0z6Xev1oOwQP0rPe02y5RH6DnO6YAIzsobc/edit#slide=id.g27c9038bd1f_0_25

Repo - https://github.com/jhaumont/enter-the-daggerverse/tree/main/runbooks/english

[potent lagoon]

Anyone have an idea how "Container.Export" is supposed to work? It doesn't appear to actually send anything to the host filesystem if I give it just a simple name like "foo.tar.gz". How do I get the data written back to the host?

OH I SEE. It's meant to be used via chaining.

[winter linden]

Mm, normally it should just export to the host filesystem. Maybe you need to force execution by calling sync() afterwards? But I would expect export() itself to never be lazy and always imply sync? cc @sharp marsh @real kite @pseudo stream

[sharp marsh]

Yes, export is not lazy. @potent lagoon is your Dagger function returning a Container and you're calling export from the CLI?

[potent lagoon]

I realized it works when I do that (return a container). I was calling .Export() in Go and that didn't work.

doing that, it would output like /scratch/latest.tar.gz (if I passed in "latest.tar.gz"

[sharp marsh]

if you call Export from Go, then your function needs to return the "dagger.File" which exports returns. Then you'll have to do dagger call my-func -o mycontainer.tar.gz for it to work

[potent lagoon]

Export() returns (string, error), not a dagger.File

[winter linden]

You're confusing with asTarballI think @sharp marsh

[sharp marsh]

right.. we changed that so it's consistent with other APIs.. So yes.. Export will write the container FS to the runtime container

[sharp marsh]

yes, I was getting confused with this one

[potent lagoon]

The doc string on Export should be updated then, it says it writes to the host filesystem

[sharp marsh]

agree that it needs clarification. I think Helder opened an issue about that 🤔

[potent lagoon]

Export, writing to the container FS, isn't super useful either I don't think?

[winter linden]

Yeah "host" is relative 🙂 It's the system running the client. Dagger supports nesting (sandboxes within sandboxes)

We will find a way to document it more clearly

[potent lagoon]

it's not "a filesystem in the context of the client", which is what I assumed "host" meant

[sharp marsh]

here it is: https://github.com/dagger/dagger/issues/8706

[potent lagoon]

@sharp marsh thanks!

[timber sphinx]

how are you invoking in your day to day?

[wanton pumice]

this might be a stupid question but is there a way to have a dagger engine docker image with custom modules pre-installed?
right now, it takes roughly 2 to 3 minutes for me to install the modules.

wondering if i can have a docker image that i can distribute to the team with specific modules already installed

[sharp marsh]

this might be a stupid question but is

[analog raven]

Community call starting now!

youtube: https://www.youtube.com/live/vCU3cZibObw
x: https://x.com/i/broadcasts/1ZkKzRjAZDdKv

[cunning jolt]

hey from Fort Worth, Texas!

[chilly arch]

https://dagger.io/blog/dagger-and-depot

[chilly arch]

PR that Andrea is demoing - https://github.com/dagger/dagger/pull/8730

[cunning jolt]

could you have run the vault server using dagger?

is it possible to get the plaintext of this secret to use within the module? or does it have to be a file?

[winter linden]

I think you can still get the plaintext

Summary of today's demos: https://tenor.com/view/michael-jackson-mj-cher-dance-dancing-gif-27508432

Awesome content, awesome crew. Keeping us excited and entertained!

[chilly arch]

If you missed it, you can check out the full recording here: https://www.youtube.com/watch?v=vCU3cZibObw

[raven stag]

Rewatching the call now...
Vault secret provider support!! 🤟 💕 😍

Can I try it? (what's the branch to pull the engine from?)

• Can I configure dagger to use a specific Vault address by default without having to define the VAULT_ADDR env var in every call?
• Is the VAULT_TOKEN env var "accessible" from dagger (apart from the secret engine)?
  • Can I check for it's presence to provide users of my module a nice error message if they haven't exported it?
  • Can I check for its contents, to distinguish the type of token that has been passed?e

[sand gorge]

Hey! SymfonyCon Vienna (Dec 2024) have an "online replay" of their conference named "SymfonyOnline". It just happened today. My talk just finished. Many curious questions.

I've directed people to the #php channel here for any questions or support. So keep an eye out 🙂

[warm temple]

Vault secret provider

[trail tundra]

Morning. Just want to say I really enjoy using Dagger and have really embraced it both for personal projects and work. I have put together a GitHub Gist that shows how to configure Renovate to update Dagger modules invoked by the dagger-for-github action. Hopefully it is of use to someone, https://gist.github.com/purpleclay/7021bfe83973abd960b60ceec7f5ff2e

[thick igloo]

hi! any running dagger on silicon mac with podman desktop? Is it supposed to work? I'm consistently getting the following error, even trying with a manual privileged engine container:

Error: failed to get module SDK: input: moduleSource.withContextDirectory.asModule failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get mod cache base dir from go module sdk tarball: select: failed to mount /tmp/buildkit-mount3406798944: [{Type:overlay Source:overlay Target: Options:[index=off lowerdir=/var/lib/dagger/worker/snapshots/snapshots/19/fs:/var/lib/dagger/worker/snapshots/snapshots/18/fs:/var/lib/dagger/worker/snapshots/snapshots/17/fs:/var/lib/dagger/worker/snapshots/snapshots/16/fs:/var/lib/dagger/worker/snapshots/snapshots/15/fs:/var/lib/dagger/worker/snapshots/snapshots/14/fs:/var/lib/dagger/worker/snapshots/snapshots/13/fs:/var/lib/dagger/worker/snapshots/snapshots/12/fs:/var/lib/dagger/worker/snapshots/snapshots/11/fs redirect_dir=off]}]: operation not permitted

[carmine oracle]

Is anyone heavily using Rust and willing to share the functions they are using for building? Looking to heavily optimize cache to decrease build times.

[chilly arch]

I was just about to recommend the Rust channel, but saw you there already 🙂 The Rust Daggernauts will be happy to help! https://discord.com/channels/707636530424053791/1141601893538332704

[coarse iron]

how to upgrade dagger from 0.15.1 to 0.15.2?

[winter linden]

That depends on your host platform. How did you install 0.15.1?

[coarse iron]

I have install using brew. I am currently using go sdk to build my application

[winter linden]

The it should be as simple as brew upgrade dagger. Did you install it with brew install dagger/tap/dagger?

[coarse iron]

yess I had used brew install dagger/tap/dagger to install.
Thanks for answer

[coarse iron]

I have been getting this error a lot can someone point me to the possible mistakes that I might be doing

! lookup gfua05feu53vi for hosts file: lookup gfua05feu53vi on 10.87.0.1:53: no such host
│ ! lookup gfua05feu53vi.gr84empk9jeps.qapor59ficap8.dagger.local on 10.87.0.1:53: no such host
│ ! lookup gfua05feu53vi.qapor59ficap8.dagger.local on 10.87.0.1:53: no such host
│ ○ .withExec(args: ["npm", "run", "test:e2e"]): Container! 0.0s
│ ✘ .stdout: String! 32.4s
│ ! start gv6erfgj8khiu (aliased as node-service): exited: lookup gfua05feu53vi for hosts file: lookup gfua05feu53vi on 10.87.0.1:53: no such host
│ ! lookup gfua05feu53vi.gr84empk9jeps.qapor59ficap8.dagger.local on 10.87.0.1:53: no such host
│ ! lookup gfua05feu53vi.qapor59ficap8.dagger.local on 10.87.0.1:53: no such host

I am using go sdk to run nestjs application

[vague needle]

👋 🐛

Looks proper if I resize the window down horizontally. I'm on Apple Studio Display with Default resolution (2560x1440)

[sharp marsh]

cc @chilly arch

[cunning jolt]

if you want to go back and forth between versions (brew makes it really hard to do), you can use a script like this - https://github.com/kpenfound/nixos-config/blob/main/users/kpenfound/dagger_switch. This is what I use and if you are on MacOS you can cater this for that. This could become a dagger module too 🙂

[magic latch]

I am trying to find in the docs where I can get my dagger session token?

[slender star]

I am trying to find in the docs where I

[stray steeple]

Managing multiple versions of the CLI

[patent slate]

I need help in understanding the innerworkings of Dagger (still). In the SDK API docs for the Container object for instance, one will find very often "Retrieves this container" in the text. Somehow, I'm missing a conceptual understanding of what "retrieves" means. Can someone explain what is meant please?

[sharp marsh]

"Retrieves" in this case is a synonym of "get a reference of". i.e: the the API reference here https://docs.dagger.io/api/reference/#definition-Directory says "Retrieves a directory at the given path.". Which means that if I do dag.Container().From("alpine").Directory("/etc"), the Directroy call will return a Directory variable type that references the specified dir in the container

[patent slate]

What is retrieves?

[ancient escarp]

Container->withDirectory() has an owner parameter; what does that take as an argument? just the file owner or can I do "www-data:www-data" to set the group as well? (if not, does setting the owner set the same group?) can't find any documentation anywhere 😦

[cunning jolt]

This? https://docs.dagger.io/api/reference/#Container-withDirectory

[fallow hound]

Hi, I just wanted to check if there is any sort of roadmap or indication of when v1.0.0 would be achieved? We are tying to adopt Dagger within a very large enterprise and are currently working through a PoC of how it will look using it with our current CI process but foresee potential backwards compatibility issues being quite a risk for us while trying to adopt Dagger and would expect pushback from some of our engineering teams.

[quaint meteor]

Good morning @analog raven @winter linden @wraith niche , checking in to see if we can find a schedule for a live demo/questions about Dagger with my team. Reached out to you guys beginning of the week, but I guess you've been pretty busy 🙂

[chilly arch]

👋 Hey Daggernauts, Miranda here!

Greg Kogan is conducting an internal study to learn how engineers are building AI agents and the challenges they’re facing. He’s setting up quick Zoom chats to gather insights.

If you’d like to share your perspective and help shape the future of AI agent development, please fill out this form: https://docs.google.com/forms/d/e/1FAIpQLSdph0Ak3lpPPwKACMt_U5mYhcsM--qSDwAZpyiphFW8twHQVg/viewform

Your input will make a big difference—thank you!

[ancient escarp]

Perfect thanks; SDK API docs weren't as thorough

[chilly arch]

Are you championing Dagger at your organization but feeling stuck?

We want to hear from you! 🚀 I’m putting together a list of resources to support Daggernauts like you in advocating for Dagger internally.

👉 Join the conversation here: https://discord.com/channels/707636530424053791/1332126539256430723

What content, tools, or assets would make your mission easier? Whether it’s a presentation template, case studies, or something else entirely—we’re all ears!

[rigid pebble]

I thought it would be relevant to share a short blog post I wrote on why I find it fun to work on CI: https://blog.matiaspan.dev/its-fun-to-work-on-ci/

[tired moth]

Are we going to consider distributed builds feature in near future in dagger? Can't we have this feature in minimal manner like docker swarm? it can help us to build and run onpremise k8 cluster for devops operations. PLEASE help
https://github.com/dagger/dagger/issues/9107

[tired moth]

@winter linden @sharp marsh @narrow nymph @cloud canyon Any consideration because to run llm on premise with dagger easily and in distributed manner it can help many developers and small companies a lot...Awaiting your response.

[sharp marsh]

there's been conversations about this internally. I'll let Solomon take this one

[tired moth]

The world badly needs dagger's following features

1. Dagger watch
2. Dagger distributed build to run llm in efficient manner

[sharp marsh]

@tired moth in the meantime you could have a pool of engines behind some reverse proxy which distributes works across them by using some sort of routing logic that you need. Would there be an issue following something like this for your curent use-case?

[tired moth]

We would love to try this to ditch the K8 inhouse cluster for devops part and run llm. is there any guide available? it can act in single resource pool manner like docker swarm?

[sharp marsh]

this would be similar to how you parallelize jobs in most of CI platforms today like GHA

[sharp marsh]

no, there are no guides available. This is somethng you'd have to build yourself

it really depends on each use-case given the requirements

[tired moth]

with dagger? is there any sample code or repo available anywhere?

[sharp marsh]

no, this is something you'd have to build on top of Dagger. Some sort of simple scheduler that given a set of dagger engines, it returns an endpoint that the client needs to connect to.

[tired moth]

ok so one dagger engine will be used to orchestrate other dagger engines from diff machines in the local network, right?
can we have a demo of this in the next community call from the dagger team?
because we are not that much expert or experienced in networking and such implementations

[sharp marsh]

can we have a demo of this in the next community call from the dagger team?

I don't want to give you false expectations but this is very unlikely to happen. We're working on other priorities at this moment.

[tired moth]

okay and dagger watch?

[sharp marsh]

We understand your situation and we have this under the radar.

[tired moth]

@winter linden we will be so thankful to you if you can share something on this...PLEASE

[tired moth]

Yes but no PR to track work right? any ETA or anything?

[slender star]

Yes but no PR to track work right? any

[tired moth]

We need to get dagger supported natively in GitHub action like docker for better and seamless adoption.
What you guys think?

[stoic knot]

I think it is crazy everyone installs tools during their CI process. It's one of the things I dislike most about GHA. Should build a custom image with everything you need to save a bunch of time & money

[sharp marsh]

yes, I also share the same feeling. It's even crazier when ops / dev teams setup makefiles that install tools locally as well.. that makes me question a bunch of things 😄

[tired moth]

@sharp marsh we should use official github action of dagger right then? https://github.com/dagger/dagger-for-github

[slender star]

You def can use it. It will handle installing Dagger cli for you.
You can also install dagger yourself with curl, for example, and the use dagger call or dagger shell -c to run things.

We haven't tried to get the dagger CLI baked into the GitHub Actions runner image because of our rapid pace of iteration. We wouldn't want to users to be stuck with a patch version of dagger by default that needs an upgrade.

That being said, if you want dagger pre-installed with latest or a version like 0.15.2 set up with caching, etc ready to go, you can use the Depot/Dagger integration.
https://depot.dev/blog/depot-with-dagger
https://dagger.io/blog/dagger-and-depot

[tired moth]

ok thanks for guide.

[cunning jolt]

I think it is crazy everyone installs tools during their CI process. It's one of the things I dislike most about GHA. Should build a custom image with everything you need to save a bunch of time & money

While I agree an image with tools baked in is the right way to go, I don't think it's a binary decision. There are some use cases where you would want to install a CLI on the fly. The following is one.

We haven't tried to get the dagger CLI baked into the GitHub Actions runner image because of our rapid pace of iteration. We wouldn't want to users to be stuck with a patch version of dagger by default that needs an upgrade.

[fossil pine]

BTW GitHub supports caching tools on custom runners. I’m not sure if the Dagger action utilizes the tool cache, but most official actions do and some tools even come baked into the image.

[brittle otter]

Hello, for the company I work for, I need to implement a dagger implementation for the development team. We are in platform engineering strategy and I need to bring this tool to each of the developers' computers, taking into account that they work with different operating systems, what would be the best way to carry out this task? thanks for your help.

[cunning jolt]

Each developer would need a container runtime (like Docker Desktop) and the Dagger CLI installed on their machines. As far as dependencies go that's pretty much it.

[brittle otter]

Hello, in what way do you think it is possible to facilitate this part for the ease of operation of the developers, or what do you think I can propose?

[cunning jolt]

This really depends on the use case and how your company has set up it's infra. The general requirement is a container runtime and dagger. I don't know if you could do much more than thoroughly document how one would get set up on each OS. If you are able to get your company to pre-load these softwares to the developer machines that would be ideal.

[coarse iron]

Hey! Can anyone point me to documentation on how to call Dagger for a private repository in GitLab CI?

[sharp marsh]

👋 sure. https://docs.dagger.io/api/remote-modules/

[coarse iron]

Thanks!!

[chilly arch]

Don't miss our Dagger Community call tomorrow at 9 am PT!

• @elfin frigate will showcase Dagger Cloud v3
• @wraith niche will show us how to build a Dagger Module from a Dockerfile
• @winter linden will build a multi-agent system with Dagger

Watch live on YouTube or X - https://www.youtube.com/watch?v=XeV1MqOFLAM
https://x.com/dagger_io

[coarse iron]

Hey guyss, We are currently trying to create a POC using dagger. At my organisation we are using gitlab and currently dagger does not have login using gitlab or microsoft account.

[calm lotus]

Hello, im trying to execute this command dagger -m github.com/samalba/dagger-modules/nvidia-gpu call ollama-run --prompt "What color is the sky?" but its loading for a long time... Is it normal ?

Its working with CPU but not with a GPU

[sharp marsh]

hey! is there a chance you can use a Github or Google account for the POC? If not, please DM @slender star and we'll try to figure something out

[sharp marsh]

hey Alex, have you seen this guide? https://docs.dagger.io/configuration/custom-runner/#gpu-support. In order to run Dagger with a GPU, some minor tweaks are currently needed

[coarse iron]

Yeah currently I am using my github account, but going ahead we might want to login using gitlab or microsoft account

[sharp marsh]

that's ok, we can make this work if it's a strict requirement for your teams to use gitlab or microsoft live 🙏

[coarse iron]

thats great thanks

[analog raven]

Starting in a few minutes!

[cunning jolt]

hi from Fort Worth, Texas!

[elfin frigate]

https://go-app.dev/

[swift inlet]

This traces page is amazing -- especially all those hidden features 😍

[cunning jolt]

Do these external 3rd party telemetry appear on the TUI too? That's incredible!

[chilly arch]

yup! For those who aren't on the community call, we just showed this in action. https://www.youtube.com/watch?v=XeV1MqOFLAM

[cunning jolt]

Does this work without AI?

[sick monolith]

from yt chat: https://github.com/samalba/dagger-modules/tree/main/docker2dagger

[cunning jolt]

I guess not :(.. (cries in Enterprise) no access to LLMs.

This is a cool pattern though. Can be used for a lot of other things too for example, converting a Jenkinsfile to dagger

[winter linden]

I'm sure that will change soon

Just plug it into llama or deepseek 🙂

[cunning jolt]

worth a try! I don't know if llama is even allowed.

[warm temple]

You might have better luck with ollama since it can all be run locally

[sick monolith]

similiar situation to you, ANY software installation = security + red tape + months of 'in review' before you install some of these tools - even though, Docker is approved

corp DNS also blocks a ton of stuff, even the dagger website was initally blocked for me 😛

[warm temple]

You can run ollama in docker 🙂 but that's not recommended on a mac host

[cunning jolt]

This is so next level! What's the backbone? What LLM is it using and how do you provide creds?

[warm temple]

https://ollama.com/ Tons of models to choose from. Different models are better at different tasks (and resource requirements). No creds needed 🙂

Generally what I've seen on my 16gb macbook is a lot less impressive than ChatGPT or other hosted LLMs. But it's local 🙂

[tired moth]

Dagger has so many features and capabilities to support ai agent workflows but it's not documented well so either we need a separate section in docs or detailed blog, what do you think? @winter linden @chilly arch

[chilly arch]

yup, we are on it! Right now, we are doing many AI agent experiments, which is what we are showing now. As we find best practices and/or specific use cases, we will be sure to document or blog about it. For example - https://dagger.io/blog/new-ai-developer-devin

[cunning jolt]

I meant for what Solomon is demo'ing. The agent stuff

[warm temple]

Ah got it 😄 I think it's all gpt-4o with the creds provided as a secret in the constructor

[cunning jolt]

ah gotcha!

[winter linden]

@cunning jolt at the moment it's OpenAI but super easy to plug in, we will probably have llama / claude / etc. support by next week it's so easy (and @warm temple @rain oriole already prototyped it in modules, so we can just merge that)

Oh you know what, it may actually work already in my branch

At the moment I take a .env file in the current directory, which accepts the following:

LLM_KEY=...
LLM_MODEL=...
LLM_HOST=...
LLM_PATH=...

So you could try running a local model and pointing at it, it might already work

FYI @warm temple if you're interested 🙂

[warm temple]

I will try it out 🙂

[winter linden]

This is the branch: https://github.com/dagger/dagger/pull/9463

[stoic knot]

can Dagger work with(in) the gVisor ecosystem?

[sharp marsh]

cc @heavy gazelle might have more info about this.

[heavy gazelle]

I was just looking into this a few weeks ago. I got stuck on setting up gVisor in Talos.

My biggest unknown is whether gVisor implements all the system calls that Dagger needs to manage the filesystem. Some network system calls for services might also be problematic.

As I had issues with gVisor, I went down the kata-containers path and hit a bunch of related issues: https://github.com/dagger/dagger/issues/9319

I still need to update that issue with my latest findings, but I'm currently working on a virtio-fs alternative config since that clashes with overlayfs.

As I am waiting on the kata-containers team currently, I am curious to try gVisor again. Maybe next week 🤔

Why do you ask?

[stoic knot]

I'm doing a stint on cloudflare only tech stack to see hof far I can push it. They seem to support gVisor and firecracker

[heavy gazelle]

That sounds interesting! Would you be up for a quick 1:1 demo?

I know that Dagger works perfectly well on Firecracker. There is a Dagger module that spins up Dagger Engines on Fly.io (runs Firecracker VMs) with a single command.

[stoic knot]

sure! dm me, I'm free all day

[maiden cloud]

Isn't it nice to end the week with a PR with the title:
Introducing Dagger CI

[cunning jolt]

Drop a bomb and take off for the weekend. love it! 😄

[winter linden]

@cunning jolt I'm curious if you have thoughts on that agent demo from yesterday 🙂

[old fiber]

can’t believe i am having to ask this (i wasn’t expecting my friction points to be from our dev team), but has anyone had to do a dagger defense to their dev side of the house? just wondering if anyone has any suggestions for points they found resonated very well.

[stoic knot]

Finally started to write up how I've tamed monorepos. Dagger gets the first slot, even before CUE (?!), because Dagger is the entrypoint to the dx for my monorepo setup
https://verdverm.com/topics/development

[winter linden]

Nice! By the way Tony, you'll be happy to know we're getting serious about bringing generated clients to external codebases - so you can more easily reuse Dagger modules in your own tools eg. dx. See #1334452944740814931 for ongoing discussion 🙂

[stoic knot]

I've got a new stretch goal for Dagger now. I want to run run a private ATProto setup in Dagger, something like this, but Dagger style: https://github.com/likeandscribe/frontpage/blob/main/packages/frontpage/local-infra/docker-compose.yml

And maybe also running Dagger on Cloudflare, sounds like a real possibility later this year when they roll out their container runtime

[timber crater]

I just dropped in to tell @chilly arch how impressed I am with the "intro countdown" on the YouTube recordings. IMHO, it's those details that evidence the spirit of continuous improvement here.

[urban lion]

i have a question related to the dagger builder.
Kaniko the popular build tool for kubernetes is now unmaintained, all the mtners have retired from the project.
my job uses GitlabCI but we are not allowed to use DinD because of security reasons.
Does dagger allows to build images directly from userspace or does it requires a actual docker daemon running with privileges ?
This could potentially be a huge way for dagger to step in and gain popularity in an unexpected way...

[sharp marsh]

hey there! unfortunately, Dagger doesn't support rootless builds yet. The main reason for this is because Dagger leverages buildkit under the hood for the DAG build and rootless buildkit (https://github.com/moby/buildkit/blob/3f4bcd37bd457275177fd78a19c5b557966d10c2/docs/rootless.md) has a bunch of gotchas and limitations.

we'd love to transparently support it in the future but the reality is that it'd require a considerable amount of work to implement it in a way that feels like a polished and 0 friction feature

[urban lion]

I understand

[winter linden]

@urban lion You can deploy the dagger engine on kubernetes separately, then configure your gitlab CI runner to connect to it. It requires installing a new resource on the kub cluster but will avoid dind. In case that helps.

[urban lion]

I was thinking of something like that at least on the gitlab ci side with a separate runner dedicated to DinD but this could also work. I'll get in touch with the relevant team with the idea in mind :)

[marsh timber]

With the enterprise plan is there the ability to host an internal dagger cloud?

[modest pier]

not today, no, but you're probably already aware that dagger cloud is just the observability piece, not the compute

[marsh timber]

yes, the observability piece would be nice to have

Ill see if I can get tempo to capture it or something.

[modest pier]

should be possible, it's all OTEL.

[tired moth]

Just wanted to share with the community we can now contribute faster with native GitHub copilot agent support.
I think we will have more features faster now. Let's leverage it and enjoy.
https://github.blog/news-insights/product-news/github-copilot-the-agent-awakens/

[tribal burrow]

seems like an incredibly bad idea

[winter linden]

Github Copilot Agent

[meager summit]

Hey everyone! I am working on Dagger integration within our corporate environment and have an issue with the Python SDK referencing pypi.org and pythonhosted.org indexes. Internally, we have a hosted PyPI mirror and direct access to public mirrors is not possible in our CI environment. Is there a workaround to instruct the Dagger engine to use a proxy, similar to what is done for Golang? I have read some tickets and documentation regarding this issue, and it seems the only solution is to configure HTTP_PROXY. Is there any simpler way to achieve this? Or could you share how exactly the proxy needs to be configured to work smoothly with the other Dagger engine functionalities? Ps. [[tool.uv.index]] is used on the module level and works as expected but the issue is that Python SDK libraries are still requiring publish indexes.

[sharp marsh]

Hey everyone! I am working on Dagger

[patent slate]

I'm confused about kube-pod connectivity. In the k8s docs, it is speaking about connecting to the engine pod.

In the Connection interface section of the docs, it is speaking about the runner pod.

I've tried both, neither work. 🤔

What works for me in my k8s (k3s), is the socket, but that is connecting to the engine via the socket in the runner. So, connecting to the engine seems to make more sense to me. 🤷🏻

[sharp marsh]

@patent slate, the kube-pod scheme uses the configured kubectl context you have locally to set a tunnel against the engine pod so the Dagger CLI can connect

[sharp marsh]

Runner and engine can be used interchangeably in this context

[sharp marsh]

Mind sharing a bit more context about where your client currently lives?

Is your intention that a kube pod connects to the engine pod within the same cluster?

[sick monolith]

Above conversation sparks a thought of my own regarding this sort of configuration. Can be hard to properly understand networking and it can be confusing about "dagger CLI on my machine" and "dagger CLI in another place". Then you throw things in like a Github Actions Runner - which then has the CLI installed, which would supposedly be able to commuinicate with the engine, running on the same cluster as the GHA runner was configured from etc

So some of those env variables only ever make sense to configure on a CI env variable as apposed to locally on your literal dev laptop

[patent slate]

the kube-pod scheme uses the configured kubectl context you have locally to set a tunnel against the engine pod so the Dagger CLI can connect

Ah, so I don't have kubectl set up in the "runner" pod. That explains why the kube-pod URL doesn't work.

Is your intention that a kube pod connects to the engine pod within the same cluster?

Currently I have a runner pod as part of a remote development environment (pod), in order to develop the CI process. And once I can run the CI process, I'll also be creating a runner pod for the CI tasks/ workflow within the same cluster, yes.

Are all forms of the connection interface handled 100% in the same way by the engine? I'm wondering for the discussion I am having with the TypeScript SDK devs.

[sharp marsh]

So yes having the kubectl binary with a correctly configured context should fix your issue in this case.

What do you mean by 100% handled in the same way?

[patent slate]

Because I'm having an issue that, when my Dagger module code is changed in the slightest, the module initialization starts from scratch and takes up to 40 seconds to run (again). We believe it is the downloading of dependencies and the TS SDK dev (braa) can't replicate it. So, I'm thinking it might be because of my socket connection, if it is handled differently. If it isn't, then I have no idea what could be going wrong.

It's also a big swing in the dark. 😛

[sharp marsh]

Oh.. yeah.. then connection strategy is definitely unrelated to the initialization issue.

One thing I'd love to see in your case Scott is two traces from a very minimal pipeline. A first trace with a regular function call and a second trace with a slight change in the pipeline to see what's happening with the case

Any change in your pipeline will require the module to be re-initialized but re-initializations should be relatively fast as the engine should be using it's local cache volumes to avoid downloading deps again

[patent slate]

How can I get the traces you are asking for made? Do I need a Dagger account? I don't have one.

[patent slate]

Yeah, this is what I'd expect too, but for some reason, it seems like caching isn't working as expected and the reason isn't known.

[sharp marsh]

Yes, you'd need a Dagger Cloud account. You can get a 1 user free account and send us the traces with that account

[patent slate]

Ok. Will work on this tomorrow. I'm in Germany and it's TV time with the wifey. 🙂

[sharp marsh]

Sure, let's check this out tomorrow

[lean mulch]

Hello! Anyone know of a succinct way to generate dagger module docs/markdown for function signatures? I want to have easy to read reference for each module in my private daggerverse.

[winter linden]

We're working on a solution to this, if you DM me we can get you on the early access list 🙂

[sand gorge]

Woke up to see this - I'm representing Dagger, and speaking at a huge DevOps conference in Lithuania.

[sand gorge]

This is on 23rd May.

On 24th May I will be representing Dagger, and speaking at PHPers Summit in Poland - another very large conference, around 1000 people there.

I timed that well, because Lithuania and Poland are next door to each other, so convenient 🙂 and travel costs are not so high

No graphics yet, they're still selecting the remaining speakers.

[chilly arch]

As some of you heard on the Community Call, we replaced our React frontend with Go and WebAssembly. Learn more about the experience here:
https://dagger.io/blog/replaced-react-with-go

[chilly arch]

See you at the Dagger Community Call tomorrow at 9 am PT! Check out our full lineup below.

Join us live on YouTube - https://www.youtube.com/watch?v=jkhSojdfuuM

[hazy phoenix]

Hey! Awesome post about the go-app wasm frontend usage. Welcome to the fold! I’ve been a long time user of go-app and wasm for the frontend https://crdtoyaml.com. where I parse a bunch of JSON all the time ( because I’m parsing crds and Openapi schema). The post said that was slow but I didn’t experience that. Anyways fantastic stuff folks very excited about this. ☺️☺️☺️

[tired moth]

Why are we not having a separate channel for Daggerverse?

[elfin frigate]

Preview CRDs

[chilly arch]

Community call starting in 5 minutes! See you there. https://www.youtube.com/watch?v=jkhSojdfuuM

[wispy tapir]

For people interested in the DockerSDK and want to share requests/feedbacks: https://github.com/dagger/dagger/issues/9577
For the source code/doc: https://github.com/quartz-technology/daggerverse/tree/main/docker_sdk

[tired moth]

This is really game changing feature for so many things. thanks a lot dagger team for such a great feature support. Dagger rocks

[jagged flicker]

Regarding the preview of the Java SDK:
-> #java for any question, idea, suggestion, bug reports, etc. Any and all feedback welcomed 🙂
-> https://github.com/dagger/dagger/issues/9486 this is my (kind of) ToDo list regarding the Java SDK, not exhaustive. But it can gives an idea of what we are working on, and a place to add comments, request features, etc

[winter linden]

Great demos @jagged flicker @wispy tapir @light garden !

[wise ermine]

Hey guys, the get involved part on this page https://dagger.io/ is leading to https://ge/

[sharp marsh]

thx! fixing now!

[chilly arch]

Fixed! Thanks for reporting

[chilly arch]

For anyone that missed the Community Call today, you can check out the full recording here - https://youtube.com/live/jkhSojdfuuM

[tired moth]

Are we going to support dagger in https://modelcontextprotocol.io/introduction

Check this also https://www.pulsemcp.com/servers

[winter linden]

@tired moth yes, with the llm branch (See #agents ) it becomes trivial to expose any Dagger object as a MCP endpoint (MCP-to-Dagger). We also need a way to do the reverse - wrap any MCP server in a Dagger API. Still exploring how to do that - could be a good use of the "magic SDK" tech that @wispy tapir is working on

[tired moth]

Okay will check it. Thanks a lot for confirming.

[winter linden]

Thinking of cleaning up the Discord to simplify... How do you all feel about this:

1. Remove the categories "USING DAGGER" and "MAKING DAGGER"
2. Consolidate #1075928318802657340, #1030538312508776540 , #links and #maintainers at the top-level
3. Retire #1121837200712142878 (redundant and not really used)
4. Retire #daggernauts (why not just use #general, the distinction between "developing Dagger" and "using Dagger" feels artificial anyway)
5. Remove the "USING DAGGER" and "MAKING DAGGER" categories

The result would be:

• 6 channels in one place, instead of 8 channels in 3 different places
• Only one extra category, DAGGER AND... for talking about Dagger + your favorite language or tool

WDYT?

[slender charm]

Although I will miss #daggernauts
I am a huge fan of consolidating and simplifying. Your proposal looks great to me.

[cunning jolt]

sgtm

[ancient escarp]

Might it be useful to allow dagger to act as a docker builder? Would allow integration into existing tools especially docker compose, so you can use your dagger functions to build containers and use them for local dev environments

[winter linden]

Yes absolutely. I would say that's still the number one use case of Dagger - most people start with some sort of container build, then expand from there

@ancient escarp do you have a specific integration in mind that is not possible today? Like a more tight compatibility bridge with docker tooling specifically?

[ancient escarp]

Well I have a dagger pipeline that builds my containers; but using them locally is a pain; you either need to export and import them or run a local registry.

My thinking was if there was a dagger tool that implemented the interface needed for build kit to use it as a builder; there could be easy integration into existing tools by configuring the docker engine

[winter linden]

Ah I see. You're thinking of a buildkit frontend, so you could embed a Dagger pipeline in your Dockerfile with the #syntax.. line at the top?

[ancient escarp]

That'd be one option available; another could be to define a dagger function to run in the docker compose file and have compose call it to build the container

[winter linden]

Mmm, I don't think docker-compose has an extension point that allows that..

Deeper integration with Docker tooling

[sick monolith]

can we put the channel description at least with the famous developers developers developers

[stoic knot]

If I use Dagger Cloud for my build results, is there an API I can hit to get history?

can I sub to any event stream or setup a webhook?

[winter linden]

not yet but we're building the event infrastructure for this

[winter linden]

It turns out you can run "AI agents" on Dagger, because LLMs fundamentally work like builds: you can model their state as a DAG of immutable objects transformed by repeatable functions. So when you need to integrate LLMs with your software (aka "agents") it is very helpful to have a software platform that works the same way. The result is that you can "contain" the LLM's environment using Dagger.

We explored this concept at an AI devtools meetup last week: https://x.com/solomonstre/status/1891205257516003344

Solomon Hykes (@solomonstre) on X

Just like we containerized webapps, we'll have to containerize AI agents... but how? What first principles led to containers in the first place, and how do they transpose to LLMs?

This was my first talk at an AI meetup, but I doubt it will be the last!

Recording 👇

X · Feb 16, 2025, 7:17 PM

[winter linden]

OK I'm doing it 🙂 Thanks for the feedback everyone

[proper falcon]

Hey friends 👋 I think I'm missing something obvious, but how do I get both the stdout and stderr for a WithExec command? I tried:

func (m *Test) CombinedOut(ctx context.Context) (string, error) {
    ctr := dag.Container().From("alpine:latest")
    stdOut, err := ctr.WithExec([]string{"sh", "-c", "echo \"This is stdout\" && echo \"This is stderr\" >&2"}).Stdout(ctx)
    if err != nil {
        return "", err
    }
    stdErr, err := ctr.Stderr(ctx)
    if err != nil {
        return "", err
    }
    return stdOut + stdErr, nil
}

Then I get back no command has been set: stderr requires an exec I was kind of expecting a CombinedOutput method on the container that would return both.

[proper falcon]

To answer my own question, this is how you do it:

func (m *Test) CombinedOut(ctx context.Context) (string, error) {
    ctr := dag.Container().From("alpine:latest").WithExec([]string{"sh", "-c", "echo \"This is stdout\" && echo \"This is stderr\" >&2"})
    stdOut, err := ctr.Stdout(ctx)
    if err != nil {
        return "", err
    }
    stdErr, err := ctr.Stderr(ctx)
    if err != nil {
        return "", err
    }
    return stdOut + stdErr, nil
}

My WithExec wasn't being set on ctr, because I had WithExec and Stdout on the same line - once I moved WithExec out everything works 🙂

[slender star]

<@&1166745944444899449> SDK is 🚀 launched!
#announcements message

This was our first Community SDK and not our last 🙂
A huge thank you to Paul Dragoonis, John Charman, and Chris Riley for making this happen!

Check it out in #php 🐘

[sand gorge]

We are setting up for PHPUK conf. Dagger socks! To prevent them from getting cold feet on trying Dagger PHP SDK

[sand gorge]

Mission complete.

300 bags done. Only got 120 socks from Miranda, so we are at around 45% capacity 🙂

Decent for short notice tho 👌

[winter linden]

Amazing

[chilly arch]

Great, good luck tomorrow!

[sand gorge]

Dagger is on all the signage. Top and center.

Tomorrow on stage at opening remarks, I will mention all sponsors, and promote dagger to everyone, in the process

And then I have selected Chris Riley (@carnage) to give the 1hr dagger presentation

[earnest jacinth]

Will there be a recording/slides available afterwards?

[sand gorge]

Yes. We actually upload talks to YouTube within 24hrs.

At PHPUK conf

Coz that's how we roll

😁😛

[earnest jacinth]

Thats awesome, looking forward to it

[ancient escarp]

quick Q: does dgger use the docker auth if available? eg if I do docker login <stuff> will dagger use those creds when i call container->publish() ?

[winter linden]

yes

[weary estuary]

Hello! Is the Dagger Cloud visualization open source? Is there an option to both self host it and expose it locally as an alternative to the subscription model?

[winter linden]

Hello! No, it's not open source. If the issue is compliance / data controls rather than willingness to pay, we're happy to get on a call and talk about enterprise needs 🙂 (you can DM @slender charm about this)

Alternatively, dagger engine emits standard open telemetry, so you can use a third party observability tool (although you will only see what a non-Dagger-specific tool can visualize)

Also note that Dagger Cloud has a generous free tier

[weary estuary]

Sounds good, thank you!

[tired moth]

@winter linden why we are not having separate channel for dagger cloud?

[warm temple]

If there are issues or requests for dagger cloud feel free to drop them here 🙂

[blazing flume]

hello。guys！ where i can strudy go+Wasm? is there any go+wasm ui libs now?

[sudden ocean]

You may learn from https://dagger.io/blog/replaced-react-with-go

[subtle zenith]

is there a tag for dagger on stack overlofw ?

i'am at 1154 on stack and it would require 1500 to create a tag on stack overflow may i suggest you to create one ?

[slender charm]

This is a good idea @chilly arch do you know if anyone on our side has enough street cred to make a new tag? 👼

[subtle zenith]

there is a dagger tag but it's a java lib :/

[slender star]

we could make a dagger-io / dagger.io tag maybe?

[subtle zenith]

it's your brand 🙂 but thanks, even for you for monotiring the rss of your tag 🙂

my class will be the 20 march, thanks for your help

[winter linden]

I agree getting our own tag would be great. Not sure what we should call it, but we can figure that out

[stoic knot]

fyi, seeing ERR failed to emit telemetry error="traces export: failed to send to https://api.dagger.cloud/v1/traces: 500 Internal Server Error"

[elfin frigate]

looking into it, thanks!

fyi, seeing `ERR failed to emit

[earnest jacinth]

You got a link for this?

[sand gorge]

Hey folks! I'm looking to hire a Dagger Freelancer/Consultant for a customer, who is looking to bring Dagger into their CI/CD processes, for their 4 development teams.

Can you please DM me if you're interested in this 🙂 I'm looking to start building up the Freelance Daggernaught Pool 🙂 We have 1 company interested now, but more will follow, in the future, for sure

EDIT: or if you know someone, and want to point me in their direction, I'll go reach out to them myself

Thanks

[coarse hatch]

Sorry if it's been already asked (i'm not a Typescript dev): are there compat issues with Bun that prevent it from being the default runtime when doing dagger init --sdk=typescript ? So far it's a been a joy using Bun over anything else.

[winter linden]

cc @wispy tapir 🙂

[wispy tapir]

Hey Tibor 👋

There’s no specific compat issue, normally the TS SDK picks up the runtime depending on your local setup or will default to node 🙂
You can also configure which runtime to use from the package.json: https://docs.dagger.io/configuration/modules/#bun-1

[coarse hatch]

Yes that’s what I use. But I guess my question is if it would make sense to default to bun instead of node for an empty project ^^

[wispy tapir]

I had this thought few months ago since bun is much faster/easier but most JS/TS people are using node, so it makes more sense to default to the most common dev tool.
There's an issue related to that topic here: https://github.com/dagger/dagger/issues/8689 for more context

[sand gorge]

Do we have a blog post on using the github actions cache, with Dagger volume cache.

I'm being asked on this.

It's a blocker for someone to start using PHP SDK in their company, as it's super slow atm

[sick monolith]

This was discussed previously, but its tricky and i think not recommended. @warm temple explained a bit behind this and from memory, would require mounting a lot of things and cause potential side effects

Maybe something has changed this that discussion but it wasnt all that long ago

let me try and find a discussion, the thread it probably still here on discord

[sand gorge]

So how can people realistically use dagger instead of github actions?

Where is the cache between runs on the same repo

Companies can't adopt this without a cache.

Their pipelines will have massive performance problems.

[sand gorge]

Daggernaughts. Me and @narrow nymph at Fusion meetup in England

[chilly arch]

We're doing something slightly different tomorrow for the Dagger Community call with a Dagger SDK Take Over!

Join us tomorrow to celebrate the latest SDK additions, and see them in action. @slender star and I are looking forward to hosting @tropic remnant , @paper grove , @sand gorge , and @jagged flicker !

Streaming links:

• https://x.com/dagger_io
• https://youtube.com/live/qQNgitu-AxY

[sand gorge]

Cool. Sounds good to me.

[sand gorge]

The website doesn't seem to have a clear example of using dagger.json to exclude directories when mounting

{
  "name": "laravel-app",
  "engineVersion": "v0.16.1",
  "sdk": {
    "source": "php"
  },
  "source": "dagger",
  "exclude": ["./vendor"]
}

Is this the correct exclude syntax?

EDIT: I tried this: "exclude": ["**/vendor"] also, still didn't work.

[winter linden]

does vendor get copied when loading the module? Normally only the source dir (./dagger in your case) should be copied I believe. Or is ./dagger/vendor what you're trying to exclude?

[sand gorge]

Nope. I'm trying to exclude ./vendor it's in the project root dir

Note: I have this as part of my args

  #[DefaultPath(".")]
  public Directory $source,

[slender star]

So you want exclude from your context directory, not from the module load.

We need to differentiate that and link one to the other
here https://docs.dagger.io/configuration/modules/#file-and-directory-filters and https://docs.dagger.io/api/default-paths/#for-all-other-cases

like this @sand gorge ? https://docs.dagger.io/api/filters/#pre-call-filtering

[winter linden]

pre-call filtering is what you want I believe 👍

[sand gorge]

okay, yeah you're probably right

just like .dockerignore

[slender star]

now we just need the PHP version doco'd 🙂

also https://github.com/dagger/dagger/issues/9490

[sand gorge]

bug report - Konsole terminal on Kubuntu thinks the full stop (period) is part of the URL, so you click it and get a 404

[slender star]

now we just need the PHP version doco'd

[ionic plume]

Hi, I try to upgrade to dagger 0.16.1 but some module failed. I have temporary remove all depencies from my dagger.json. The I have impoted one by one too found the hell. It no more working when I try to install local dependency dagger install ../golang. I look the release note and I have already added "include": ["../golang"], on my dagger.json. But it not work.

I get
! failed to load dependencies as modules: failed to load module dependencies: select: failed to call module "golang" to get functions: call constructor: process "go build -ldflags -s -w -o /runtime ." did not complete successfully: exit code: 1 │ ✔ 2386e42091822984 load dep modules 0.4s Error: failed to update dependencies: input: moduleSource.withDependencies.withEngineVersion.generatedContextDirectory failed to load dependencies as modules: failed to load module dependencies: select: failed to call module "golang" to get functions: call constructor: process "go build -ldflags -s -w -o /runtime ." did not complete successfully: exit code: 1

[lime snow]

We've possibly encountered a bug: Dagger did not invalidate our cache when we made changes to the dagger.json.

Steps taken:

1. We updated the pin of one of our dependencies from an old hash to the latest hash.
2. We ran CI
3. CI used the cache that installed the dependency from the old hash.

[winter plank]

Hey folks general question i can't find a good answer to: should i add dagger/sdk to my gitignore, or should i check those files into version control

[cunning jolt]

This is up to you. It works regardless and dagger re-generates them when it's run anyway. I personally don't push generated code so my repo is a bit cleaner. They only help with IDE integration so not much of a benefit unless it's in your local workspace. https://docs.dagger.io/api/ide-integration/ suggests not pushing the generated code as they are by default added to .gitignore.

[analog raven]

Starting in a few minutes!

[slender charm]

Is there an issue with the youtube stream? Still pending it seems

[analog raven]

Having a couple technical difficulties, should be starting any second now!

[slender charm]

Thanks!

[chilly arch]

https://github.com/jhaumont/enter-the-daggerverse

[slender star]

@tropic remnant @paper grove can't wait! 👆

[chilly arch]

Thank you to everyone that joined the community call! If you missed it, you can check out the recording here:

https://www.youtube.com/live/qQNgitu-AxY

Resources:
https://github.com/jhaumont/enter-the-daggerverse
https://gist.github.com/dragoonis/c48d41bbb4212e4aa9f1932daeed6559

[winter plank]

Does anyone have a recipe for stripping node build dependencies? I need to do a full npm i to install build tools, but they're ending up in my deployment artifact and increasing the size by a lot. How could I blow away my node_modules folder after build and do a new npm i --omit=dev 🧐

[neon warren]

Usually best to call npm ci and I think that will remove dev dependencies (afk at the moment)

[winter plank]

I'm now getting this weird opaque error:

Error logs:

✘ .packageLambda(
│ │ astroKey: ✔ secret(uri: "env://ASTRO_KEY"): Secret! 0.0s
│ │ awsAccessKeyId: ✔ secret(uri: "env://AWS_ACCESS_KEY_ID"): Secret! 0.0s
│ │ awsRegion: "us-east-1"
│ │ awsSecretAccessKey: ✔ secret(uri: "env://AWS_SECRET_ACCESS_KEY"): Secret! 0.0s
│ │ awsSessionToken: ✔ secret(uri: "env://AWS_SESSION_TOKEN"): Secret! 0.0s
│ │ githubPackagePat: ✔ secret(uri: "env://GITHUB_TOKEN"): Secret! 0.0s
│ │ source: no(digest: "sha256:365f999efab28474b5d0dbdcdba66c1fda95b48dba8e8d00853ba17489c576f5"): Missing
│ │ workspace: "development"
│ ): File! 2.4s
Error: GraphQL Error (Code: 422): {"response":{"status":422,"headers":{}},"request":{"query":"\n      { c
tainer  { from (address: \"joshkeegan/zip:latest\") { withDirectory (path: \"/build\") { withWorkdir (pat
 \"/build\") { withExec (args: [\"zip\",\"-r\",\"lambda.zip\",\"node_modules\",\"index.mjs\",\"dist/serve
\"]) { file (path: \"lambda.zip\") { id  }}}}}} }\n    "}}
Error: API Error
! process "tsx --no-deprecation --tsconfig /src/server-islands-demo/dagger/tsconfig.json /src/server-islands-demo/dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

[winter plank]

This is really throwing me off:

source: no(digest: "sha256:365f999efab28474b5d0dbdcdba66c1fda95b48dba8e8d00853ba17489c576f5"): Missing

because i'm passing in --source!

[warm temple]

Hmm I'm not sure about that error, but an option to remove the node_modules would be something like

.withExec(["npm", "i"]) // full npm install
.withExec(["npm", "run", "build"]) // or whatever build steps you need
.withoutDirectory("node_modules") // removes the current node_modules dir
.withExec(["npm", "i", "--omit", "dev"]) // do your npm i without dev deps

[winter plank]

good tip, but i sadly i don't think that's the issue. the build function is passing

oh maybe it is

[warm temple]

Got it. I don't think I've seen that kind of API error, but it looks like it's between the dagger client and engine. I think the Missing thing is normal

[winter plank]

this is passing:

@func()
  package_lambda(
    source: Directory,
    workspace: string,
    astro_key: Secret,
    github_package_pat: Secret,
    aws_region: string,
    aws_access_key_id: Secret,
    aws_secret_access_key: Secret,
    aws_session_token: Secret,
  ): File {
    const built_dir = this.build(
      source,
      workspace,
      astro_key,
      github_package_pat,
      aws_region,
      aws_access_key_id,
      aws_secret_access_key,
      aws_session_token,
    );

    return dag
      .container()
      .from("joshkeegan/zip:latest")
      // .withDirectory("/build", built_dir)
      .withWorkdir("/build")
      .withExec(["zip", "-r", "lambda.zip", "/etc/passwd"])//"node_modules", "index.mjs", "dist/server/"])
      .file("lambda.zip")
  }

but when i put back the use of built_dir it's failing

but what's wild is that build itself reports success

[warm temple]

is that when you call build directly or with the exact snippet above (the withDirectory commented out)

[winter plank]

if i call build directly, I get a success

[warm temple]

and with the snippet above and the built_dir uncommented you get the 422 API Error every time?

[winter plank]

oh mylanta, i just figured it out.. noticed that my sdk dir was gone, reran dagger develop, and a type error appeared

apparently i accidentally removed return from my build

[warm temple]

Phew 😅 It sounds like dagger could do a better job of bubbling that error up!

[frail mulch]

hi folks!

@warm temple can I ask you some weird questions?

[cunning jolt]

no(digest:) Missing spans

[sharp marsh]

good to see you around Mauricio!

[frail mulch]

good to see you around Mauricio!

[proper falcon]

Hey friends 👋 Since upgrading from 15.4 to 16.2 , I'm seeing a line starting Module.dependencies: followed by around 40k lines of output at the beginning of every pipeline run, without any -v's. Checking in here before I file a bug, in case I'm doing something silly.

[jagged flicker]

A useless, but fun, module 😅
I'm working on the Java SDK.
I wanted to render markdown content.
In Go that's easy, I would use glamour library. But in Java, I don't know.
So I wrote a Go module for that:

package main

import "github.com/charmbracelet/glamour"

type Glow struct{}

func (m *Glow) DisplayMarkdown(str string) (string, error) {
    return glamour.Render(str, "dark")
}

(yes that's the full code)
And to see how that works, I wrote a Java module for that: https://github.com/eunomie/java-glamour

So yes, that's where it starts to be useless, as glow binary is probably better.
But beyond that, it means that I can easily use a Go library inside my Java code, without to really notice it.

That's it for me, you may now return to your regularly scheduled programming 😄

[stoic knot]

Is it safe to share *dagger.Container or *dagger.Directory across dagger clients/connections in the Go SDK?
(it looks like it so far)

[winter linden]

You mean their ID?

I believe not 100% safe.

But if you wrap in dagger run, or dagger call, or dagger shell, then it's all the same session and then it's 100% safe

[stoic knot]

I have a dagger run that wraps the whole thing. Trying to find a way to work around the GAR auth token expiration issue in long builds. So this attempt is to establish a new connection for each component of the 20+. Will doing so refresh the auth tokens dagger is managing under the hood?

#1346181024975163413 message

#1316460168942190684 message

Ideally I wouldn't need to do this connection refreshing within a session

[winter linden]

I want to share this one-liner because it is so much easier with dagger than without...

• Goal: spin out a subdirectory of a git repo (github.com/dagger/agents/toy-programmer) as its own git repo (github.com/shykes/toy-programmer)

• Without dagger: I have to research obscure git commands; setup temporary git repositories and branches; be careful not to brick my environment between iterations.

• With dagger: I already have the GitRepository.FilterSubdirectory() function implemented. It's fully repeatable. I can just pipe my git repo state through it, until the output is what I want. Then the final command:

dagger -m github.com/shykes/git <<EOF
clone https://github.com/dagger/agents | filter-subdirectory toy-programmer | directory | export .
EOF

[stoic knot]

This did not work, GAR auth has been a pain with long running sessions

[torpid maple]

How does Dagger work internally, are there any good resources or talks on the subject?

My guess is it's making heavy use of moby?

[jagged flicker]

I'll let the real experts answer, but my understanding is it's more a heavy usage of Buildkit. All the dagger commands are translated in a graph that is then executed by buildkit. Moby allows to run containers, but Buildkit as this notion of graphs. That's what is used for instance to build multi stage dockerfiles in a more classical docker world.

[narrow nymph]

yeah pretty much that! overall, we're not really integrating with BuildKit as a library (which is why you can't use it as a build frontend), but instead by consuming different internal components, and building them into dagger

[torpid maple]

How do I stop dagger from automatically provisioning the dagger engine container? I've created a custom runner but it keeps getting killed and recreated by dagger.

[narrow nymph]

you can set DAGGER_LEAVE_OLD_ENGINE

in the environment of the client

it shouldn't be trying to kill it though, assuming that the versions match 🤔

[torpid maple]

It appears I missed this.
https://docs.dagger.io/configuration/custom-runner/#connection-interface

After the runner starts up, the CLI needs to connect to it. In the default situation, this will happen automatically.

However, if the `_EXPERIMENTAL_DAGGER_RUNNER_HOST` environment variable is set, then the CLI will instead connect to the endpoint specified there. This environment variable currently accepts values in the following format:

Setting that env var stops the CLI from automatically provisioning the dagger engine container and just uses the one specified instead.

[stoic knot]

finally got this working, will update one or both of my threads. The details were pretty obscure, is this something that would be good for the docs (cc @chilly arch )

[fleet remnant]

Issue DOCS-408 created.

DOCS-408 - GAR auth token expiration issue in long builds

See user issue here - [#general message](#general message)

Status

Triage

Documentation · Mar 4, 2025, 6:49 PM

[stoic knot]

#1346181024975163413 message

[ancient escarp]

I'm scheming again... Is there a way to extend an existing SDK? I'm thinking I'd like to do something like dagger init --sdk=github.com/laravel/laravel-dagger --source=dagger where the laravel-dagger SDK extends the PHP SDK; it'd need to defer to it for most stuff; however I'd want to override the codegen to generate an example pipeline which is Laravel specific and has a bunch of functions prebuilt for running laravel tools (It'd also ideally be written as a php dagger module)

[sharp marsh]

is "extending" the right approach here? Since after all, isn't laravel PHP? Are you only looking for to updating the example pipelines dagger init generates? If that's the case, maybe we could conisider a different way of doing this vs having to extend an entire SDK

[ancient escarp]

Perhaps not; essentially what I'd like is to generate a different module template. Bonus if it could auto install some modules as well

[winter linden]

Well SDKs are just modules, maybe your SDK module could install the PHP sdk module as a dependency, and call it as needed?

[sudden ocean]

I have been toy it with Elixir SDK while ago here https://github.com/wingyplus/elixir-script-sdk

The idea is include the SDK as a dependency and use it in your SDK. See https://github.com/wingyplus/elixir-script-sdk/blob/main/elixir_script_sdk/lib/elixir_script_sdk.ex for the example

[ancient escarp]

I've got something working locally; proxies calls to the underlying PHP SDK and init's a module just fine

question now is how to modify the init process. I tried adding a dependency to the modSource object before calling codegen but it's blowing up

$modSource = $modSource->withDependencies([dag()->moduleSource('https://github.com/sai-php/sai')]); Is that the correct way to do it?

[ancient escarp]

I think it is the correct way to do it; however there may be a bug in the php sdk's handling of arrays of idables

worked around it; but the dependency wasn't installed 🤔

    #[DaggerFunction]
    public function Codegen(ModuleSource $modSource, File $introspectionJson): GeneratedCode
    {
        $modSource = $modSource->withDependencies([dag()->moduleSource('https://github.com/sai-php/sai')->id()->getValue()]);

        return dag()
            ->phpSdk()
            ->codegen($modSource, $introspectionJson);
    }

[tired ridge]

Hi, can anyone tell me if there is some standard for how people do semantic versioning with Dagger?
Is the npm semver used? Can anyone link me an example or some documentation about this topic?

[sand gorge]

@ancient escarp can't we just pass a new flag to the dagger init call?

I really do not want us to diverge and have more than 1 SDK for the PHP community at such a crucial time, in early adoption.

So we stick to 1 SDK, since you don't need a new SDK you just want a new template generated, which is a very trivial thing, and we can pass a flag in so we drop in a new template.

Think of it like a "starter kit" for respective PHP communities. Drupal. Magento. Symfony. Laravel ..etc

It's been on my mind.

Cool?

[ancient escarp]

Don't think it's possible to pass flags to dagger init; it's not a regular function call. I tried this as a regular module but that didn't do the codegen properly; was missing dagger.json and had no way to add dependencies. Still playing with options; will be a long way from my current experiments to having framework specific SDKs; but I want it to be possible.

[coarse hatch]

Any reason why there's no Terminal() method on a dagger Service ? I'd like to poke around the container of the running service

[winter linden]

buildkit

😛

[sand gorge]

Alright. Let's schedule a call for early next week about it? @ancient escarp

[winter linden]

There is a feature in development for allowing per-SDK configuration. /cc @cerulean wraith @narrow nymph @rain oriole

It was prompted to allow things like passing proxy configuration to language-specific package managers. But configuring SDK support could be another use for it

[ancient escarp]

Might be the route to go down then

However I do seem to have hit a wall getting the SDK init to be able to add dependencies and make other modifications to the module

[winter linden]

Doesn't have to be either or, since anyone can develop a SDK and host it anywhere without asking for permission. So you could make a specialized SDK on the side, test it easily as a specialized wrapper SDK; then in parallel we look for clean ways to embed it as a feature of the official PHP SDK

one could be an incubator for the other

[ancient escarp]

Mostly experiments atm - pushing at the limits of what dagger can do

[chilly arch]

**Hey Daggernauts! Going to KubeCon EU or around London on April 1st? So are we! **

We'll be hosting a Dagger Hack Night on April 1st in London.

​Come hack with us in a cool space with great food, build something new, and connect with awesome people.

**If you are interested, please register to secure your spot! ** https://lu.ma/hlx7s6ym

[hardy umbra]

Hi all
We are looking for input on whether we are using Dagger in an unintended way/using an anti-pattern.
This has come about because we are struggling with controlling the user facing output when containers fail before they reach step that returns stdout/stderr (I will describe this issue further in a GH issue depending on the feedback on the following question).

The way we use dagger is that we have a github action that calls a single dagger module function called CI . This function then calls Build, Test etc. and collects the results. We have been experimenting with different return types:

• (string, error)
• (*dagger.container, error)
• ([]*dagger.container, error)

It seems the only way the Dagger CLI can handle the above mentioned type of error (eg. our Test container can fail while installing prerequisites, before it actually outputs anything) is if we return a dagger.Container. However, with our setup we would need to return a slice of containers but the Dagger CLI cannot handle that in the success scenario (Error: json: cannot unmarshal array into Go value of type string)

So this is why I ask, are we holding it wrong? Is the intended use case to execute each step of our pipeline in a separate dagger cli call, thus forgoing any deduplication the engine might be able to do?

[split olive]

We are also using the ci function approach.
We have one call in the GitHub action, see the GitHub pipeline
The ci function collects all artifacts and results and returns a Directory (Dagger function).
With this content we can handle unit results, reports, artifacts inside GitHub.

[narrow nymph]

this is actually a known bug I think: https://github.com/dagger/dagger/issues/8202

we actually had a fix for this in https://github.com/dagger/dagger/pull/9425, @rain oriole was there a reason to not merge that one? (I don't see a reason in that PR)

[rain oriole]

I don't think that was the right approach. I remember discussing it, but not sure where. I actually started working on that last summer believe it or not, but then other things took over. It's been on my mind among the next things to solve.

[tired moth]

Can anyone help me to get confirmation, as of now dagger can be used for local development like with dagger watch functionality?

[slender star]

Dagger Shell is great for local dev, but you need to quit and reload yourself still.

[tired moth]

any issue to track that feature development?

[slender star]

I was going to ask you that? I know you’ve asked this question dozens of times. Did you ever find an issue or open an issue?

[slender charm]

This one - I think you've already seen it. https://github.com/dagger/dagger/issues/6990

[tired moth]

ohh this same...issue...thanks man...i am aware about this issue...i thought if anyone has started working then issue or PR might change so. nothing else. thanks a lot for your help.

[slender charm]

Happy to help. No one is working on this yet, but I agree this would be a really great addition.

[slender star]

If you or anyone else opens a PR it will/should say “fixes #6990” in the body and will be linked

[tired moth]

We are a rust and typescript team, if dagger was written in rust then we may have done tons of contributions, we have never ever used go...sorry...

jfyi here is another feature request for which we are waiting from day one...https://github.com/dagger/dagger/issues/9107

[hardy umbra]

Thank you for the info. Sounds like we should just try and work around it for now

[stoic knot]

Just hooked our container builds up to our private yum mirror / sync. Now we can be sure that we have the exact same packages both in and outside of containers. Dagger services are pretty cool

[clever burrow]

Hey guys! I'd love to see if we could get a mini hackathon going in the dagger office! I feel like it'll be an exciting opportunity for devs to learn and use dagger. IDK how I can help set it up, but I'd love to help out in any way possible

[chilly arch]

Hi! Do you mean at an office you work at or a "Dagger office"?

The Dagger team doesn't have an official office since we are fully remote, but we do have an upcoming hack night coming up on April 1st in London - https://lu.ma/hlx7s6ym

And another one coming up in SF on May 14th - landing page coming soon!

[hardy umbra]

Thank you very much for these examples

[torpid maple]

What's the recommended way of getting things like TARGETARCH and TARGETOS like you would get in Dockerfile?

[cunning jolt]

Depends on the SDK, there's an example here for Go https://github.com/dagger/dagger/blob/main/modules/go/main.go#L177

[old fiber]

anyone using cursor/claude3.7 to generate dagger modules? this is pretty wild

i feel like i need to take up artistic landscaping so i can ensure i remain gainfully employed in the future 😢

lol - i love the fact that you bask in my tears

i'm here for it 😉

[winter linden]

@warm temple is getting pretty close 🙂 https://youtu.be/Vqqz052vmHc?si=lwZtbCmhwuieS_pd

[old fiber]

does he need any artistic landscaping done? 😉

[slender star]

Can also get and set with platform depending on your needs

✔ container | platform  0.0s
linux/arm64

✔ container --platform linux/amd64 | platform 0.1s
linux/amd64

dagger -c 'container --platform linux/amd64 | platform' | cut -d/ -f1
linux
dagger -c 'container --platform linux/amd64 | platform' | cut -d/ -f2
amd64

🪄 current

dagger -c 'container --platform current | platform' | cut -d/ -f1
darwin
dagger -c 'container --platform current | platform' | cut -d/ -f2
arm64

[slender charm]

No! That was a supporting heart haha

[old fiber]

lol fineeeeee. either way i'm here for it

[umbral nimbus]

Dagger agent looks promising

[winter plank]

Hey all! The monorepo docs are leaving me a little lost: https://docs.dagger.io/api/module-structure/#monorepos my setup is like situation #1, but the docs don’t really describe how to functionally get to the place with a top level orchestrator module, and submodules. Do I dagger init in each submodule? What does the orchestrator look like? Anyone have a reference?

[winter linden]

Hi! We'll look into making the docs more clear.

The simplest approach is to simply start with 'dagger init' a separate module in each logical component of your monorepo that you want to daggerize. Then you can develop each module independently, with the functions and types relevant to each.

Anything else is optional.

Then there's the question of "top-level module". This one really depends on what top-level operations you want to expose at the root of the monorepo. Is there a "top level build"? Maybe a top-level e2e test suite? That's the kind of functions you would expose in that top level module. That module can wrap other modules simply by installing them as dependencies, and calling their functions from its own functions. Just like a regular library might wrap another library

[spiral bough]

Can someone please share the slide decks and any visual materials related to using Dagger as an LLM agent manager?

I’ll be speaking on this topic

based on the following lecture:

https://youtu.be/XWO_3My2eVU

[chilly arch]

We’re working on getting an update dagger deck done soon! I can share it with you as soon as we have it.

What date is your session?

Hey Daggernauts!

Solomon is speaking at SCALE right now.

You can watch the Keynote Live here:
https://www.youtube.com/watch?v=umCTEjTYoWs

[split tangle]

I just did the basic setup as described on https://docs.dagger.io/ai-agents/ with llama3.2 and ran
dagger -m https://github.com/shykes/toy-programmer
go-program "develop a curl clone" | terminal

once it gets to the terminal, all I see in the dir is assignment.txt
I expected a go program, what am I missing?

[winter linden]

The llm might have screwed up. Do you see any messages from it in the terminal history? You should be able to scroll up (need to exit that terminal first).

If you setup tracing in dagger cloud (just press 'w' while the llm is running) then you could see a full trace of what the llm did

you can also share the trace url

[split tangle]

I exited the terminal and see messages:

[winter linden]

yeah it looks like a "prompt engineering" issue. The model gives up too soon and tells you to do it, basically. I tested that toy-programmer prompt on gpt-4o primarily. You might have to tweak the prompt so that llama is more persistent and less lazy 🙂

[spiral bough]

I have not scheduled anything yet but want to start preparing for it.
It is

not a thing of high priority for me for now

[merry crest]

hi! I've been a big fan of Earthly but it seems like they're really pivoting away from CI due to $$$. I'm considering Dagger as an alternative but I noticed that it also seems to be pivoting from CI to AI.

am I going to regret migrating from Earthly to Dagger? I realize the answer here is probably going to be a bit biased haha

[loud briar]

Earthly

[split tangle]

I switched to 4o-mini and it actually write the program and it works. It exists in the container. I'd like to capture it to my computer. does dagger setup a volume mapping to the directory I've launched it from?

[winter linden]

You can export to the host filesystem using the Dagger API

if you have a Container type you can call Container.directory(".").export("./some/where") to export the working directory from the container.

so in your case: goProgram().directory(".").export("somewhere") since goProgram returns a container

[alpine dock]

Hi, I’m new to computer programming and science, and I am taking APCSA at my school right now. Is there any tutorial on how I can implement dagger into my terminal and use GPT ?

[stray tendon]

Was this talked about?
Dagger is trending on GitHub
https://github.com/trending

[slender charm]

That’s so cool! 😁

[slender charm]

Hey! Welcome check out the tutorial here as a good place to get started https://docs.dagger.io/ai-agents

[winter linden]

Dagger is trending on GitHub 🙂

If you haven't yet, now would be a great way to star the repo ⭐ to keep that fire burning! https://github.com/dagger/dagger

[winter linden]

Thanks Orsan we hadn't even noticed !

[neon crescent]

Hi Folks — I'm Harsh from the LocalStack team. How can we get in touch regarding the "Dagger Powered" program? I saw that you folks have been internally using LocalStack for AWS tests and have a LocalStack module (https://daggerverse.dev/mod/github.com/grouville/daggerverse/localstack) and wanted to explore any potential partnership. Thanks!

[sand gorge]

Welcome @neon crescent 🙂

[torpid maple]

How do I detect if the dagger function is running in a CI environment?

Say if it's ran under the dagger-for-github action for instance, I want to perform some conditional behaviour in the function.

[cunning jolt]

It's not possible today unless you pass in an additional parameter to your function to indicate whether you are running on CI or not. I use a --ci flag for it. But automatically detecting the running environment is a big need for us also. I thought there was an issue created for it but doesn't look like it.

[torpid maple]

What is required for this GitHub Action to cache effectively?
https://github.com/dagger/dagger-for-github

At the moment it appears to be doing everything from scratch everytime it runs at the moment.

Does it require a container registry to cache to and from, or does it expect the volumes on the Docker host to persist between runs?

[slender star]

localstack :: Daggerverse

[mystic citrus]

Hey all, we have made some progress on Daggerizing our CNCF Harbor Project. We can do a lot of cool things now.

1. get rid of 2-5k LoC Makefiles, dockerfiles, scripts. We are still working to removing more.
2. build and sign artifacts for different architectures,
3. run the whole application stack locally
4. build multiple favors of the application stack
5. remote debugging application stack, with the debug flavor.

Overall, we are thrilled, one thing that didn't improve is the build time.
This was one of the reasons to move away from Makefiles, we are still in 20 min range to build everything. (on the first run)
We have cache enabled, so locally we can rebuild the stack in a 2-3 min, which is still too much, but on the GH action side we didn't gain any advantage...

Are there ways we might be able to explorer that would allow us to reduce the build times on GH action from 20 min to 5 min?

[split tangle]

What's the context I run this from? it looks like it's a go function call. Do I run this from the terminal prompt that go-program "develop a curl clone" | terminal drops me into or from the toy-programmer prompt once I exit from the terminal? sorry, very new to this environment

[winter linden]

No worries.

(side note I recommend going through the regular Dagger quickstart, even though it's not specific to AI agents, it will get you up-to-speed with the runtime & programming environment in no time)

Short answer: it's all the same Dagger API under the hood. Whether you call these functions from the command-line, curl, or Go/Typescript/Python bindings from a SDK, it targets the exact same low-level GraphQL API. That's why we sometimes use notations interchangeably in discussions

• From the command line: go-program "develop a curl clone" | directory . | export ./curl-clone-example
• From a Go client: dag.ToyProgrammer().GoProgram("develop a curl clone").Directory(".").Export("./curl-clone-examplw")

[split tangle]

sweet!, thanks, that works. I'll go through the quickstart.

I can understand the notation now

[winter linden]

Also to answer your other question: go-program | terminal calls goProgram(), then calls terminal() on the resulting type - which happens to be the low-level Container type. So terminal() is a low-level container operation ("give me a shell in this container") nothing special beyond that.

Here's the Container type reference:

• Low-level graphql: https://docs.dagger.io/api/reference/#definition-Container
• Go: https://pkg.go.dev/dagger.io/dagger#Container
• Python: https://dagger-io.readthedocs.io/en/sdk-python-v0.16.2/client.html#dagger.Container
• Typescript: https://docs.dagger.io/reference/typescript/api/client.gen/classes/Container

PS. I have a dream that one day, we will have a unified API reference doc 😇 @slender star

[chilly arch]

**SF/Bay Area Daggernauts! **

Solomon will be speaking at AI Demo Days on March 20th. RSVP here: https://lu.ma/ip1yzjx6

[tired ridge]

Hi @winter linden,

If I remember correctly, in the Kubernetes Podcast you talked about a future where storing artifacts on registries could be not needed (or minimal), but instead if a system needs a specific version of an artifact, a just-in-time pipeline (I am probably not using correct technical words) would be triggered to built such object.

Is there some already explored way to do this?
If not, do you know if there are projects actually working on this (even better if around the world of Kubernetes and GitOps)?

P.S.
I'm not sure if this would have been better as a private message. In case let me now.

[winter plank]

did you ever figure this out?

[sharp marsh]

👋 the current state of caching container builds (Dagger, Docker, Racher, etc) in GHA and Gitlab is quite limiting. The main reasons for is is because the primivites that the VCS build providers give you are not best suited for container workloads and generally don't provide the best availability (https://github.com/search?q=repo%3Amoby%2Fbuildkit+gha&type=issues).

The reality is that, different users and workloads have different requirements and environments which makes it difficult for us to ship a solution that will make the majority happy. Having said that, we had an attempt to provide such experience by developing a distributed cache service (#1306996556540543016 message) which we provided as a closed beta feature for a while which we have retired some time ago mostly because of the reasons mentioned above.

A different approach towards getting "the best" possible caching and compute performance if you're not running Dagger on premise, has been our recent partnership announcement with Depot (https://dagger.io/blog/dagger-and-depot) which, for users that can afford the spend, provides the best UX and performance to run Dagger efficiently and in a scalabe way. If you are self-hosting Dagger, we also have some docs and guidelines on how you could operate it and scale it by using kubernetes (https://docs.dagger.io/ci/integrations/kubernetes/).

To wrap this up, we're constantly working on primitives that will allow Dagger to run better anywhere. There are some ideas and initiatives floating around which will ultimately improve the experience to run it in free services like GHA and Gitlab CI. For now, we prefer to be entirely transparent and point you towards a direction that we have tested and have high confidence that it works really well.

If none of the above are a viable alternative and this is currently blocking the adopting of Dagger within your project or company, we're happy to chat more about your specific use-case and try to find an alternative together.

[torpid maple]

@sharp marsh right now what does Dagger use to cache effectively between runs?

I see the dagger-engine container has a volume mount at /var/lib/dagger.

Would it be reasonable to assume if I can persist that Docker volume between runs then things would be effectively cached?

[winter linden]

Would it be reasonable to assume if I can persist that Docker volume between runs then things would be effectively cached?

Yes 🙂 But be careful, it doesn't support concurrent writers.

There is a lot of accumulated infrastructure experience in this server about different ways to do this. Don't hesitate to ask, and share your findings.

[sand gorge]

Github action caching

[sand gorge]

@winter linden what if we made a cache key per PR number .. that way the concurrency is less of a problem. Coz it's unlikely that 1 person will run their code multiple times after they push.

[high meteor]

Hello

I have an unusual scenario 😭
I'm using Dagger (Python SDK) to create an image and would like to execute a command within the container without internet access.
In terms of docker cli it would be something like:

docker run --rm --network none ${IMAGE}:${TAG} my_command

I'm not sure how to parameterize network for withExec
Am I out of luck?

[slender star]

Hello

[torpid maple]

How can I specify the cache-to and cache-from parameters so Dagger will use a registry for caching?

Like so with docker.

docker buildx build \
--push \
--tag private-registry/ci-env:latest \
--cache-to type=registry,ref=private-registry/ci-env:cache,mode=max \
--cache-from type=registry,ref=private-registry/ci-env:cache \
.

[hardy umbra]

Hello

We are having some problems with a engine/runner infrastructure after migating 11.7 -> 16.2.

Our setup is:

• Kubernetes using helm chart
• Self-hosted github actions running on same nodes as engines
• Nodes have 8 vcpu and 64gb memory

We are struggling to run more than 6 runners per node (plus the engine). The symptoms we see when number of runners increase is something that looks like a logjam where something is preventing the runners from completing but once that something is resolved (often us killing the oldest runner) they all finish relatively fast. In some cases however, some runners never finish.

The obvious answer of course is to increase the amount of resources or decrease the number of runners but we see no signs of resource contention as the nodes max out at about:

• 65% cpu
• 65% memory
• 50% disk

Can anyone help us debug this situation. What should we be looking at that we are not?

[torpid maple]

Made this project effectively demonstrating what I'm trying to achieve with Dagger.
https://github.com/jshbrntt/docker-build-cache-demo

I want to run the same Dagger function in a local dev environment, and in the CI environment and have it make use of a shared registry that both clients have access to in order to cache effectively.

[candid gyro]

very general software question (sorry)... generally speaking, how is dagger achieving "live traces" ? IME so far most of otel is usually append-only, where you don't really have any info about the span until it's closed, but dagger is obviously showing "in progress" spans as well. Is that all custom, perhaps smart combination of traces/metrics/logs together or is this something that can actually be done with OTEL traces (e.g. upload span without end time, then update) ?

[narrow nymph]

Performance issues on migrating from v0.11.7 to v0.16.2

[high meteor]

Howto publish images to Depot Registry in Github Actions

Hello.
How do I publish images to Depot Registry via Dagger in Github Actions?
I couldn't find any documentation about this in Dagger/Depot docs.
Do I need to use PAT?

[sand gorge]

Hey! I have a module checked out locally, and I wanna add it to my appliation, so I can bring it into my project, and test it ..

what's wrong with this?

dagger install ../../phpstan-module/

[chilly arch]

See you all at the Community Call tomorrow! We'll have special guests @honest oasis and @rigid pebble 🚀

Join the livestream on X or YouTube

[torpid maple]

What's the best way to have timed cache busting?

For instance you generate some API token that will expire in 30 minutes, to be used in subsequent .with* statements.

How can I store the token in a cache volume, but have the volume expire and the command that generates the token re-run after 30 minutes has elapsed?

[narrow nymph]

i wouuuld probably aim to avoid putting any secret data in a cache volume if at all possible

but if it's not secret, you can kind of hack it by having a truncated timstamp in the filename

so something like https://pkg.go.dev/time#Time.Truncate

time.Now().Truncate(time.Minute * 30).Unix() will give you a timestamp that rounds down to every 30 minutes

[torpid maple]

I don't think the truncated approach would work because you'd have to round it to a fixed interval, rather than 30 minutes ahead of when it was first generated.

[narrow nymph]

yeah, you're right 😦

i am not sure then 🤔 we don't really have this kind of level of fine-grained cache control... yet

[torpid maple]

I was thinking maybe something like this?

const container = dag.container().from('alpine:latest').withExec(['sh', '-c', '[ $(( $(cat .cached-at) + 10 )) -gt $(date +%s) ] && cat .cached-at || (date +%s) 2>&1 | tee .cached-at'])

But I don't think that would work.

[analog raven]

Just a couple more minutes until our community call!

[chilly arch]

AI Agents Quickstart that @rigid pebble mentioned: https://docs.dagger.io/ai-agents

https://daggerverse.dev/mod/github.com/matipan/skunkworks/athlete-workspace

[honest oasis]

Here's an example Dagger shell-native Notebook if you want to check it out: https://github.com/stateful/vscode-runme/tree/main/dagger/notebook

[chilly arch]

Thank you @honest oasis and @rigid pebble for your demos today!

Recording: https://www.youtube.com/live/p6ZhM42LdM4

Resources:
https://github.com/stateful/vscode-runme/tree/main/dagger/notebook

https://daggerverse.dev/mod/github.com/matipan/skunkworks/athlete-workspace

https://docs.dagger.io/ai-agents

[honest oasis]

vscode-runme/dagger/notebook at main · s...

[austere maple]

I have been in the wilderness over the past couple of days configuring a persistent Dagger cache. I hope this will help folks on a similar journey. I am able to cache npm install operations for a couple of sets of ~1200 dependencies for a time savings of 30s per run in Github Actions now.

[winter plank]

You’re getting the Medal of Honor this year

[azure herald]

Hi! Sorry if I have missed this in the docs, but I cannot seem to find it. Does dagger cloud still offer caching? I remember that marketing for dagger cloud was around caching but it seems that It has changed to observability (whis is pretty cool btw). All I can find about caching in dagger cloud is "Pipeline cache visibility" which I guess is not what I was looking for?

If it does not offer caching, is dagger with depot the unique not self hosted option?

I was also very excited to see this PR https://github.com/dagger/dagger/issues/8004. Do we know if dagger will continue working on this in the future?

[honest oasis]

re, agrents running via Dagger shell in Runme works (cc @winter linden)

to do prototyping effectively using multiple cells a long-running Dagger Shell would drastically redcue upstart latency

☝️ Q that came up during the community call yesterday

[winter linden]

@honest oasis sorry I couldn't join yesterday to discuss this live - what I was wondering is if you could solve this simply by wrapping the runme server in dagger run. Then every nested dagger shell would share the same dagger session, without having to make them long-running. Shared session is where most of the performance gains are unlocked

[honest oasis]

i'll try! i suspect there might be some internal issues. however, using a long-running Dagger shell session is definitely possible in one way or another.

[winter linden]

internal issues as in "internal to dagger"?

[blazing ginkgo]

On the note of caching, one thing that I've been looking into is caching volumes remotely. There are some steps in our builds which would benefit greatly from this. So, I tried tinkering with a way to remotely push/pull volumes and this is what I came up with: https://github.com/cbochs/dagger-modules/tree/main/remote-cache

It allows pushing cache volumes to a remote registry/repo and pulling them on subsequent builds to improve builds when the layer cache is invalidated

[austere maple]

I have a need coming up to cache eslint via a volume cache - I'll give this a whirl when I do! I am very curious how the registry cache will scale with cache size. My experience with caching to a registry has been mostly negative.

[blazing ginkgo]

Please do! I think different backends could be supported here, which are not a Docker registry. This just uses a registry since it's the easiest backend to point to with Dagger. However, swapping out code here with a container to push the directory to an S3 bucket (or other) would probably be a relatively simple change

[modern mountain]

Looking for some advice - I'm building a CI/CD using dagger. I've setup dagger to use kubernetes which works fine, and will be triggering it with github actions. I'd like to do automatic version bumping in our project (using poetry), but I'm wondering what the best way to do that actually is. I can have a script run poetry bump, then git commit the change then trigger dagger to build - but it seems like it might be better to do that from within dagger?

[austere maple]

There are a lot of good ways to do this - my inclination would be to have

1. a dagger function that uses your preferred version of a Poetry docker image as a base
2. with a mounted directory (withMountedDirectory) of your git repository ($GITHUB_WORKSPACE in actions)
3. that runs poetry version with your preferred options

then use plain git commands in your GHA workflow to commit that change

but there are results on Daggerverse for https://daggerverse.dev/search?q=Poetry that may do some of this for you

alternatively you could do the commit from a second dagger function if needed. I often opt for native platform tooling when the command involved is something established like 'git commit -m'. In the case of Github Actions you also have some nice support for Actions pushing commits with a token authentication that may or may not have an established pattern in dagger

[honest oasis]

Internal, to Runme; or if you look at it differently how Runme <> Dagger integrate. Working on writing up some docs for your Shell launch next week and will come back with more details after.

[modern mountain]

Thanks for this! This gives me a pattern to look at. One of the things about dagger that I haven't quite internalized yet is taking output and using those values as input; for example, I want to bump the version with poetry, then retrieve that new version and tag my docker image with it before uploading it to the repo

[winter linden]

@austere maple @shrewd plume in case it helps, I have a git module that I use regularly for reproducible git operations. For example I have occasionally used it for syncing one repo to another, with transformations along the way (ie. merging 2 repos, or extracting a subdir of one repo into another). Anyway it made me realize, git operations themselves are often brittle and not reproducible, you can easily brick the local checkout state etc. It feels very good to daggerize them 🙂 fyi

[modern mountain]

Overall this has been my thought as well; while this operation is basically just to commit a new pyproject.toml with a new version, so many of these kinds of things can either get wedged or are not idempotent at all. My least favorite way of buidling a docker image is via a Dockerfile, for instance.

[winter linden]

a small milestone to celebrate 🙂

13,000: our lucky number 😛

[winter plank]

🧐 has anyone tried wrapping services up in dagger modules and providing them as tools to the llm? I do data platform engineering mostly and I’m thinking about this idea of self healing ci, but for my data pipelines. I’m imagining somehow wrapping my dagster jobs up inside of dagger tasks and then when there’s some kind of failure the LLM doing first level triage

[warm temple]

This is maybe a very simple version of what you're thinking but I went for some kind of self healing ci here: #1349235356184350770 message

[winter plank]

Yeah i have watched your video a few times and tbh it was my a-ha moment with agents generally- I’m thinking about the next step which in my mind is interacting with running systems

[warm temple]

Awesome! I know a few of us have started thinking about what a Dagger agent looks like that observes an active system from the outside ( @slender star ). Definitely interested to hear some use cases and I'll keep making demos 🙂

[winter plank]

Something about dagster (lol maybe just the name) makes me feel like there’s some serious synergy between the two systems. I can imagine a DagsterRunLauncher that launches a dagger function and if the asset materialization fails the llm jumps in to triage if not self correct

With the dagster graphql api bound as a service so the llm can get whatever it needs to debug

[warm temple]

Nice that sounds pretty cool, I'll do some brainstorming! I think in general I'd probably have a regular old API client polling (or webhook endpoint listening) and waiting to feed new information to the LLM instead of giving the client to the LLM just because the polling part doesn't need any generative capabilities, but maybe there's some cool reasons to hand it all over

[winter plank]

Yeah, i’m just spitballing at this point. Really still wrapping my head around working with llms and dagsters programming model

[pale nexus]

any chance of getting the code for the application in this demo?

https://www.youtube.com/watch?v=1_-bTOs9Ky0&ab_channel=Dagger

[high meteor]

Howto publish images to Depot Registry

[winter linden]

cc @rigid pebble 🙂

[neon warren]

not really a service, but I did make a postgres module/workspace that I'm using in an agent to discover the database schema/discovery - there is no readme just the code that takes a connection for an external db. Not sure if thats useful or not. https://github.com/jasonmccallister/postgres-workspace

[tired ridge]

Hello,

Is there a way to handle secrets with Dagger? I would like to not expose some credentials (Registry login) in the traces. I wonder if this is possible with received args or in some other way

[winter linden]

Yes Dagger supports secrets natively. See https://docs.dagger.io/features/secrets

[tired ridge]

Thanks!

[lime ferry]

If I wanted to work smoothly with dagger in a monorepo setup, so that I can define the most common functions centrally for the repo, but also customize the build for each individual service or CLI within the repo, what would be the best approach? 🤔

We've been using Earthly so far, and we have an Earthfile at the root of the repo that defines the "regular" build steps for Go. Then, in eg. cmd/my-binary, I set up a local Earthfile that..

1. adds all relevant files to the build setup (so that we only get binary-affecting files included)
2. calculates a hash based on the included files, to use for tagging
3. builds & pushes a Docker image by calling the central build step definition, with a custom image name (eg. my-binary)

Ideally I want to be able to do my work within cmd/my-binary, just running dagger call publish from within that directory, and the "local" workflow for cmd/my-binary would run, calling any top-level Dagger functions as needed...

EDIT:

Searching here in Discord a bit, I found this response from a week ago... #general message

So I'm guessing that mostly, the "answer" here would be just that.. to for example set up a Dagger module inside cmd/my-binary/dagger, and.. maybe define common things in the top-level dagger/ module, and somehow import them from within each component module?

[winter linden]

hello! yes having a dagger module in each component of your monorepo is my preferred approach.

For common logic, yes you can use common modules as you describe. Note, I would recommend having shared utility modules live in a subdirectory rather than the root of your monorepo. This way you can reserve the root as a higher-level "entrypoint" to the whole repo, to implement end-to-end builds for example. It's not mandatory to do it that way, but it works well

another benefit of this approach is that you can model dependencies within your monorepo as dependencies between their respective dagger modules. Particularly useful for cross-platform dependencies that your native tools can't model

[slender star]

I'd like my agent to play my Snake Game over http API 🙂 🐍 🎮

me playing, agent trying...in 🧵👇

[winter plank]

@slender star man that makes me want to try running my application and telling an agent to red team it as part of my ci

[lime ferry]

So.. hmm, given a Directory, is there a convenient way to find out its relative path vs. the root of the git workspace?

If I go for the most obvious way to find out more about the path of a Directory...

func (m *OpensearchReindex) WhatIsMyRelativePath(ctx context.Context, dir *dagger.Directory) (string, error) {
    return dir.Name(ctx)
}

..then I get a rather disappointing / as the result:

❯ dagger call what-is-my-relative-path --dir=.
...
✔ .whatIsMyRelativePath(
│ │ dir: Host.directory(path: "/home/johan/matspar/datahub/cmd/opensearch-reindex"): Directory!
│ ): String! 0.3s

/

[lime ferry]

So.. hmm, given a Directory, is there a

[tame quail]

I hope it's ok to ask a question about the daggerverse here. Currently github.com/shykes/gha is listed at the top as a featured module. It seems it has since been moved into the dagger codebase proper under github.com/dagger/dagger/modules/gha, which is has been made available on daggerverse. Is there a plan to change which one is featured? I assume github.com/shykes/gha is no longer going to be updated

[sharp marsh]

Daggerverse

[barren vapor]

hi, I think there should be a "Resolved" tag in the #1030538312508776540 channel when we create threads, I think it would be way more helpful for the support team helping us.

[high meteor]

Hello

I don't see the #depot channel. Let's create it! 🙂

[hardy umbra]

Cache disk usage
We can see that we write a lot to the disk but read almost nothing. Even if we delete the our engine-pod from the node (thus removing the in memory cache) basically nothing is read from the disk. Is this an expected pattern? We fear we are not using the full potential of Daggers caching capabilities.

[rigid pebble]

Your AI Training Agent That Hurts Your F...

[slender charm]

Good idea! I just added it 😄

[quaint meteor]

Hi there. This is a quick kudo to the team. The support/help you provide us on Discord during our Dagger journey is very qualitative.

[torpid maple]

Is there a way to cancel a dagger call mid run and enter interactive mode, say if something is timing out indefinitely?

[wispy tapir]

You can use -i that will open a terminal on the current container / directory if a failure happens.
I'm not sure you can open a terminal on a specific step though.

If you want to verify something because it's timing out without failure, you can add .Terminal() before the operation that times out to manually test it and understand what's happening

[torpid maple]

Yeah I'm aware of those ways, I was trying to cancel a withExec mid-run because it's never timing out or finishing, so I wanted to Ctrl-C it and have a terminal at that point.

Regarding running services with Dagger?
https://docs.dagger.io/api/services/

What's the best way to pass config to the host environment, imagine you're starting a k3s cluster in a container as a service.

On start up it generates a .kube/config file needed to communicate with the API server exposed on port 6443.

How should I make that generated configuration file accessible on the host once the service has been started?

[wispy tapir]

Hmm that's an interesting question, I think @rigid pebble or @heavy gazelle may have already solve that problem

Here's an idea that I'm thinking about (never tried though)
I think what you can do is create a volume, mount it in the container that will run your service -> start your service in a thread, then mount the volume that should now have the .kube/config in a container or somewhere and export that file.
Note that to keep your service running, you must not exit the current dagger call so I would just return the running service in the function then

[wispy tapir]

If you can make your withExec fails, you could open a terminal, otherwise I'm not sure there's such feature yet

[torpid maple]

Fair enough, so just use a bind mount and have the container write the file there.

[torpid maple]

Hmm copying a file to the mounted directory doesn't appear to work. :/

Inside the interactive terminal the copied file is there, with files that were mounted from the host.

However, on the host the copied file is not there.

    Container.withMountedDirectory(
    │ │ path: "/root/.kube"
    │ │ source: Host.directory(exclude: [], path: "/Users/jshbrntt/test/.kube"): Directory!
    │ ): Container!
    Container.withFile(
    │ │ path: "/tmp/kubeconfig"
    │ │ source: Container.file(path: "k3s.yaml"): File!
    │ ): Container!

Container (terminal

dagger ~ $ ls -lah /tmp/kubeconfig 
-rw------- 2 root root 2.9K Mar 19 14:53 /tmp/kubeconfig
dagger ~ $ cp /tmp/kubeconfig /root/.kube/config
dagger ~ $ ls -lah /root/.kube
total 4.0K
drwxr-xr-x 1 root root   12 Mar 19 14:54 .
drwxr-xr-x 1 root root   10 Mar 19 14:53 ..
-rw-r--r-- 1 root root    0 Mar 19 14:41 .gitkeep
-rw------- 1 root root 2.9K Mar 19 14:57 config

Host

$ ls -lah .kube 
total 0
drwxr-xr-x   3 jshbrntt  staff    96B Mar 19 14:41 .
drwxr-xr-x  21 jshbrntt  staff   672B Mar 19 14:41 ..
-rw-r--r--   1 jshbrntt  staff     0B Mar 19 14:41 .gitkeep

[wispy tapir]

Yeah you might need to copy the file into a regular directory, mounted volumes are not exported

[torpid maple]

Like a directory passed in as an argument to the function?

[winter linden]

@torpid maple generally dagger services are immutable black boxes. If you want to generate config file and use it outside of the service, you should not generate it within the service process. You should generate it in a separate function, then 1) pass it into the service pre-start, 2) do anything else you want with it

[torpid maple]

So if I want to start a k3s cluster as a service under Dagger and then connect to it from my host, I should be expected to generate the kubeconfig outside (on my host?) and pass it into the k3s server start up? Then I can use it to connect to the API server exposed by the Dagger service?

https://docs.k3s.io/cluster-access#accessing-the-cluster-from-outside-with-kubectl

[raven stag]

@sharp marsh k3s module is worth looking at if you're trying to run k3s in dagger 🙂
He is using a CacheVolume to be able to retrieve the kube config once the service is up && running IIRC

[torpid maple]

I'm using that module, I'm trying to start the service created by that module and expose the API server to the host. Then connect to the cluster on that exposed port with it's kubeconfig.

The issue I'm having is getting the generated kubeconfig onto my host system.

Maybe I need to do something like this, as part of the service up?
https://docs.dagger.io/cookbook#export-a-directory-or-file-to-the-host

[raven stag]

I used it a while ago, but IIRC, you need to get the service up, and then call the

dagger -m github.com/marcosnils/daggerverse/k3s@833ec36632b2457862f6e3bf1f7107ad65e3e515 call --name my-cluster-name config --local -o /path/to/kube.cfg`

[torpid maple]

I think it's closer to...

$ dagger -m github.com/marcosnils/daggerverse/k3s call k3s --name cluster-name config --local export --path=/path/to/kube.cfg

Haven't got it working yet though

[stoic knot]

One of our developers saw their dagger engine container in a restart loop with the error: buildkitd: could not lock /var/lib/dagger/buildkitd.lock, another instance running?. They ended up restarting their VM and it went away. Any idea on the conditions that may lead to this? We do have a nightly job that runs ansible that can change the engine if we have updated the version in source control

[winter linden]

Looks like your provisioning system is the main suspect - my guess is that the error message is correct, the provisioning system (ansible?) sometimes runs 2 engines on the same state directory, which the engine correctly detects

[stoic knot]

We shut it down before we spin up a new one, wonder if there is some condition leaving the lock file around

We only saw it on one of many

Like if a build was going on and the engine was forced to quit instead of being graceful?

[winter plank]

Is there a way to call --ipc=host from the dagger API? it's recommended by the playwright docs, https://playwright.dev/docs/docker

[winter plank]

How do you call a dagger function which has a constructor? the constructor args don't show up as function args

[winter plank]

Am i doing something wrong here?? I'm trying to copy node_modules from one container to another but it keeps failing,

@func()
  async testEnv(
    @argument(
      { ignore: ["**/node_modules"]}) source: Directory,
      npmrc: Secret, 
      testUrl: string
    ): Promise<Container> {
    const dependencies = await this.buildEnv(source, npmrc)
    // .directory("node_modules")
    // .terminal()
    .directory("node_modules")
    .sync();

    return dag
      .container()
      .from("mcr.microsoft.com/playwright:v1.51.1-noble")      
      .withEnvVariable("PW_TEST_HTML_REPORT_OPEN", "never")
      .withEnvVariable("TEST_URL", testUrl)
      .withDirectory("/work", source)
      .withDirectory("/work/node_modules", dependencies)
      .withWorkdir("/work")
    // //   .withExec(["npx", "playwright", "install"])
  }

  @func()
  buildEnv(
    source: Directory,
    npmrc: Secret
  ): Container {
    const container = dag
      .node()
      .withVersion({
        version: "22",
        isAlpine: false
      })
      .withCache(dag.cacheVolume("node-22"), "/root/.npm")
      .withNpm()
      .withNpmrcTokenFile(npmrc)
      .withSource(source)
      .install()
      .container();

    return container;
  }

[winter plank]

Error: /opt/app/node_modules: cannot retrieve path from cache: {"response":{"data":null,"errors":
"message":"/opt/app/node_modules: cannot retrieve path from cache","path":["container","from","wi
MountedCache","withMountedCache","withMountedSecret","withMountedDirectory","withMountedCache","w
hWorkdir","withExec","directory"]}],"status":200,"headers":{}},"request":{"query":"\n      { node
{ withVersion (version: \"22\",isAlpine: false) { withCache (cache: \"ChV4eGgzOmRlODE2NzI1Mjg4YmE
jMSlQEKFXh4aDM6ZGU4MTY3MjUyODhiYTcyMxJ8Eg8KC0NhY2hlVm9sdW1lGAEaC2NhY2hlVm9sdW1lIhAKA2tleRIJOgdub2
LTIyIjMKCW5hbWVzcGFjZRImOiRtb2QoY2N3ZmNjcmVkaXRjYXJkcy53ZWxsc2ZhcmdvLmNvbSlKFXh4aDM6ZGU4MTY3MjUyO
iYTcyMw==\",path: \"/root/.npm\") { withNpm  { withNpmrcTokenFile (npmrcFile: \"ChV4eGgzOjQ2YzRlM
kYzdlNDE1ZDAS2gEKFXh4aDM6NDZjNGUzMWRjN2U0MTVkMBLAARIKCgZTZWNyZXQYARoSbG9hZFNlY3JldEZyb21OYW1lImwK
FjY2Vzc29yEmA6XjY2Njk2YzY1M2EyZjJmN2UyZjJlNmU3MDZkNzI2M2I2MTM2NzlhMDgxNGQ5ZWM3NzJmOTVkNzc4YzM1ZmM
mYxNjk3YzQ5MzcxNTY1M2M2YzcxMjE0NDI5MmM1YWQiGQoEbmFtZRIROg9maWxlOi8vfi8ubnBtcmNKFXh4aDM6NDZjNGUzMW
N2U0MTVkMA==\") { withSource (src: \"CkdzaGEyNTY6Y2U2YWEyNzE0OTMwNWU0NDE1Y2FiODQ1ZWRjN2FjOGI5MWQw
M4OTQ2NDQyZTczZTUwNGQ0ZTcyNWZiMjhhZRKxAgpHc2hhMjU2OmNlNmFhMjcxNDkzMDVlNDQxNWNhYjg0NWVkYzdhYzhiOTF
GNjODk0NjQ0MmU3M2U1MDRkNGU3MjVmYjI4YWUS5QEKFXh4aDM6ZmY3M2M4NTIxNGI4Zjc1YxINCglEaXJlY3RvcnkYARoJZG
ZWN0b3J5IiAKB2V4Y2x1ZGUSFUITChE6DyoqL25vZGVfbW9kdWxlcyJHCgRwYXRoEj86PS9Vc2Vycy9hbGF1bmkvQ29kZS93Z
vd2VsbHMtZmFyZ28vY3JlZGl0Y2FyZHMud2VsbHNmYXJnby5jb21KR3NoYTI1NjpjZTZhYTI3MTQ5MzA1ZTQ0MTVjYWI4NDVl
M3YWM4YjkxZDBjYzg5NDY0NDJlNzNlNTA0ZDRlNzI1ZmIyOGFlEkAKFXh4aDM6ZmY3M2M4NTIxNGI4Zjc1YxInEggKBEhvc3Q
RoEaG9zdEoVeHhoMzpmZjczYzg1MjE0YjhmNzVj\") { install  { container  { directory (path: \"node_modu
s\") { sync  }}}}}}}}} }\n    "}}
Error: /opt/app/node_modules: cannot retrieve path from cache

[warm temple]

it might be because it's copying from a cache volume, but I'm not sure. I can try to repro

[winter plank]

fwiw it was failing before i added any caching, that was me just trying to make it work

dagger is being fussy for me tonight

[sharp marsh]

Seems like you're trying to return a directory from the cache? That's not currently possible without copying the directory to the container first

[winter plank]

Am I approaching this the wrong way? I'm just trying to avoid having to install node_modules over and over across different tasks

[sharp marsh]

that's fine you can return a container from testEnv instead of a directory

What do you do with testEnv's output?

[winter plank]

buildEnv feeds testEnv to run some playwright tests, and also feeds build to build the projet

[sharp marsh]

yes, I can't see the build function

the "issue" is that you can't return node_modules from the testEnv function

because cache volume entries are not effectively objects in the DAG

the right approach should be to return a dagger.Container from the testEnv funciton

and use that in build

I'm not entirely sure what testEnv is used for though.. I'd need more context there

[winter plank]

build doesn't have a dependency on testEnv -- testEnv and build have a dependency on buildEnv -- but testEnv needs its own container base. I'm really just trying to avoid running npm install inside of that container

[sharp marsh]

yes, that's perfect. so, IIUC you want to reuse the same container base for build and testEnv correct?

and that container base will be built by buildEnv?

[winter plank]

no -- build uses the node daggerverse module and testEnv uses mcr.microsoft.com/playwright:v1.51.1-noble

i'm trying to run npm install once in build and then copy node_modules out of. build into testEnv

[sharp marsh]

oh, I see

in that case you don't need to pass the node_volumes around

you can just bind the same cache volume in both containers and that's pretty much it

you can define your cache volume at your module (class) level and then bind it to both the testEnv and build function in whatever path you need

[winter plank]

ok, so i'd still run npm install from my testEnv, but it would share a cache so install should be ~instant?

[sharp marsh]

yes, exactly. The cache will be shared by both functions

and it'll be instant

[winter plank]

but i'll need to call like, this.buildEnv().sync(), right? -- to make sure the side effect of initializing the cache actually happens

[sharp marsh]

the cache volume is basically a shared volume across different steps in your pipeline

[winter plank]

since there's not an explicit dependency between steps

[sharp marsh]

yes, correct

you can also call install in your testEnv function as well

if the cache volume is empty, it'll populate it

if it's populated, it'll be super fast since all the deps will be there

[winter plank]

ah nice, ok

[sharp marsh]

it's basically the same thing as doing npm install twice in your local project

edit: forgot to start a 🧵

[true trench]

Hi! Really excited by what you're building (although initially was more interested in the CI/CD stuff, hope that won't be discontinued)

I'm trying to wrap my head over what Dagger really offers for LLMs, is it just a kind of "MCP server" of sorts, where you define tools that the LLM can use to solve tasks? just that they run in containers?

Also I'd be interested in having a kind of open-source "web UI" for the dagger shell, so that I could share it on my intranet, and people could just ask tasks to the AI, which would use dagger containers for isolation and to solve tasks (have read-access to our database, etc), and then output a bunch of artefacts (eg file urls of resulting report on S3, JSON description of a visualization graph, etc), that the "notebook / UI" can natively display, and the user can interact with? is that something that's planned/exists already? or not in the scope of dagger?

[true trench]

(I see that dagger is well designed, but I don't see how it radically differs from any MCP server? it's basically: defining an API, and letting the LLM access it and iterate. Just "cleaner" because of the container isolation and the easy way to share modules? Or is there more to it?)

[winter plank]

@pseudo stream / @narrow nymph i'm seeing the same thing ya'll are reporting in https://github.com/dagger/dagger/issues/9557

[sharp marsh]

@winter plank by any chance have you gotten a consistent repro?

[hoary sierra]

Good conversation on CI over at Hacker News, some commentary about how Dagger is funded which I think is off base but good insights on using code instead of YAML
https://news.ycombinator.com/item?id=43419701

qianli_cs

The Pain That Is GitHub Actions

Hacker News · Mar 20, 2025, 3:37 AM

[honest oasis]

as replied in https://discord.com/channels/707636530424053791/1350228561998516285, this is what the Dagger+Runme combo solve for and it's entirely self-contained and open source. You can even render Mermaid diagrams or CSVs in tables. Happy to show you more if you have questions.

[winter linden]

Hello! Don't worry it's the exact same dagger engine powering CI and Agents, and often a combination of the two 🙂 We will continue to support both.

Dagger is more than a MCP server (although it will soon be one, we're merging native mcp support soon). It allows for sandboxing and composition of tools, llms and the agent code between them. It's not just the tools that need sandboxing: so does the agent. Dagger provides a standard runtime and composition system for both.

[true trench]

okay I see what you mean, however assuming the agent is properly coded and can only effect actions through the MCP server, what's the real benefit of sandboxing the agent itself?

[true trench]

this looks promising! I'd really like an UI very close to https://duckbook.ai/ but allowing me to use my own tools / dagger-powered containers etc, perhaps that can be an inspiration?

[winter linden]

as long as you have a simple static architecture with one agent and one mcp server, and you don't need composition, or multi-agent, or the ability to switch quickly between calling a tool from the llm or from the code; and you don't need end-to-end testing of your agent - then you're right you might not need dagger :-p

[true trench]

ah I see, the AI-composition you're speaking of would be more suited to, eg a SaaS offering AI+tools as a service, everything dynamically binded at runtime?

Indeed in my case it's more a usecase of "we have a business with an API + database", and we need simple safe and robust tooling with an AI to do queries and some limited business processes on it

but I have a general idea of what your AI-containerization could be offering now, thanks! Anyway, on my side I was going to use the "dagger for CI" stuff, so might as well use it to provide some containerized tools for our "LLM dashboard"

[torpid maple]

If I get a no space left on device error from k3s running under dagger-engine, running on a Docker host.

However, the docker host that dagger-engine is running on still has disk space, and so does my host system, then where is all the disk space?

$ docker ps -a    
CONTAINER ID   IMAGE                               COMMAND                  CREATED       STATUS       PORTS     NAMES
fd7078061d97   registry.dagger.io/engine:v0.16.3   "dagger-entrypoint.sh"   4 hours ago   Up 4 hours             dagger-engine-custom

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         485.2MB   -2.94e+08B (-60%)
Containers      1         1         507.9kB   0B (0%)
Local Volumes   4         1         26.75GB   4.357GB (16%)
Build Cache     0         0         0B        0B

$ docker exec dagger-engine-custom df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 282.7G      7.8G    274.9G   3% /
tmpfs                    64.0M         0     64.0M   0% /dev
shm                       3.9G         0      3.9G   0% /dev/shm
/dev/vdb1               282.7G      7.8G    274.9G   3% /etc/resolv.conf
/dev/vdb1               282.7G      7.8G    274.9G   3% /etc/hostname
/dev/vdb1               282.7G      7.8G    274.9G   3% /etc/hosts
/dev/vdb1               282.7G      7.8G    274.9G   3% /var/lib/dagger
mac                     460.4G    171.0G    289.4G  37% /usr/local/share/ca-certificates
/dev/vdb1               282.7G      7.8G    274.9G   3% /etc/dnsmasq-resolv.conf
overlay                 282.7G      7.8G    274.9G   3% /etc/resolv.conf
overlay                 282.7G      7.8G    274.9G   3% /var/lib/dagger/worker/cachemounts/buildkit410436133
overlay                 282.7G      7.8G    274.9G   3% /tmp/buildkit-mount498599036
overlay                 282.7G      7.8G    274.9G   3% /var/lib/dagger/worker/cachemounts/buildkit3254702669
overlay                 282.7G      7.8G    274.9G   3% /tmp/buildkit-mount1285043353/file
overlay                 282.7G      7.8G    274.9G   3% /var/lib/dagger/worker/cachemounts/buildkit1742911645
overlay                 282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933
overlay                 282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933/etc/resolv.conf
/dev/vdb1               282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933/etc/hosts
/dev/vdb1               282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933/etc/rancher/k3s
overlay                 282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933/etc/ssl/certs/custom-ca.crt
overlay                 282.7G      7.8G    274.9G   3% /tmp/rootfs1205012933/.init

[sharp marsh]

@torpid maple how does running kubectl debug node and checking the space there look?

[torpid maple]

I'm using OrbStack which apparently according to this.

A fresh install of OrbStack uses less than 10 MB of disk space. Usage increases and decreases as needed thanks to fully dynamic disk management, with little overhead.

So is OrbStack failing to do "fully dynamic disk management"

[sharp marsh]

👍

[torpid maple]

Is it though???

[mystic citrus]

hey Daggernauts, I need some upstream convincing for Dagger.
What CNCF do you know that use Dagger?

[winter plank]

Can you interact with the output of a function call after it completes in the shell?

[sharp marsh]

CNCF project you mean @mystic citrus ?

Can you interact with the output of a

[stoic knot]

Is this something that could happen if

1. We are running a custom engine (config, manual start in docker)
2. Someone mistakenly updates the SDK to be a patch version newer than the server? (and it wants to start a new engine?)
   Saw a second instance today, but also found that someone updated our go.mod to a newer version... 🎣

[lime snow]

Since lists are comma delimited, Is it possible to pass a list of strings that already contain commas?

i.e. this list of strings ["Hello, World!", "Howdy, Planet!"]

Specifically, is it possible to pass it as part of a command such as dagger call foo --arg=<insert-list-here>

[cunning jolt]

not possible today, there's an issue to track it - https://github.com/dagger/dagger/issues/9426

[lime snow]

Thank you!

[surreal geyser]

Anyone getting "No such image: registry.dagger.io/engine:v0.17.0" after updating (mac)?

[neon warren]

I have not seen that, I upgraded pretty soon after the tag. Can you try dagger core version?

[surreal geyser]

✔ connect 0.2s
✔ loading type definitions 0.2s
✔ parsing command line arguments 0.0s

✔ version: String! 0.0s

v0.17.0

[neon warren]

I just pulled, can you verify this works? docker pull registry.dagger.io/engine:v0.17.0

maybe something is blocking your network access? VPN, Firewall?

[surreal geyser]

I should have tried just pulling it via docker, duh. that worked.. Weird... Thanks!

[neon warren]

odd that it did not pull automatically, maybe a network hiccup?

[surreal geyser]

Probably what happened. I hit update and walked away and came back to an asleep computer.

[modern mountain]

I'm feeling pretty dumb trying to do any sort of git operations with dagger; can someone point me to a public example of cloning a repo (or using a Directory object that is a clone of a repo), changing or adding a file, committing it, and pushing it up? I tried installing the shykes git module but it appears to not work

[torpid maple]

@sharp marsh giving 0.1.8 of the k3s module a go but getting this on startup.

Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory

[slender charm]

Hey!

Did you already get a chance to see this example in the cookbook? https://docs.dagger.io/cookbook#clone-a-remote-git-repository-into-a-container-by-branch-tag-or-commit

Can you give us one more piece of info about your use case.

Making commits and pushing as a part of a CI pipeline is not the most common thing we see, and you will have to do some key wrangling, so want to make sure I understand the core problem 🙂

[cunning jolt]

To add to this, reading from a git repo is easy, but to actually commit and push (or fetch more) you have to make sure your credentials are available within the container you are using the cloned repo from. It's not automatically handled by dagger today although some of the plumbing is already in place for that. Here's an issue that tracks this partially - https://github.com/dagger/dagger/issues/9858. There is an example there you could use if you are using PAT auth.

[quiet tiger]

What is going on?
│ ✔ Container.file(path: "/src/api/apikey/middleware.go"): File! 0.0s
│ ✘ .export(path: "."): String! 0.0s
│ ! unexpected closing recv msg: rpc error: code = Unknown desc = destination "/scratch" is a directory; must be a file path unless allowParentDirPath is set
│ │ ✘ export file /api/apikey/middleware.go to host . 0.0s
│ │ ! unexpected closing recv msg: rpc error: code = Unknown desc = destination "/scratch" is a directory; must be a file path unless allowParentDirPath is set

What is /scratch even?

[warm temple]

/scratch might be the workdir in the runtime container, I'm not sure. Do you have Container.file().export(".") in code that you want to export to your host machine, or is it a function returning Container.file() that you've chained with export?
To solve the path thing, maybe try "./" or "./middleware.go" as a workaround

[quiet tiger]

a, err := container.File(filepath.Join("/src", fp)).Export(ctx, fp)

a, err := container.File(filepath.Join("/src", fp)).Export(ctx, "./"+fp)

Produces ...

┃ -- api/apikey/middleware.go
┃ --!! /scratch/api/apikey/middleware.go
│ ✔ Golang.baseImage: Container! 0.5s
│ ✔ .withExec(args: ["go", "fmt", "./..."]): Container! 0.1s
│ ┃ api/apikey/middleware.go
│ ✔ .stdout: String! 1.2s

│ ✔ Container.file(path: "/src/api/apikey/middleware.go"): File! 0.0s
│ ✔ .export(path: "./api/apikey/middleware.go"): String! 0.0s
│ │ ✔ export file /api/apikey/middleware.go to host ./api/apikey/middleware.go 0.0s

(the -- stuff is my debug output)

and the file isn't actually exported to the host

[warm temple]

check out this for exporting https://docs.dagger.io/cookbook/#export-a-directory-or-file-to-the-host
The code in your snippet will export, but it's exported to the sandbox since the function doesn't have direct access to your host. Instead, the function should return a file or directory

[quiet tiger]

oh right. FWIW: I really dislike this about dagger. The docs at least shouldn't say "Writes the file to a path on the host"

ended up with dagger -v call <stuff> go <other stuff> fmt image directory --path=/src export --path=.

[winter linden]

@quiet tiger sorry. We've discussed renaming "host" to something less confusing, like "client". In our defense we called it "host" in the API before sandboxed Dagger Functions were a thing. There is a detailed thread about this somewhere.

[quiet tiger]

fwiw: I'd very much like the export SDK call to actually write data to the host host, even if it has to do some magic to talk back to the dagger cli to make it happen. It would solve a lot of things I seem to keep having to work around.
And while I realize there are security implications, that is true any time you run code that can change your filesystem.

/rant

[latent fable]

I've been puzzling a bit over container platforms and am curious what Dagger's approach is given the "write once, run anywhere" kind of philosophy. As I understand it, if you have User A and User B running arm64 and amd64 machines respectively, then by default all of dagger's activities will be native to the host platform, meaning users A and B are producing different results that are not cache-compatible. Depending on what you're doing, you could even have builds that fail in one environment and not the other. Is this considered acceptable up until the pipeline disambiguates by explicitly declaring a platform instead of defaulting to the host platform?

For a bit, I was actually assuming Dagger would default everything to amd64 for the sake of reproducibility, but I understand that would just be a lot of overhead for everyone on a different platform.

I'm experimenting currently with building a raspberry pi SD card image using Nix tooling, and figuring out how to daggerize that process. The nix build needs to either be running on arm64, or have access to buildfmts and qemu in its runtime (e.g. a debian amd64 container in privileged mode with the necessary configuration). I don't yet have an intuitive sense of whether it's faster/more efficient to do an architecture emulation "closer to the host" or "closer to the guest" -- for example, to run nix in a debian arm64 container and force dagger/buildkit to employ qemu if necessary for the host, or to run nix in an amd64 container and give that container the tools to emulate with qemu. What's counterintuitive about this is that running the former case results in more instructions being emulated, though the emulation environment is running closer to the metal and thus should have better performance, whereas in the latter case only the minimum number of instructions are going through emulation; though the emulation environment is wrapped in an extra layer of abstraction and perhaps subject to more overhead.

The former case also has the benefit that the dagger engine is responsible for figuring out if emulation is required at all (since it may not be on a given host), whereas the latter would result in two layers of emulation on an arm64 host unless there is a way to code around this.

[latent fable]

Or another idea -- if one is running dagger at large scale, conceivably you could have multiple hosts, with different architectures available. Could/does dagger facilitate distributing work out to hosts based on host architecture to optimize against requiring qemu?

[sharp marsh]

Dagger, in the same way as buildkit and Docker provides out of the box buildfmt + qemu native integration. By default, if you hardcode a pipeline to use a specific image from one platform, it'll fallback to emulation and try to run it using that.

[sharp marsh]

yes, this is also possible. However, dagger doesn't currently handle the automatic engine selection for you. You'll have to prepend which dagger remote engine to use while calling your dagger pipeline to select the appropriate engine cpu architecture in each case.

I don't yet have an intuitive sense of whether it's faster/more efficient to do an architecture emulation "closer to the host" or "closer to the guest" -- for example, to run nix in a debian arm64 container and force dagger/buildkit to employ qemu if necessary for the host, or to run nix in an amd64 container and give that container the tools to emulate with qemu

If you have the ability to run the build process in the target platform cpu architecture, why would you still need to emulate some things from the guest machine? Not sure I follow that. Can't you run everything in the target architecture?

[latent fable]

Well, that was a hypothetical. In practice what I'm trying to solve right now is only one architecture available for local development and only the other available for CI, so one of the two will have to use emulation

Not "solve" exactly, since I have it working, but just curious about what's most elegant and how Dagger thinks about defaulting to the native arch

[sharp marsh]

or not. i.e: Your local machine could be running in amd64 and CI in arm64. You can still run the same pipeline both locally and in CI as long as your pipeline does not require quirky some platform specific steps

if you mostly need to pull an image, install dependencies and then run some make / deboostrap targets, the Dagger pipeline code should mostly be the same regardless the architecture

[latent fable]

But isn't (e.g.) generating a container image inherently platform specific? If I do container | from debian in Dagger on amd64 and save it, that'll be a different image from the same pipeline run on arm64.

[sharp marsh]

yes, but the pipeline definition will still be the same. I'm mostly refering to the pipeline code not changing, not the DAG shape and its outputs

[latent fable]

Right, but the outputs will have an impact on cache retrieval -- say assuming a distributed cache between engines

I guess at the end of the day if you're building images to deploy to dockerhub or a host somewhere, you do have to be explicit about the platform you're building the image for

[sharp marsh]

yes, that's correct but.. do you really care to optimize about distributed caching between local and CI? I mean.. CI will have its cache and local will also have its cache. If you run multiple builds per day / week / whatever, then sharing the cache between local and CI doesn't become that relevant

[sharp marsh]

yes, also this

[latent fable]

In practice quite likely not, but for whatever reason I tend to be highly motivated to understand my tools at this level of depth 🙂 Nice to know I've got an accurate mental model; thanks for chatting!

[sharp marsh]

Nice awesome! thx for the weekend chat as well 🙌. Would love to know what's specifically bringing you to dagger in an alternative to your nix pipeline. If you have ideas or suggestions, we're happy to help you out!

I've personally been an enthusiast from the Docker + rpi world a few years ago as a hypriot contributor (https://blog.hypriot.com/) so it makes me happy to see someone thinking about Dagger for this since it feels like a natural evolution from the Dockerfile madness

[latent fable]

I've actually got about the same level of experience with both nix and dagger -- slightly more dagger I think. I strongly value both portability and testing reproducibility/rigor, and I think Dagger's exactly right for CI, but nix is incredibly powerful for what it offers if you can eat the tradeoffs. You might be interested in this talk from Matthew Croughan at SCaLE 2023 -- it's both very long and unstructured, but as a docker/dagger person I think it can be eye-opening as to what nix's capabilities are: https://www.youtube.com/watch?v=6Le0IbPRzOE In particular around 40:00 he starts demoing a build of a raspberry pi SD card OS image, and I'm psyched about how easy that capability is to use, so I wanted to figure out how to shoe-horn it into a Dagger pipeline so I can run it on commodity CI instead of needing a nix-specific platform

[latent fable]

Hadn't heard of Hypriot before -- I started using Docker years ago for application deployment, and did a little bit of homelab docker-pi experimentation using Balena -- I'm sure they benefitted from hypriot's contributions to the ecosystem

For the most part though I tend to use Pis as appliances, e.g. flash one image, deploy it, and it manages itself -- so the nixOS pi image build is very compelling

[sharp marsh]

yeah for sure. We were the ones to send the patches to the upstream rpi kernels and docker so it could work in the first rpi's... a whole ecosystem exploded after that... so yes... good old memories

[sharp marsh]

It'd be awesome to see a Dagger module which can produce a rpi ISO. I mean.. my Nix knowledge is quite limited but I can't fundamentally think of anything extraordinary which Nix allows you to do which Dagger can't. After all, they're both build systems which run in "sandboxed" environments.

I guess the edge tht Dagger might have over Nix for this specific use-case is its UX and programmatic model as well as the fact that you're able to boot your built ISO image into dagger once it's built which makes it super cool 😛

[latent fable]

It's really the nature of that sandboxing that's the source of nix's power; it's kind of more of a package manager than anything else. The sheer volume of nixpkgs that already exist are an incredible resource. I was recently tearing my hair out trying to get an old version of PHP running on macOS which had been rendered uninstallable due to a non-backwards-compatible libxml2 update, and since nix's sandbox gives you hard guarantees about every library and dependency in use by the software, it was almost trivial to get this old PHP installed and working. But then it extends that concept to an entire operating system. If you were going to build an OS image with Dagger I think you'd need a lot of other software outside the scope of containers, which by nature would expand your dependency graph and increase surface area for breakage and irreproducibility. It seems to me like Nix already offers a great solution for that, even more since its build can just run inside a dagger container -- feels like a best of both worlds situation.

[modern mountain]

Yes, I've done that portion of the process. I find it odd that committing and pushing isn't a part of most CI, because otherwise how are people automatically doing version bumps and then doing CD? I mean, I don't like this pattern but I'm unsure of any other way to do it. I'd be interested to hear how other people manage this problem - basically, instead of asking the user for the new version, how do you generate it and commit it to git automatically so your deployments can pick it up?

[warm temple]

I think in general, automatically generating additional commits in CI is a bit of an antipattern. Are these commits updating some configuration or something that gets picked up by CD?

[modern mountain]

Yes, that's exactly it. When a PR is merged, a github action will trigger a build; the build will increment the verison using the poetry version function, build, and commit the new version to the tanka config which will then be used to deploy the new image automatically to our dev cluster. And I do agree that this all feels like an anti-pattern, and is one of the reasons why in general I hate CI systems, so I'd love to figure out another approach if possible.

[latent fable]

Presumably you're doing version bumps because of semver; otherwise you could arguably say your version number is the timestamp of the build plus the commit hash; then you don't have to commit anything back. For semver my impulse is to almost invert the process -- rather than have CI increment the version number, use CI to check that the version number has already been incremented since the last release as evidenced by a tag. In theory (???) one could even test that the semver is valid based on changes in API schema

[modern mountain]

Well, I would still want to commit something back to my tanka config and deployed so that the latest deployed version is in git

[warm temple]

Does that really need to be recorded in git though? You already know a CI run happened. It sounds like a lot of commits are getting added with no value

[brazen compass]

Hi, I have to run a tests for which multiple containers must be on the same network with no filtering between them, is that possible with dagger ? I have found WithExposedPort but opening individual ports is not sufficient for my use case.

[sharp marsh]

yes, you can use Dagger services for that (https://docs.dagger.io/api/services/). The TL;DR is that you can bind multiple services together via the WithServiceBinding or WithHostname calls and they'll be able to talk to each other within the same network

[brazen compass]

Thanks, I've been able to use this to have a container connect to a service but I haven't been able to make multiple services connect to each other

[modern mountain]

Hm - I was planning on doing this for dev, stage, and prod but really the only true need for this is in dev which might have multiple builds per day happening. Stage and prod happen after a human has actually chosen to cut a release...

The tension is between 'what is deployed is the source of truth' vs 'the latest commit in git should contain the most recent and currently running deployment'

[rustic brook]

If concerned about having a "central buildkit cache" for local+ci and multi-arch, I'm betting there's a way to pull that off by configuring BuildKit, which has "cache storage backends" including a plain-old-registry. https://docs.docker.com/build/cache/backends/registry/
@sharp marsh does Dagger remote caching work yet (chatGPT thinks it does lol)
If you're focused on the fastest builds possible from anywhere, I've used depot.dev which is remote highly-tuned and performant arm64 and amd64 BuildKit builders (and also GitHub Action Runners) that are meant to centralize the BuildKit cache for a team of people and CI jobs. Dagger has a Depot module https://daggerverse.dev/mod/github.com/depot/daggerverse/depot
The devils in the details though... if you have a bunch of quick build steps but a large project (context) to transfer over the wire (and your connection isn't awesome) then remote builds and shared cache might make things slower due to transfers taking longer than builds.
Not Dagger specific, but I did a bunch of testing of remote shared caches on Dagger last year, and in many use cases it wasn't faster than my local builds, even when employing QEMU locally.... it just depends on all the factors and testing was required to see if Depot would improve local build speeds. I made this with my learnings https://www.youtube.com/shorts/0tuogSKA4f0

[brazen compass]

So I had another attempt at this today and found my issue which was unrelated to dagger. I have to run systemd in these service containers and it turns out systemd thinks it's a good idea to change the MAC addr of the containers veth when it starts. Except it set the same MAC for all the containers running systemd so they couldn't talk to each other 🤦‍♂️ Works fine now that I've masked systemd-udevd in these containers...

[winter plank]

Is anyone else having weird hangs? it's just been doing this for >6 mins (0.17.1)

[winter plank]

this may just be my hotel wifi 🫠

[torpid maple]

I just updated to 0.17.1 and now I'm getting this error.

✘ asModule getModDef 1.0s
/src/sdk/src/telemetry/init.ts:1
import { getBooleanFromEnv } from "@opentelemetry/core"
         ^

SyntaxError: The requested module '@opentelemetry/core' does not provide an export named 'getBooleanFromEnv'
    at ModuleJob._instantiate (node:internal/modules/esm/module_job:177:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:260:5)
    at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:543:26)
    at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:116:5)

Edit: deleting node_modules,sdk directories fixed it.

[maiden cloud]

General question

When we do

dag.Container().From(<image_X>:latest)

(supposing the image is periodically overwritten on the latest tag)

• is there a way to force dagger to pull "always"
• is there any guarantee Dagger will look for new versions of the same image/tag (e.g. comparing local with remote digest)

[winter linden]

yes, dagger will look for the latest version of the tag

[maiden cloud]

ahhh. I see the digest in dagger shell

[celest crane]

👋🏻 If you happen to be at JavaLand 2025 at the Nürburgring next week: Drop by for a chat! I will to an introductory talk to dagger. Very happy that the Java-SDK got traction recently so i can update my examples. https://meine.doag.org/events/javaland/2025/agenda/#agendaId.5310

[winter linden]

@here We just officially launched Dagger Shell 🙂 https://dagger.io/blog/dagger-shell

• https://x.com/solomonstre/status/1904978651629892016
• https://x.com/dagger_io/status/1904980305532108858

[chilly arch]

Thanks for sharing! Best of luck with your talk!

Don’t miss the Dagger Community Call tomorrow at 9 AM PT!

We’ll explore Hybrid Cloud AI workflows in a GitHub pipeline and discover how to build an Autonomous QA Agent for the Daggerverse with @thick igloo @tacit hull and @slender charm

Watch live on YouTube or X, and join the conversation!
https://www.youtube.com/watch?v=uOSmyFx7O7Q

[winter plank]

I must be doing something dumb, but i'm struggling to put together a local module. i keep getting this error:

✘ ModuleSource.generatedContextDirectory: Directory! 32.3s
! failed to run codegen: failed to load dependencies as modules: failed to load module dependencies: select: failed to call module "workspace" to get functions: process "tsx --no-deprecation --tsconfig
! /src/project/.dagger/workspace/tsconfig.json /src/project/.dagger/workspace/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1
Error: failed to generate code: input: moduleSource.withEngineVersion.generatedContextDirectory failed to run codegen: failed to load dependencies as modules: failed to load module dependencies: select: failed to call module "workspace" to get functions: process "tsx --no-deprecation --tsconfig /src/project/.dagger/workspace/tsconfig.json /src/project/.dagger/workspace/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

[neon warren]

did you happen to change something in the other local module? Try cd'ing into that modules directory and run dagger develop and dagger functions

[winter plank]

cd into the workspace module?

[winter plank]

ah this was helpful! you can't return this

[neon warren]

yeah, this got me the other day as well 🙂

[winter plank]

🫠 how do you call a dagger function that is a method? i see @warm temple 's demo he's defined a struct and the dagger functions are methods, but when i do this (TS) i can't figure out how to pass data to the constructor

[warm temple]

Here's a TS example with a constructor https://github.com/kpenfound/agents/blob/main/tictactoe/tictactoeClient/src/index.ts
and the code calling it's constructor https://github.com/kpenfound/agents/blob/main/tictactoe/src/index.ts#L57
if that helps! What are you trying?

[winter plank]

I really mean how do i call the function at the cli

[winter plank]

when i dagger call i can't pass in the args for the constructor

[warm temple]

Ah thanks, I understand! Constructor args come before the function name, so it's like <module> <constructor args> <function> <function args>. So like dagger call --foo=bar myfunc --boo=baz
here's the doc but I think we can add more explanation here https://docs.dagger.io/api/constructor/

[winter plank]

ahhh! snap! 😿 i'm so used to relying on --help

[winter plank]

🤔 no dice, dagger call --source=. --npmrc=file://~/.npmrc check --extra-args="--grep=landingpage" ... Error: could not find result property source

[warm temple]

ah yeah, if you dagger call --help (without the function) it'll show constructor args, but it might not be obvious to look there

[warm temple]

can you show your constructor?

[winter plank]

private ccwfc: Ccwfc;

  constructor(
    @argument({ ignore: ["**/.git", "**/node_modules", "**/*-snapshots/", "**/dist"] }) source: Directory,
    npmrc: Secret,
  ) {
    this.ccwfc = new Ccwfc(source, npmrc);
  }

[warm temple]

that error sounds like it's not seeing the constructor which is strange. Digging deeper

if you run dagger call --help does it show source and npmrc under Arguments?

[winter plank]

yeah it does! it seems like it's actually starting and then blowing up after initialization? I added a log after this.ccwfc = new Ccwfc(source, npmrc); and it's printing. Is there some special way i need to init a module?

dag.ccwfc?

[warm temple]

What is Ccwfc? A type from another dagger module?

[winter plank]

yes

[warm temple]

ah ok, if it's from another dagger module you'd want this.ccwfc = new dag.Ccwfc(source, npmrc)

[winter plank]

~Backend from the greetings example

[winter plank]

Error: dag.Ccwfc is not a constructor

[warm temple]

sorry it might need to be lowercased 😬

oh wait and drop the new

[winter plank]

ah ok! yes that works (i had that a minute ago) but then it blows up when i call a method on this.ccwfc -- is there a special way i need to call methods too?

[winter plank]

dagger call --source=. --npmrc=file://~/.npmrc check --extra-args="--grep=landingpage" ...

Error logs:

✘ .check(extraArgs: ["--grep=landingpage"]): String! 3.7s
Error: GraphQL Error (Code: 400): {"response":{"status":400,"headers":{}},"request":{"query":"\n      { workspace (checker: \"REMOVED FOR BREVITY\") { id  } }\n    "}}
Error: API Error

[winter plank]

and i can call the module function from the cli successfully

[warm temple]

any chance you have a cloud trace?

[winter plank]

it's just .. blank? https://v3.dagger.cloud/lamalex/traces/9f01693a1971d6145b8e9b4331267c41

[winter plank]

i think i was missing an await, i added await and now get Error: this.ccwfc.e2eTest is not a function

i can call e-2-e-test from the dagger cli

[warm temple]

looks like the typescript sdk converts that to e2ETest, although I don't know if that's right or not

[winter plank]

no longer complaining that it's not a function at least, but still a 400 (and a blank trace)

[winter plank]

the function name not matching what's defined is very unintuitive

[warm temple]

Yeah totally understandable. Each dependency goes through code generation to provide the functions under dag and it follows the language-specific casing. But it's surprising if you used a different casing in your file. And on top of that numbers throw a surprise wrench into it, I would have guessed e2eTest over e2ETest

[winter plank]

any idea about this API error? I can't seem to call any methods on the local module

[winter plank]

but i can call the functions from the cli

[warm temple]

Could be missing an await, but that's not based on much. Can you share one of the function calls in the main module calling the other module?

[winter plank]

  @func()
  async check(
    extraArgs: Array<string> = [],
  ): Promise<string> {
  
    let testResults = "";

    // try {
    testResults = await this.ccwfc.e2ETest(extraArgs);
    // } catch {
    //   testResults = await this.debugTests(extraArgs);
    // }

    return testResults;
  }

[sonic obsidian]

Should apk update run inside registry.dagger.io/engine:v0.17.1 image or something is wrong on my side?
I want to use dagger engine as base image but without apk getting everything inside if it's not a static binary is difficult

it's giving me

/ # apk update
ERROR: Unable to read database state: No such file or directory
ERROR: Failed to open apk database: No such file or directory

[sonic obsidian]

Should apk update run inside registry.

[jagged flicker]

Shell question:
I was wondering if there's an other way to chain results.
Let's say I have a function that returns a string (for instance by calling an LLM) and a function that takes a string in input. How to connect them in a call?
If I func-returning-str | func-taking-str it will not work as func-taking-str is not a member of string.
I can do func-taking-str $(func-returning-str) but I was wondering if we shouldn't have something else.
Like func-returning-str > func-taking-str (or any other char, > might not be the best). That way the output of the first is sent as the input of the second.

[slender charm]

I think this is the current recommended pattern

I can do func-taking-str $(func-returning-str) but I was wondering if we shouldn't have something else.

[sharp marsh]

hello!

[chilly arch]

Community call happening now. @thick igloo and @tacit hull are about to show a super computer connecting to Daggerverse 😉

[thick igloo]

interlink: tool of k8s extension from the presentation https://landscape.cncf.io/?item=runtime--container-runtime--interlink

[chilly arch]

Module they were showing - https://daggerverse.dev/mod/github.com/interTwin-eu/itwinai/ci@3076fa36813e33d6a062f6999874705069c84621

[winter linden]

Today's community call as a Holy Roman Empire icon panel 😛

@chilly arch @warm temple @slender charm @thick igloo @tacit hull

[warm temple]

we have our youtube thumbnail

[winter linden]

I like this one, but it erased Diego 😭

I feel like @slender charm gets older on each iteration 😂

and his headache more pronounced

[slender charm]

Thanks for joining everyone, here's the agent I showed! https://github.com/levlaz/agent-playground/tree/main/daggerverse-qa

[chilly arch]

Thank you @slender charm @thick igloo @tacit hull @warm temple for joining the Community call today! Still absording how cool both of those demos were.

For anyone who missed it, you can check out the recording here: https://www.youtube.com/watch?v=uOSmyFx7O7Q

[thick igloo]

Link to the next Kubecon talk: https://kccnceu2025.sched.com/event/b8b03a1b2eca360598f6e05ea20aecad

[rotund meadow]

hi i was wondering if i can participate in the hacknight remotely

[chilly arch]

hi! Thanks for reaching out. We don't have a remote option for this hack night, but here is what we'll do if you want to participate. Feel free to join us on the Hack Night channel during the event hours, so people can help you if you have questions - https://discord.com/channels/707636530424053791/1352393343878955008

• For CI use cases, we'll be pointing users to Puzzle's tech lab - https://dagger-techlab.puzzle.ch/
• For AI agents, we'll ask everyone to do the Agent Quickstart and then start building their own agents for the rest of the hacking time - https://docs.dagger.io/ai-agents/quickstart

At the end, we'll have time for people to do 5-minute demos. Since you can't present your demo remotely during Hack Night, I'd be happy to host your demo on our Community call! Just DM me, and I can add you to the schedule 🙂

[uncut thicket]

hey can I get some quickstart simple hello world example on dagger ? Like I do sudo dagger and now I am not sure about its interface

this seems so specific on ai that I am not sure ..

[neon warren]

There is a QuickStart for CI and Agents on the docs here https://docs.dagger.io

is there anything specific you need clarification on that we can help with?

[uncut thicket]

I mean , what is this for?

It seems that I have to mess around with programming languages code.

When it wants to replace docker.

don't get me wrong , but I truly can't seem to do anything with it.

I don't want to build an ai agent with dagger right now , something like a simple CI / CD example that just prints "hello world" / I don't know , some way to just check that it works / see what it really is.

I am just soo confused over dagger

[neon warren]

Dagger is great for CI. I’ve built an entire cloud platform with Dagger at its core. Part of the hard thing about Dagger is that it’s really flexible based on your needs.

There are some examples here: https://docs.dagger.io/examples#cicd-examples

also worth checking out https://daggerverse.dev for examples of build modules and what you can do

[neon warren]

Also, anything you can do with a container and a programmable API… dagger is really good at!

[cunning jolt]

I don't want to build an ai agent with dagger right now , something like a simple CI / CD example that just prints "hello world" / I don't know , some way to just check that it works / see what it really is.
There is a "Build a CI Pipeline" quickstart here. Try following that? https://docs.dagger.io/ci/quickstart

[uncut thicket]

yea but it essentially only boils down to https://docs.dagger.io/ci/quickstart/daggerize which requires programming languages code and a whole setup.

I have essentially come from the hackernews post and it says Dagger: A shell for the container age

How is it a shell ? Doesn't shell means for interactive use cases like you don't have to create a whole file / go project / any other language project / you can sort of just do it with simple terminal

[warm temple]

I'd start by playing with some of the commands in the blog post you referenced https://dagger.io/blog/dagger-shell
for example container | from alpine | with-exec apk add git | terminal

To get in the shell, just run dagger!

[cunning jolt]

The Dagger shell is just a tool that dagger provides. You have an app that you want to build a ci/cd pipeline for right? Dagger helps you with that. It's not a "no code" platform. You may be able to get away without writing any code by using a pre-built module that supports it but at the end of the day, you will have to at the very least understand code. Nice thing with dagger is, it can be in a language you are already familiar with. You are not forced to use a specific language or a bespoke DSL.

[inland oyster]

Is there any plans to expand WithTimestamps style statements? I've noticed the path argument doesn't get the same timestamp applied as the files in the directory.
Would also be nice to just 0 timestamp the entire container too but not a huge problem haha

[winter linden]

Random Dagger-themed genAI images, because why not 🧵

[modern mountain]

Wondering if anyone is using dagger in EKS and has successfully configured IRSA (IAM Roles for Service Accounts)? I was assuming my current IRSA setup would work but running the aws CLI from inside my dagger terminal() it doesn't have the environment variables set and access to the data provided to the underlying pod.

[neon warren]

you will probably need to pass the AWS env vars from the system calling the dagger function to the dagger function (as secrets) because the function will also run in a container? Those are usually used/assigned on demand from AWS IAM.

[vivid mango]

Oh my... Since I was last here (v0.14.0), so much has changed...

• nix
• LLM
• dagger shell

Guys, you've been keeping yourselves busy!

The feature set is growing and looks very interesting! Even the remote runners are out in the docs. Nice!!!

[slender charm]

Welcome back! Even more on the way 😄

[vivid mango]

I promise to read all the docs this time, to see if I can understand those features. Because just looking at examples sometimes raises more questions than gives answers 🙂 (e.g. WTH is llm's workspaces... what's the with-mounted-directory do vs. with-directory... etc.)

EDIT: that is, read the docs NOT in the archive 🙂

[slender charm]

Do you have some specific use case in mind? There is so much you can do with dagger and it can be a bit overwhelming to start sometimes. But we have a growing body of useful examples that I'd be happy to point you to depending on your use case 😄

[vivid mango]

well, currently I'm only using it as a CI build engine in BitBucket pipelines, with BB CI runners deployed in my k8s (EKS) cluster. So with every CI run I am downloading, installing and starting dagger engine v0.14.0 anew (which is painfully slow TBH) and running CI code builds, OCI image generation and upload to ECR. My current use case is not that flashy. Maybe you have some suggestions on how to design my dagger CI flows differently, maybe to increase startup times?

There are ~10, maybe more, repositories in the project. One of them has a multi-gig java codebase, compiling and building into several ~2GB OCI images, requiring ~30GB for the build of each. Last I checked, there was no way to bind-mount directories/volumes into dagger runtime (it used to copy everything into the runtime), which ruled dagger unfit for this repository. Maybe you have smth to advise to make dagger a feasible option for it too? (hence the search for with-mounted-directory; I was hoping it would do docker-fashion volume mounts..)

Also, our company is very interested in AI-enabled solutions, so I'd like to explore what dagger has to offer in this department.
I'd also like to see where else I could integrate dagger as our BAU tool.
Also, I'm exploring the nix+direnv approach, since I'm shuffling multiple repos/projects and don't want to trash my host with version conflicts. I see a 'nix' section in this discord channel, but I can't find 'nix' in docs and I don't see dagger in nix's package repo. What's dagger's relation to nix then?

[winter plank]

Is there a "delete all traces" function in dagger cloud?

[elfin frigate]

WithTimestamps

[jagged flicker]

For those who will be at KubeCon next week, you should join the Docker + Dagger event on April 2nd about the new LLM capabilities of Docker Desktop: https://www.eventbrite.com/e/kubecon-london-ai-meetup-join-docker-and-dagger-for-drinks-and-demos-tickets-1303283007559

[modern mountain]

I'm attempting to use the kubernetes functionality where you specify one of the daemonset dagger pods you run in your cluster; while the pod has access to the proper variables, the dagger process running inside doesn't pick them up. But that's me launching the dagger task from my workstation into the cluster; maybe I need to have another pod running the dagger cli to trigger it and pass them in there. But at that point I'm unsure of the benefit of the native dagger kubernetes functionality?

[frozen basin]

I've got a similar heavy build - we ended up moving away from more dynamic dagger runners towards static shared build servers (via _EXPERIMENTAL_DAGGER_RUNNER_HOST env var) - this helped us cache our large clones, and lots of the intermediate work.

[sharp marsh]

no, not currently. You can open a support request if you need that. cc @slender charm

[rotund meadow]

sent

[chilly arch]

Hey Daggernauts! 👋

If you’re attending KubeCon EU or are in the London area next week, we’d love to connect and say hi!

Here’s a quick rundown of where and when you can find us:
https://www.linkedin.com/posts/mirandamyerholtz_london-kubecon-activity-7311469450816286720-2CJJ?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAbjwqUBPZEp__AD2xcvfBAb5pzs_ir11nM

[winter linden]

are there any Earthly users? Since it's based on buildkit I've been wondering if it could be ported as a frontend to Dagger, and get the best of both worlds?

[winter plank]

Does anyone have an opinion or suggestion on the current best solution for production dagger? We’re GHA users, and like has been discussed the caching situation is a little dire. The canonical answer seems to be “use depot”, but even with depot there isn’t caching between GH and local runs. Is there a solution which is broadly regarded as “the current best”?

[winter linden]

You can run Dagger & Github Actions runners on a Kubernetes cluster. With Dagger engines running as a daemonset, the cache of each node will be persisted across CI runs.

From there you can configure storage infrastructure like EBS, Ceph or Regatta to persist storage further

alternatively, you can opt for a beefy long-running machine in a colo. Often a full blown serverless cloud model is overblown for CI (depends on the exact workload). Depending on your load, when factoring in efficiency gains of Dagger caching, a single machine may be all you need.

In this architecture you can scale by sharding: allocate a different machine to each CI workflow. This also has the benefit of maximizing cache reuse

[winter plank]

How does this work with local runs? Do i need to like.. point the dagger cli at the dagger engine in my eks cluster?

[winter linden]

there's very little benefit to a remote cache from your dev machine at the moment. It's likely that the cost of transferring cache data outweighs the benefits.

[winter plank]

Fwiw im not actually thinking about local i was just being lazy.. I’m really thinking about being able to trigger runs from multiple places- in my case gha, and our IDP

[winter linden]

Ah I see. Yes you can configure the CLI to connect to a remote engine.

[nova dome]

Hey there!

Quick question: are there any chances for me to secure a spot tomorrow?

[chilly arch]

hi! Community lead here 👋 I'll DM you!

[sharp marsh]

@gleaming flame yes, there's a way to set registry authenticaton. The right was is to call WithRegistryAuth in the container type 🙏 (https://docs.dagger.io/api/reference/#Container-withRegistryAuth)

[solemn vigil]

Hi @narrow nymph , how did you add the scan to dagger? I am struggeling with issue to scan the containers I have built with the dagger "build" and "publish" command. I want to avoid to download the image again from the container registry. I want to use Snyk to scan the container.

[narrow nymph]

so the scanner we use (trivy) can scan an oci tarball I think? so we just Container.asTarball and then mount that into the scanning trivy container

I don't know snyk, so not sure if it has the same functionality

[warm temple]

Try this out https://daggerverse.dev/mod/github.com/lukemarsden/dagger-snyk

[vivid mango]

I'm curious about dagger shell.

1. no dedicated channel in discord yet? 🙂
2. is it supposed to work with shebang? Can't make it work...
3. so the only way to parametrize a dagger script is by interpolating the script in another shell, e.g. bash? seems like hell of escapes ($). Will there be a way to pass CLI args later on? Is there now?
4. how to show helppages of commands/functions in REPL? container | directory --help fails (yet displays what I asked for... twice.. 🙂 )

[slender charm]

Hey!

Thank for giving it a try!

1. We can add a tag in help for shell, but I think we'll hold off on a making a dedicated channel for now in the interest of keeping things a bit clean

2. Yes it should, can you show me what you tried?

If you make a file with #!/usr/local/bin/dagger at the top it should execute with dagger (this is on MacOS, might be different on other OS) and execute it should just work

3. Are you trying to do this interactively or via file?

4. There is help built in but there are a few ways to get it depending on what you are trying to get help for 🙂

https://docs.dagger.io/features/shell/#built-in-help

[winter linden]

Small note I recommend #!/usr/bin/env dagger that way it will work everywhere

[vivid mango]

Thanks!!

1. <skip>

2. hmm, I tried with #!/usr/bin/dagger --progress plain and it prints me usage (helppage). I guess it means a parametrized shebang is not yet available?

3. via file. A dagger script. Smth like run_tests.dagger --env dev.

4. aahh, so it should be container | .help directory ... gotcha!

5. is there a way to recover shell after typing > w/o exporting the LLM auth token? The error is kind of blocking the view 🙂

[winter linden]

@vivid mango if you want to pass arguments in the shebang (--progress plain) then it's platform-dependent, I think on mac you need to pass -q -S to env?

@vivid mango try #!/usr/bin/env -S dagger --progress=plain

[vivid mango]

that worked! thanks.

Now what's left is to figure out daggerscript parameters and those damn directory permissions (posted in help section)

[winter linden]

Yeah passing arguments to the script is not supported but I agree we need it

[slender charm]

Did you see my response about permissions?

[scarlet prawn]

anyone can point me towards the best way to let dagger functions push to AWS ECR?

[vivid mango]

does dagger shell accept env vars in any shape or form perhaps? I'd like to pass them to containers and do the logic there. Does not replace script args, but should be sufficient for my use-case

[winter linden]

At the moment you can access env vars from the shell only when passing arguments of type secret. For example:

container | from alpine | with-secret-variable PASSWORD env://VAR_ON_THE_HOST | with-exec env | stdout

[vivid mango]

a bit annoying that then I can't print that value (for debug/log purposes, + if the value is "dev" - there will be lots of false-positive *-masks in the log) 😄
but it will do for now. Thanks!

[winter linden]

yeah sorry. The env vars you actually need are not secret correct?

[vivid mango]

yepp. They are environment names I'd like to run tests against. These will be selectable in the CI solution's web-ui. So definitely not a secret.

Any plans to introduce the env:// and similar syntax to with-env-variable ? Or interpolation of envs directly in the dagger shell?

Also, the literal commas in with-exec (commands) issue is still not resolved, right?

Someone tried to help me, but I still could not make it work. https://github.com/dagger/dagger/issues/7678#issuecomment-2768636623

[winter linden]

@vivid mango here's a fun debug bash function for you 🙂

getenv() { container | from alpine | with-secret-variable VALUE "env://$1" | with-exec -- sh -c 'echo $VALUE > output' | without-secret-variable VALUE | file output | contents; }

[vivid mango]

wait, dagger shell supports functions...? How did I not see that in docs...

[winter linden]

Yeah we haven't fully documented it yet... But it's a complete bash parser. So everything the bash syntax supports, it supports.

(thanks to @radiant crane for implementing the awesome mvdan.cc/sh go package which makes this possible)

[vivid mango]

including flow control operators? i.e. if/else/while/for/var interpolation bashisms (e.g. ${myvar:-default_val}, ...)?

[winter linden]

yeah I believe so

[vivid mango]

omfg....

[winter linden]

be careful what you wish for 😛

actually I've been experimenting with some fun consequences of late-binding bash functions

for example it supports mixin patterns which "regular" programming language don't

[vivid mango]

✔ { if [ $SHELL = "bash" ] ; then container | from alpine | with-exec -- echo OK ; else container | from alpine | with-exec echo ${SHELL}; fi ; } | stdout 0.0s
/bin/bash                                                                                                                                                                                                                                                                                                                     

⋈     

oooohhh booooiii. It's gonna be a fun night.

[winter linden]

install() { with-exec apk add $@; }

container | from alpine | install openssh git rsync | with-exec git version | stdout

[vivid mango]

While I have you.. What's the with-mounted-directory and how's it different from with-directory?

[elfin frigate]

with-mounted-directory avoids copying the directory; if you publish the container as an image for example, it won't contain the directory contents. with-directory on the other hand does copy the contents in, though it will also transparently perform a "merge" optimization to avoid doing a literal copy, if it can.

[stoic knot]

Is it possible to Lstat a file/dir in a container? I'm being passed a string and do not know which it will be, so I don't know which function to call

[stoic knot]

Is it possible to Lstat a file/dir in a

[vivid mango]

Is it still ephemeral? Or does it work like docker volumes - like a persistent volume?

[chilly arch]

Thank you to everyone who attended the Dagger Hack Night in London last night!

You packed the house, and came through with amazing demos.

We can't wait for the next Hack Night in SF. If you are in SF on May 14th, join us!
https://lu.ma/p85doawg

Huge thanks to the entire Puzzle ITC team for guiding users through the CI Hands-on Lab—and to Dagger Commanders @thick igloo and @sand gorge for supporting all the attendees as well.

Can’t wait for the next one! 😊

[static vessel]

Thank you all for organising the event. We would love to see more in the UK. It was quite fun.

Is there any link or place that we can download photos and demo recordings? Thank you

[chilly arch]

Recordings are coming soon! Please subscribe to our YouTube channel to be notified when they are uploaded. https://www.youtube.com/@dagger-io

I'll add more photos in the Hack Night channel, so you can grab them here - https://discord.com/channels/707636530424053791/1352393343878955008

Looking forward to doing more events worldwide soon! Until then, we look forward to seeing everyone on our community calls 🙂 https://www.youtube.com/@dagger-io/streams

[vivid mango]

yeah I believe so

[elfin frigate]

yep still ephemeral. for persistence, the closest thing is a CacheVolume

[boreal pine]

👋

[cunning jolt]

I am seeing these errors on 0.18.1. Regression?

It's actually not affecting pipeline functionality but looks ominous.

[austere maple]

Does this go away if you disable telemetry? https://docs.dagger.io/faq/#does-dagger-send-telemetry

[winter linden]

is that hostname from a custom otel collector you are running?

[cunning jolt]

This is in our eks cluster so yes it's possible. I can check to make sure. How ever this isn't occurring on 0.17.1

[tribal quail]

Hey everyone, i'm finding myself in a pickle and i'm seeking for advice ( maybe i'm overthinking all that )
I have a re-usable module pushed to private git repo. let's call it upload-to-s3
When creating the deploy pipeline for one of our apps i cant decide what's best to use that module:

• option 1 : Reference upload-to-s3 as a dependency of my app's dagger module and create a dagger function UploadToS3 which basically calls the external module
• option 2: do a dagger -m MY_S3_MODULE call XXX in my pipeline steps

I can see benefits for both options but i'm keen to know what's best practice

[fossil pine]

The downside of option 2 is that you still need an additional layer to configure your pipeline.

I generally prefer packing as much into dagger, so I have a single layer I have to deal with. It’s also easier to test locally.

That being said, if the only step in your pipeline is uploading something to s3, I can see option 2 working just fine.

My default would be option 1

[tribal quail]

obviously i got more than just an upload, building a frontend , running tests etc .
my pipelines are broken in two : build and deploy . the deploy bit only uses the external module

[fossil pine]

In that case, I would integrate the upload into the deploy pipeline.

[tribal quail]

as in call the app's deploy function and that delegates the logic to dependency (external module ) ?

[fossil pine]

Yep

[gleaming chasm]

why is this:

@func()
  async pipeline(
    @argument({ defaultPath: "/" }) source: Directory
  ): Promise<Directory> {
    let result = dag.directory();

    try {
      await Promise.all([
        (result = result.withDirectory(
          "/backend/test",
          await new BackendTest().test(source)
        )),
      ]);
    } catch (error) {
      result.withNewFile("error", error.toString());
      return result;
    }
    return result;
  }

not terminating gracefully when the backend test fails? i would have thought catching the error, would let the pipeline call finish and return the result directory, but it's not working 😦

Error: input: ci.pipeline process "bun /src/.dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

[winter linden]

My guess is that the error is caught by Dagger first?

Try passing await: Any to the underlying withExec

Then you can get the exit code yourself and decide how to handle it

[gleaming chasm]

mhm

you mean { expect: ReturnType.Any } ?

but i still need it to throw

i just want to handle the error myself

[winter linden]

Ah right. Yeah actually my explanation might be wrong

[normal needle]

I expected .refresh to reload the module functions after I just changed the code of a certain function in a module, but without quitting the dagger shell, it seems not picking up the latest function as I tested. Is it the right expectation ?

[cunning jolt]

I've noticed this too. I wonder if .refresh is bugged.

[sand gorge]

KubeCon baby! 🥳

[gleaming chasm]

disregard, this obviously doesn't work

[chilly arch]

I spy Dagger at KubeCon! Amazing presentation @thick igloo and @tacit hull

[winter plank]

🫠 wtf is going on, mv keeps failing in my container because it claims it can't find files, but they're clearly there! (https://v3.dagger.cloud/lamalex/traces/fc6ffad2d6c5d3197c02d9a16fd5f34b?listen=19bed63d99f309be&span=d39d91dd02cf9eae) ,
process "mv lambdas/fetch/*.js ." did not complete successfully: exit code: 1

.
├── lambdas
│   ├── fetch
│   │   ├── index.js
│   │   └── tokenManager.ts
│   └── result_push
│       └── index.js

[sharp marsh]

@winter plank it fails because * is a shell thing, not mv

the shell expands the * and passes the list of files to the mv command

you need to run the command within a sh -c context for that to work

[winter plank]

i wish that was it, but this was failing prior to me using * -- here it is with explicit paths doing the same, thing: https://v3.dagger.cloud/lamalex/traces/e77576c4af3cfd5ac99c0b35d793f5a2?listen=741201f17e81d545&span=443ff46f67589f2c

[sharp marsh]

@winter plank you have a typo in the tokenManager file. The extension is .ts, not .js

that's why the cp fails with the error that it can't find the file

[winter plank]

brb, gonna go kms. thank you

[rose magnet]

Hey @narrow nymph we're headed to Bletchley Park tomorrow! I'm super excited. Sadly our video editor isn't able to meet with us in London tomorrow as planned so that sucks. But now we have a free day. I'm so glad you mentioned B.P. because now I get to go!🎉

[frozen basin]

neat feature in datadog CI - finds error in the giant haystack of logs. could be a good feature for dagger cloud

[slender charm]

Dagger cloud bubbles up errors to the top of the page like this as well by default. Based on exit code.

[winter plank]

Question about local modules: the docs seem to indicate that local modules must strictly be in a child directory; i’ve got a monorepo with a few projects with their own dagger module, and they’re all using a fork of the awscli module, am i reading this right that there isn’t a way for them to share the awscli module? What I want is to add another sibling project just for project-local shared modules so like

.
|__ project-a/
|____ .dagger/
|__ project-b/
|____ .dagger/
|__ shared-modules/
|____ awscli/

and for a, and b to have a dependency on awscli

[winter plank]

Hm maybe i could symlink

[winter linden]

@winter plank the only restriction is that a module can only depend on local modules in the same git repository

[cunning jolt]

As long as your root directory is a git directory, you can install modules with relative path. So from project-a you can dagger install ../sharedmodules/awscli.

[winter plank]

Ah ok, i guess i misunderstood this: https://docs.dagger.io/api/module-dependencies/

You can also use local modules as dependencies. However, they must be stored in a sub-directory of your module. For example:

[winter linden]

yeah that sentence is wrong

[tired moth]

Can dagger work as otel collector to show traces in dagger cloud?

[sharp marsh]

It's tailored to display the Dagger engine traces. Not sure how random traces will be rendered. Having said that, you can always try it out

[winter plank]

https://github.com/dagger/dagger/pull/10075

[winter plank]

Just curious why you’d do this? What’s the goal?

[tired moth]

Idea/through is now a days so many technologies support otel tracing natively like https://docs.deno.com/runtime/fundamentals/open_telemetry/ but we are strugulling to make it work with dagger easily...so if dagger can be configure/act as otel collector then it can solve so many things...that's it
and basically we want to visualize traces of otel in dagger directly without any other third things intigration

[winter plank]

Would this be for like, something running as a dagger service, and collect tracing info for e2e testing?

[tired moth]

Yes, correct

[north dune]

hi! Has Dagger Cloud been discontinued? I've run into several links in various places to this pricing page which doesn't exist: https://dagger.io/pricing

[winter linden]

Sorry about that! Dagger Cloud is most definitely NOT discontinued but we just rolled out a new website design and it appears some pages are broken.

[north dune]

ohh ok! I just learned about Dagger and it looks really cool, but that was throwing me off 🙂 I have a monorepo with nearly 100 packages that deploys ~55 Cloudflare Workers and publishes ~10 docker images that I think I want to move to to Dagger

[winter linden]

That sounds fun, and a good use csse for Dagger 🙂 We would love to help you with that

[north dune]

thanks!! I'm using earthly right now, but Earthfile seems to be kind of abandoned so I'm looking at other options.

I really like Earthly's docker layer caching, so I'm excited that dagger can do that too

[winter linden]

indeed 🙂

for monorepo layout, I recommend reading this thread from another user recently migrating a monorepo to dagger. good discussion of repo layout: #1351362096901656638 message

see my advice at the end: #1351362096901656638 message

[north dune]

Thanks! I'll take a look

[slender star]

https://dagger.io/cloud

We'll fix up those links. Thank you!

[north dune]

Do people typically use the .dagger directory where you can do dagger -c ...? Or custom scripts with the sdk with commands like dagger run bun ./build.ts ?
I'm currently leaning towards the latter since it seems to have way faster startup time

[winter linden]

We recommend the former especially for a monorepo with lots of moving parts. otherwise you'll miss out on the composability

if you have a simple monolithic build that you don't plan to extend or modularize in any way, the former (call core dagger API from a custom client) works perfectly fine. Given your description of your setup though, I think you'd probably find yourself reinventing a lot of what modules get you for free

the typescript SDK definitely has a startuptime issue -we're working on that. though once the cache is warm it gets better

[north dune]

ok good to know. happy to hear TS will get faster - I can live with it until then

[winter linden]

(there might also be some knobs you can turn on the TS sdk too. like enabling bun maybe? sorry I don't know the TS part as well but others can chime in who know more. also check out #typescript )

[north dune]

ooo ok cool. FWIW I really like the idea of the normal TS SDK because I already have a TS CLI that lives within my monorepo that I use for tooling (replacing bash scripts and such), but I'll have to play with both to see what works best

In case you're interested - here's what the build graph looks like on Earthly - I think this will be cool to see in dagger 👀

[north dune]

Okay yeah now it's fast - earlier, for some reason it refused to cache anything so I thought the whole tool was really slow 😅

but now it seems to be caching the module loading and layers so it's fast now

(for reference, here are two runs I did where they didn't even cache the npm install command after the from())

[kindred dove]

I'm giving an internal demo of Dagger this week, and one of the potential questions I'm anticipating is around the long Dagger call commands we have at the moment. I've recently discovered Taskfile and Just which I thought could be a possible way to simplify my Dagger calls, especially with remote modules.
e.g. dagger -m https://github.com/MY-ORG/daggerverse.git/build-mod call build --source=./main-folder/sub-folder export --path=./buildresults
However I've also seen some discussion on here for simplifying the UX for Dagger with some sort of config file, so you'd end up with the command
dagger call build
This is a preferable option as opposed to adding another tool.
What's the progress of the Dagger call UX improvements?

[winter linden]

Hello! There are a few improvements already:

• Slight simplification but you can use the shorter github.com/my-org/daggerverse/build-mod

• Even better you can dagger install the remote module then reference it via its short name

• If you create a dagger module directly in the target repo (recommended) then your directory arguments can have default paths relative to the module or surrounding git repo. So that can save you the --source argument

• lastly you can use the new dagger shell syntax for less verbose chaining

altogether that will get you to:

dagger -c 'build | export .'

[kindred dove]

Thanks Solomon, that's a much neater command in the end. I'll look into that

[winter linden]

First step would be: dagger init --sdk=xxx at the location of the app you want to build. Then edit the module source to implement a custom build function. Install and call dependencies as needed .

For the source argument, look for "default directory" or "default arguments" in the docs

[marsh scroll]

Hey guys, any chance of this amazing dagger thing working with Apptainer?
without me converting stuff around? I can hack away at it if required, any help is appreciated

[winter linden]

Apptainer?

[north dune]

Is it possible to remove log prefix for certain withExec() logs? GitHub Actions logs aren't grouping together turborepo logs like it normally would because dagger adds a prefix

[winter linden]

Not sure, my guess is probably not. But we could add a command-line flag or env variable for overriding the prefix in plain output mode

[north dune]

that would be awesome

[winter linden]

Of course if you setup tracing in Dagger Cloud you will never feel the need to look at Github Actions logs again 😇 your turborepo logs are definitely grouped there

(spoken from experience, we use GHA + Dagger Cloud and I never never ever need to go to the GHA logs ** * **, which feels like an important milestone in my life)

** * ** unless I'm developing something specific to Github Actions, like the Dagger/GHA integration of course

[north dune]

That's true. The only annoying thing is that VS Code has shortcuts directly to my GHA jobs, so I'm always 1 click away from the latest job run in GHA, but not Dagger tracing

[winter linden]

That's a good idea for a VSCode integration 🙂

[north dune]

that would be cool!

[winter linden]

When you run your dagger pipelines locally, and you're looking at the cool terminal UI, you can press 'w' and it will send you to a web view of the same open telemetry trace (the TUI is actually rendering the otel stream locally)

Also once you configure your GHA runner with a Dagger Cloud token, there will be a trace URL in the GHA logs, so you can click that

(none of that addresses the convenience of the VSCode/GHA integration unfortunately)

[north dune]

yeah I did that and have used the link in Dagger Cloud - it's just extra steps 🙂 Not the biggest deal

[winter linden]

https://github.com/dagger/dagger/issues/10083

[north dune]

What's the difference between specifying excludes in dagger.json and the ignore list in a source Directory within a module?

[winter linden]

Excludes in dagger.json are for excluding files when loading the source code of the dagger module itself. It's only useful when your module is mixed with an existing software project. For example if you have a node app in ./myapp, and a dagger module defined in ./myapp/dagger.json with a source directory in ./myapp/.dagger, then you might exclude the node_modules directory of the "host" webapp.

Ignore list in a source directory is for pre-call filtering of that directory - dagger filters out the specified files before uploading the directory into the engine

[north dune]

ok I think I get it. I was confused because it seemed like I had to exclude node_modules in both to speed up dagger (presumably because it was taking a long time to copy over node_modules)

Right now I have a .dagger directory at the root of my monorepo, and a 1.7GB node_modules at the root as well, so it gets really slow if it's not excluded

[north dune]

Can I add custom spans to traces using the OTEL api? I just realized how amazing it would be if I could add spans for each of the tasks that turbo runs 👀

[winter linden]

Yessir 🙂 cc @elfin frigate

[north dune]

Omg what, this will be soooo nice because my CI job runs nearly 500 scripts with turbo

[winter linden]

Also you can execute otel-instrumented tools in a container, and by default Dagger injects the otel collector config in the container's environment. For example we run our own Go test suite in Dagger, in such a way that the individual tests also show up in Dagger Cloud, as children of the corresponding withExec(["go", "test", ...])

[north dune]

Oooo ok I need to learn how that works

Right now, I use a flag to tell Turbo to export to json and then write that to Axiom so that I can graph it in Grafana, but traces would be sooo much nicer

Is it possible to get the full trace after dagger completes so I can send it to Axiom? That way I can query the same data in Grafana

[elfin frigate]

If you run the dagger CLI with standard OTEL_* config env vars it'll send everything to it - looks like you would need something like this:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.axiom.co/
export OTEL_EXPORTER_OTLP_HEADERS="x-axiom-dataset=$DATASET_NAME,Authorization=Bearer $API_TOKEN"

[elfin frigate]

For custom spans, those same env vars come up again - Dagger configures OTEL_* vars for things it runs which sends telemetry to the CLI which then passes it along (and shows it in the TUI), so you can use the usual OTel SDK for your language (Java in this case?) to detect those env vars and emit telemetry. You'll need to also pick up the TRACEPARENT env var so your spans inherit from the right parent span + trace. I have a wip PR that adds an API for all this which should make things a bit simpler - using OTel SDKs directly can sometimes be a bit unwieldy.

[north dune]

Amazing! I’m excited to try this

Oh and I’ll be instrumenting with TypeScript - I have a TurboRepo wrapper that grabs all the timings after turbo finishes running

[tired moth]

We also. 🥳

[winter plank]

Does anyone have an approach to polling that works?

       const initValues: GetCanaryCmdOutput = JSON.parse(await awscli
            .exec(getStatusCmd)
            .stdout());
        let state = initValues.Canary.Status.State;
        const lastStart = initValues.Canary.Timeline.LastStarted;

        await awscli.exec([
            "synthetics",
            "start-canary",
            "--name",
            canaryName,
        ])
        .sync();

        let updatedLastStart = lastStart
        while (state === "STOPPED" && lastStart === updatedLastStart) {
            const resp: GetCanaryCmdOutput = JSON.parse(await awscli
                .exec(getStatusCmd)
                .stdout());

            console.log(JSON.stringify(resp, null, 2));
    
            state = resp.Canary.Status.State;
            updatedLastStart = resp.Canary.Timeline.LastStarted;
        }

        while (state !== "STOPPED") {
            const resp: GetCanaryCmdOutput = JSON.parse(await awscli
                .exec(getStatusCmd)
                .stdout());

            console.log(JSON.stringify(resp, null, 2));
            state = resp.Canary.Status.State;
        }

this isn't working like i'd expect, i'm getting hung in that first loop, even though in another terminal i can see the values changing (running aws cli outside of dagger)

[winter plank]

lol hm is it because of cacheing? 🧐

[winter plank]

lol yep, it's cached

[winter plank]

hm, is there a way to force a command to run and ignore the cache

[neon warren]

I’ve seen a pattern of setting an env var with the current timestamp to prevent or bust cache

[winter plank]

that's what i was just about tot ry

and since i can't figure out how to sleep, this way i can fake sleep by adjusting the cache buster haha

[neon warren]

That’s the best approach I’ve seen, but worth a bigger discussion on how to control the caching outside of code changes

[winter plank]

🎉 it worked, thank you 🧠

[tired moth]

In dagger can we prune/erase cache for only affecting files connected to dagger call command?

[old fiber]

i was doing this for my go-migrate cicd pipes. the pg service container state would change, but because the inputs didn’t, i had to bust the cache to force it to run the migrations. i didn’t play around with it tremendously, but i did try to bust it at the last possible point to use as much of the cache as was prudent. some of the location optimization of the cache busting did save me about a minute on the execution time (in lieu of just smashing it first thing like an absolute barbarian - which i am accused of regularly fwiw)

[torpid maple]

Is it possible to return multiple files from a dagger function?, imagine I wanted to return multiple build artifacts etc.

[winter linden]

You can only return one value, but that value can be a custom object with as many fields as you'd like. Basically you can define your own composite artifact and return that

[vivid mango]

Why doesn't dagger work when using k8s services?

On my k8s cluster I am running a dagger daemonset and I tried running CI jobs configuring CI dagger to use remote engines via a service name. But in my CI job a simple dagger run echo HELLO fails with

input: moduleSource failed to stat local path: failed to stat path: failed to get requester session: session for "v8o21rqyxyv5ezwoxx8ckn3k3" not found

Stracing dagger I see that during this simple job execution dagger CLI opens 4 connections to the dagger engine. Is that the reason? kube-proxy routes service connections to another pod each time in a round-robin fashion. Is this why connection attempts to dagger engine through k8s service name fail?

It works perfectly fine when I connect through a node IP.

[chilly arch]

**Want to start building agents, but not sure where to start? **

Join the Kubesimplify livestream happening now! @winter linden , @warm temple and @slender star are walking through how to do it step-by-step!
https://www.youtube.com/watch?v=baX9JPvFgD0

[slender star]

I'm lurking nearby and happy to answer questions here 🙂 🙌

[winter plank]

The new home page looks great

[slender charm]

Thanks so much 😍

[tough ferry]

I really like chaining in the dagger cli. I haven't seen if it is possible to chain between 2 modules. For example a community GO module outputs and dagger.File, I would like to be able to chain that file and package it up in a container using a different module. Is that possible with one cli call?

[warm temple]

Welcome! Yes in dagger shell you can have subshells, so it might look something like

container | from alpine | with-file /bin/foo $(github.com/foo/daggerverse/golang | build $repo)

Where the Go module is github.com/foo/daggerverse/golang with a function build

[tough ferry]

I was trying to invoke from a github action which why I was thinking CLI comman vs Shell

[warm temple]

Got it! You could run the same command with dagger -c 'container | ... '. I wonder if you could literally set the shell to dagger in github actions. Trying that now

[tough ferry]

I had totally missed the -c option. Thank you!

[warm temple]

ok this is pretty cool. You can set dagger as the shell in github actions and then use the dagger shell syntax in the run step. For example

jobs:
  ci:
    name: ci
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dagger/dagger-for-github@8.0.0
        with:
          version: 0.18.0
      - name: Check
        run: check | with-foo bar | stdout
        shell: dagger {0}

the trick is shell: dagger {0}

[slender charm]

This is dope, belongs in the cookbook!

Thanks chef 🙂

[warm temple]

on it boss 😛 was inspired by this post I saw yesterday https://news.ycombinator.com/item?id=43617493

woodruffw

Any program can be a GitHub Actions shell

Hacker News · Apr 8, 2025, 1:20 AM

[winter linden]

@warm temple do you need that {0}? I was expecting it would work with just shell: dagger no?

(not sure what it does actually)

[warm temple]

yeah it's a weird github thing for non-default shells. It passes the whole run block as a file to the {0} argument

https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell

[winter plank]

Can File only return string? (i need it to return binary data 😥 )

[slender charm]

File can return the file, you just need to pipe export at the end

[winter plank]

I may just ... broadly be doing something i shouldn't,

const data = XLSX.read(await disclosureDocument.contents(), {
      cellDates: true,
      cellStyles: true,
      cellHTML: true,
      cellText: true,
    });

where XLSX is a npm package

[slender charm]

I see, yeah contents is a string, don't think there is currently any way to change the output type here

However, there is likely some library in node that can handle this for you. You can use grab the file using the currentModule api https://docs.dagger.io/api/types/#currentmodule and go from there

[slender star]

base64?

[winter plank]

the function i'm passing it to is expecting an array of u8

[chilly arch]

**See AI agents in action! **

On tomorrow’s Community Call, @static vessel will demo the cloud misconfig agent he rapidly built at our London Hack Night, and @slender charm returns to share progress on his autonomous QA agent since our last call.

Join here: https://www.youtube.com/watch?v=hH7xVouYr6w

[torpid maple]

Any ideas why I'm getting these errors running to run dagger in GitHub Actions?

21  : go SDK: load runtime DONE [10.9s]
18  : moduleSource(refString: "."): ModuleSource!
24  : │ moduleSource(refPin: "833ec36632b2457862f6e3bf1f7107ad65e3e515", refString: "github.com/marcosnils/daggerverse/k3s@k3s/v0.1.7"): ModuleSource!
23  : │ │ git://github.com/marcosnils/daggerverse#833ec36632b2457862f6e3bf1f7107ad65e3e515 ERROR [10.4s]
23  : │ │ ! failed to fetch remote https://github.com/marcosnils/daggerverse: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/marcosnils/daggerverse/': Recv failure: Connection reset by peer

I assume this is due to git using the default SSH credentials provided to the workflow, and by default it only has permissions to interact with it's associated repository?

I thought it would use HTTPS over SSH to do the clone though.