daggernauts

[deft rain]

can be reproed more easily with dagger call -m github.com/fluentci-io/daggerverse/deno-sdk@main

[deft rain]

fix in https://github.com/dagger/dagger/pull/8667

[wheat river]

Maybe we can leave this channel mostly focused on module builders. Seems like #general is broader. When building modules and testing them, it's easy to discover edge cases and things that aren't spotted easily when you're just using Dagger (even writing simple modules). Nevertheless, I understand the hassle of too many discord channels 😅 .

[wheat river]

0.13.4 🐛 ?
Is it expected to require as mandatory the --sdk argument when doing dagger develop on version 0.13.4?
After upgrading a module to the latest version, I'm getting:

Error: dagger develop on a module without an SDK requires either --sdk or --source
error: Recipe `reloadall` failed on line 124 with exit code 1

Previously, without specifying the SDK, just running Dagger, development in a module's directory worked well.

False alarm 😁 — it was a local issue from my justfile and the way I was using to execute/wrap Dagger commands.

[kind carbon]

env: prefix

[restive shore]

I kind of feel the same way. Somehow, this content doesn't feel like it goes with community call announcements and/or intros and basic dagger questions. I can go either way though, i understand the management overhead.

[foggy stream]

Hey folks 👋 Is there someone who uses dagger programatically without the CLI?

[wheat river]

On version 0.13.4, I'm getting

Error: dagger develop on a module without an SDK requires either

On a module (where I'm in the actual module's directory) that has this dagger.json file.

{
  "name": "tests",
  "sdk": "go",
  "exclude": [
    "../../.direnv",
    "../../.devenv",
    "../../.vscode",
    "../../.idea",
    "../../.trunk",
    "../../go.work",
    "../../go.work.sum"
  ],
  "dependencies": [
    {
      "name": "parentmodule",
      "source": ".."
    }
  ],
  "source": ".",
  "engineVersion": "v0.13.4"
}

This is odd, when doing a cd parent module/tests where the actual module resides and just doing dagger develop, it started to throw this error. I don't know if it's related to the recent version. What's also odd is that the SDK is explicitly stated in the JSON configuration file. Is there any reference for a schema or valid structure for the dagger.json file?

[deft rain]

what's the contents of ../dagger.json?

[wheat river]

The parent module has this dagger configuration file

{
  "name": "terragrunt",
  "sdk": "go",
  "exclude": [
    "../.direnv",
    "../.devenv",
    "../go.work",
    "../go.work.sum",
    "tests",
    "examples/go"
  ],
  "source": ".",
  "engineVersion": "v0.13.4"
}

[deft rain]

i am a bit confused, when you say cd parent module/tests, does parent module exist as an actual directory? or is it ..?

[wheat river]

Let me simplify. This is the module's structure (there is a parent module, and other submodules that installs the parent)

mymodule
├── .gitattributes
├── .gitignore
├── LICENSE
├── README.md
├── apis.go
├── base.go
├── commands.go
├── config
│   └── presets
│       └── base-alpine.yaml
├── dagger.json
├── defaults.go
├── err.go
├── examples
│   └── go # example module, it's a dagger moduile that installs the parent (mymodule) module
│       ├── .gitattributes
│       ├── .gitignore
│       ├── dagger.json
│       ├── go.mod
│       ├── go.sum
│       ├── main.go
│       └── testdata
│           └── common
│               ├── README.md
│               └── test-file.yml
├── go.mod
├── go.sum
├── main.go
├── tests # test module, it's a dagger moduile that installs the parent (mymodule) module
│   ├── .gitattributes
│   ├── .gitignore
│   ├── apis.go
│   ├── base.go
│   ├── dagger.json
│   ├── err.go
│   ├── go.mod
│   ├── go.sum
│   ├── main.go
└── toolkit.go

Previously, in version 0.13.3 by just doing cd mymodule && dagger develop it worked. Now, after upgrading it works only if I pass the --source flag which I'm not sure why would that be required since my dagger.json file (of the test/ module) already includes the attribute source=. on its configuration.
This command works: @cd mymodule/tests && dagger develop --source=.

[deft rain]

if you remove mymodule/tests from the top-level excludes does it work?

the only thing we've changed around this area in the last release looks like https://github.com/dagger/dagger/pull/8575

cc @kind carbon do you have any better ideas?

[wheat river]

Yeah, I checked that as well, and the error was the same. It only works now by passing explicitly the --source (which seems to be redundant in my particular case).

[latent trellis]

What's the current recommended way to notify daggerverse about a new version of a module?

This is what I do at the moment: https://github.com/sagikazarmark/daggerverse/blob/main/.github/workflows/publish.yaml#L62

(Call dagger publish for the module)

But it clearly doesn't work. A bunch of my modules are quite outdated on Daggerverse. :/

[deft rain]

cc @wintry prism have we changed this recently?

[latent trellis]

I noticed it a couple weeks ago. Maybe even months.

[wheat river]

Oh, actually. By (from the top-level module's Dagger Json file) removing the two submodules from the exclude directive, dagger develop just works again.
For the record, it's something that only occurs to me when upgrading to the most recent version. Maybe it's a change introduced regarding context directory functionality. I've been adding this exclusion (of the two submodules) in the top-level exclude option in the parent's dagger.json file for all my modules without problems. Only after upgrading, I started to get these issues.

[plucky ermine]

Hey! There are a few folks doing this, is there any specific questions you have around this approach?

[wheat river]

Oh, actually. By (from the top-level

[wintry prism]

What's the current recommended way to

[foggy stream]

Yea I would love to know about their experiences – are they use it within go, are there some examples? I'm having a hard time finding something on the docs about it.

We want to explore using it at Netlify but it would need some programatic API.

[plucky ermine]

This branch of the docs discusses how to use dagger without modules. This is still fully supported but we’re working on nicely integrating both pathways into the docs - https://archive.docs.dagger.io/0.9/

One great example of this approach is in the airbyte project https://github.com/airbytehq/airbyte/tree/master/airbyte-ci

this uses the dagger library in python without modules or requirements around the CLI

I am curious to hear more about why the CLI isn’t going to work for you.

[sharp zealot]

Hi Luka! Welcome 🙂

@plucky ermine already mentioned the use case of using Dagger purely as a library. The fact that it's supported but poorly documented etc.

I will add that whether to use the Dagger CLI is orthogonal to whether to use it programatically. In every case, Dagger is programmable and that is how we all use it. The question is simply where is the client.

The most powerful way is to extend the Dagger API with a module. Then you are basically programming Dagger from the inside. That is what most Dagger users do. In this scenario the client is running inside Dagger. The only task remaining outside of Dagger is to load your module and call your entrypoint - typically that entrypoint is very simple, just one function call and a few arguments. You can still call your entrypoint programmatically if you really want to, but it's such a trivial call that it doesn't really matter - most users just use the CLI to call it.

The point is to clarify - when you see people using the Dagger CLI, it's not that they're not programming Dagger. They're just using a more advanced way to program Dagger (a module) and using the CLI to bootstrap that.

Other ways to load your module:

• dagger listen -> this will serve your module's API on a local socket, then you can make raw http/graphql queries
• dagger query -> this will take a raw graphql query and craft the http request for you in a temporary session

I strongly recommend following the standard path and writing a module, to extend the Dagger API. This will give you the most power. Then know that at any time you can eject from the CLI and instrument your module directly - that is the easy part.

(sorry that the standard docs and onboarding doesn't make that more clear. We are working on a docs improvement which will help I think)

[dense canyon]

I'll look into this Mark

[wintry prism]

#1293942036814106655 message

[latent trellis]

After upgrading to 0.13.5 from 0.13.1, I noticed that the installing module step at the beginning takes an unusually long time. It doesn't seem to be consistent.

For example, I have a job here that's been running for 17m now: https://github.com/openmeterio/openmeter/actions/runs/11315761867/job/31467309762?pr=1662

I haven't been able to dig deeper yet, but it's definitely new (not sure if it's dagger related, but started right after upgrading), so if anything changed around that part of the code since 0.13.1, it may be worth checking out.

[latent trellis]

Just a hunch, but the problem could be Dagger Cloud. Downgrading to 0.13.3 didn't help. But disabling cloud did.

[latent trellis]

Opened an issue

[wintry prism]

@latent trellis for me to repro, what would be the best local command?

[latent trellis]

dagger call -m github.com/openmeterio/openmeter ci

0.13.5 with dagger cloud caching consistently hangs for me at the moment

Though it looks like this one hangs with dagger cloud disabled too: https://github.com/openmeterio/openmeter/actions/runs/11317123433/job/31470176906

[digital sparrow]

Let’s say our company has a set of distributed services. Most of the time you only work on a single service (or maybe you integrate with a second, but that’s still fine). 5% of the time however you need to bring up most of the system and that becomes a pain (most people know some systems better than others).

I decided I don’t want to use containers to bring up each individual service (I tried).

Now I’m back to the idea of running services natively, but having a boot script that checks for dependencies, env variables, etc.

Is dagger a good fit here given I don’t want to containerize our services?

I don’t want to solve this with bash scripts and yaml (process-compose).

[plucky ermine]

The dagger.Service abstraction is normally used for services that run as containers, but they can also be things that are running on your host

https://docs.dagger.io/manuals/developer/services/#expose-host-services-to-functions

Now I’m back to the idea of running services natively, but having a boot script that checks for dependencies, env variables, etc.

If you pass in a service to a dagger function this way you will be able to connect to it as normal, but you will not be able to run scripts on the host where the service is running, so doing things like boot checks and env checks is not possible with dagger.

I decided I don’t want to use containers to bring up each individual service (I tried).

Can you tell me more about why this isn't an option for you? We have folks modeling some very complex services as a part of their dagger pipelines with success.

[digital sparrow]

Cool, thanks for the clarification

It's been a few weeks since I last tackled this project, but here's two things that come to mind:

1. first off it seemed like we were going down the wrong path when we couldn't reuse any of our production Dockerfiles - instead we wrote new ones that were very minimal and had little in common => I realize this is where Dagger would shine (at the time we wanted to see how far we can get with plain docker compose)

2. we still want to run those services natively (as I said, 95% of the time we don't need to start the whole system) and that's where we had issues - switching back and forth from the docker setup back to native services things started breaking - needing to remove node_modules or various pid files, etc ... seemed like there was no way for the two approaches to coexist => that led me to wanting a way to run the services natively, either in isolation, or orchestrated with the help of a tool like dagger or dev-env

Hope that makes sense

[dire steppe]

not a feature request because this may be a bad idea, but there isn't a way to access an environment variable from the shell that is invoking the dagger function via dagger call, right?

[warped canyon]

Correct! They must be passed explicitly as arguments

[warped canyon]

With how often we have WithMountedDirectory().WithWorkdir() I wonder if we could have a new API for WithMountedWorkdir() instead 🙏

https://github.com/dagger/dagger/issues/8726

[plucky ermine]

This is a great idea

[dense canyon]

cc @sharp zealot since this is an API change. If you LGTM we can ship it fast

[wheat river]

I use that a lot. When dealing with IaC, it's important to tell the binary (i.e. terra grunt) where it needs to run the commands — mounting the dir and setting the working directory always come together. Great idea.

[sharp zealot]

🙏 🙏 🙏 https://github.com/dagger/dagger/issues/8740

cc @deft rain @rocky harbor as discussed 🙂

[spiral whale]

Do i missing something. I've added following function to add cache config to my internal module and called using an go sdk. However, I'm getting following error while doing it.

Error:

error: parse selections: parse field "withMountedCache": init arg "sharing" value as dagql.DynamicOptional (CacheSharingMode) using dagql.DynamicOptional: invalid enum value "locked"

Function:

// Mount a cache volume for Go build cache.
func (m *Go) WithBuildCache(
    cache *dagger.CacheVolume,

    // Identifier of the directory to use as the cache volume's root.
    //
    // +optional
    source *dagger.Directory,

    // Sharing mode of the cache volume.
    //
    // +optional
    sharing dagger.CacheSharingMode,
) *Go {
    return &Go{
        m.Container.WithMountedCache(
            "/root/.cache/go-build",
            cache,
            dagger.ContainerWithMountedCacheOpts{
                Source:  source,
                Sharing: sharing,
            },
        ),
    }
}

[wintry prism]

https://github.com/dagger/dagger-go-sdk/blob/v0.13.5/dagger.gen.go#L8067-L8076
Do you need "LOCKED" ?

[spiral whale]

No it is same for any other option as well

[spiral whale]

I’m using these options in func call

[wintry prism]

oh, I see, maybe optional and ENUM interaction...

@upbeat herald 👆

https://github.com/dagger/dagger/pull/8148

@deft rain any ideas on what @spiral whale is seeing above? 🙏 👆

[spiral whale]

I’m suspecting there is a lower case conversion somewhere. Because enum defined uppercase but error is lower

[wintry prism]

cc @deep hedge in the chance you've see some case conversions.

[deft rain]

if you make it non-optional does it work?

how are you invoking the function @spiral whale?

[spiral whale]

Using go sdk from my ci module

[spiral whale]

out of office at the moment. Will try later

[spiral whale]

no it's not working

[deft rain]

@spiral whale can you open a gh issue?

[tacit kestrel]

Is there any way to run commands against a Service once the Container is converted and started?

[surreal sable]

👋 I think I found a 🐛 - this Go code in a dagger function:

func (m *Rack) Join(r *Rack) *Rack {
    m.Name = m.Name + "+" + r.Name
    m.Services = append(m.Services, r.Services...)
    return m
}

Gets generated as:

func (r *Rack) Join(r *Rack) *Rack {          
        assertNotNil("r", r)
        q := r.query.Select("join")           
        q = q.Arg("r", r)
                                             
        return &Rack{
                query: q,                   
        }   
} 

Which fails on the redeclaration of r - if I change my input variable to anything other than r it builds fine.

[surreal sable]

https://github.com/dagger/dagger/issues/8774

[daring bramble]

I spent 5hrs trying to make Dagger use the GitHub Action cache. I got pretty far but cache is still not effectively used.

Has anybody done it, or can you suggest a different way? Like tar and cache some paths and use the "local" way but on actions?

[restive shore]

When using enums from external modules, you can't use the same type (as defining module) as a parameter input to the external module functions right? You have to use a string or another type of parameter and then convert internally? Am I using this right?

[deft rain]

I think that's right, yes

[restive shore]

Thanks for confirming!

[dense canyon]

hi there, it's generally not recommended to cache things by copying things from /var/lib/dagger as it's very ineficcient and error prone if you're trying to distribute the cache across multiple parallel workloads. Since Dagger is based on buildkit, you can try using buildkit's experimental GHA cache exporters (https://github.com/moby/buildkit/tree/master?tab=readme-ov-file#github-actions-cache-experimental) which some users in this community have tried before without a lot of success given some instabilities and performance issues.

On the other hand, Dagger Cloud (https://dagger.io/cloud) offers an experimental distributed caching service that you can try and see if it fits your needs.

[restive shore]

I was using apk commands to install things into my custom dagger engine image. It no longer works in v0.13.6 I get

ERROR: Unable to read database state: No such file or directory
ERROR: Failed to open apk database: No such file or directory

What's the alternative?

[latent trellis]

maybe needs apk update? if the database is removed from the final image which is a common optimization.

[restive shore]

That's what I usually do. Getting the same error for apk update

apk update works in v0.13.5 so, a regression?

[latent trellis]

probably it is then

[restive shore]

https://github.com/dagger/dagger/issues/8787

[sharp zealot]

We'll try to help but you're in unsupported territory here

you might be better off customizing the engine build pipeline, to use your own base image

then you'll be building on solid ground: assuming you wrap the official build functions for the binaries

[restive shore]

The only thing I needed to install is the docker-cli. But I've moved away from using the engine image that way.

you might be better off customizing the engine build pipeline, to use your own base image
then you'll be building on solid ground: assuming you wrap the official build functions for the binaries

What makes a dagger engine the dagger engine? Is there documentation somewhere on what's required if I'm using my own base image?

I'd rather not maintain a custom base version of dagger engine as all I need is a few env vars and the CA certs for it to work

[sharp zealot]

that's a reasonable dependency, we could add a installDockerCli flag in the official build function

[restive shore]

It would be nice if there was a dagger engine module which generates the engine to the official Dagger specs. Then I could use it and add on my requirements and publish to my registry.

So instead of using the dagger engine as my base, I'd be using the "Dagger Engine Module" as my base

[dense canyon]

wouldn't that be the module we use ourselves to build our engine? dagger/dagger is a Dagger module which has all the required functions to develop and build the same verison of the engine we publish ourselves

[warped canyon]

dagger -m github.com/dagger/dagger@v0.13.5 call engine container --help

[dense canyon]

having said that @restive shore there's a stopgap you can do to add packages to your engine currently since our engine module doesn't allow customizing the underlying image packages yet.

FROM registry.dagger.io/engine:v0.13.6

RUN <<-EOF echo > /etc/apk/world
alpine-baselayout
alpine-keys
apk-tools
busybox
libc-utils
EOF

RUN apk -X "http://dl-cdn.alpinelinux.org/alpine/v3.2/main" add curl

edit: added a note to your issue to extend our module so it allows installing extra packages

[restive shore]

oh interesting! I can try that. BTW, I am using dagger to build that image. So, not a dockerfile 🙂

[dense canyon]

awesome!

[restive shore]

Let me try this. I have not been able to get these to run in my corp env

[restive shore]

I ran it. but I keep getting tls: failed to verify certificate: x509: certificate signed by unknown authority. My CA certs chain is installed in the engine container.

It's what I saw earlier also, I didn't have time to follow up

I am seeing these too and my proxy is set up inside the container

Syncing symfony/polyfill-mbstring (1.x-dev 8740a07) into cache
                        ┃     Failed to download symfony/string from dist: curl error 28 while downloading https://api.github.com/repos/symfony/string/zipball/c38cbd6dcf2a45bcfbd79fb89d6bd08f7afdc7dc: Proxy CONNECT aborted due to timeout

it's coming from ✘ Container.withExec(args: ["composer", "install"]): Container! 2m42.5s

If I git clone one of these failed pulls from within my custom engine container, it clones fine. So that tells me the proxy is fine inside the container but it's not propagated to sub containers. Possibly the same issue with CA certs.

[latent trellis]

Use Dagger version from dagger.json in GHA: https://github.com/marketplace/actions/dagger-version

See it in action: https://github.com/openmeterio/openmeter/pull/1753

[restive shore]

I am working on a demo and noticed that for context modules the pragma for default is defaultPath where as for all other defaults its default. Which is ok but the error I get when I use default for a dagger.Directory is very confusing

get module name: input: failed to get schema: failed to get schema for module "ci": failed to get field spec: failed to decode default value for arg "src": decode "Directory" ID: failed to decode base64: illegal base64 data at input byte 0

I think this should be improved.. maybe something like for directories use defaultPath instead of default

[dense canyon]

Agree, mind opening an issue please? We intentionally used a different pragma given that context directory has a particular behavior.

[restive shore]

of course not! Here it is 🙂 https://github.com/dagger/dagger/issues/8791

I have another one. Is it possible to add default values to the go doc? Not sure about other languages but for go, when I look at the generated doc for an optional type, it's not showing the +default pragma values. Which I expected to see. It does show if I create a comment myself.

The CLI shows the default values correctly, so the same should be reflected in the SDK docs.

[dense canyon]

not sure how we could do that since we don't control what go doc effectively outputs 😬

daggerverse docs shows the default values though

[restive shore]

the doc shows my custom comments though? Aren't the pragmas effectively comments?

[dense canyon]

not really, they're pragmas, the parser treats them differently

probably go doc has some custom logic to hide them or something

still.. not something that we can control though 😬

[dense canyon]

I wish there was a command to vendor or add a replace directive somehow so I can make changes to a module dependency faster. I'm basically manually vendoring the module now and replacing my dagger.json to point to it.

WDYT? cc @sharp zealot

[dense canyon]

It'd be awesome if Go had native optional arguments and that context directory would work across functions, so you can do something like this:

func (m *Lala) All(
    ctx context.Context,
    // +defaultPath="/"
    src *dagger.Directory,
) error {

    eg := errgroup.Group{}

    eg.Go(func() error {
        _, err := m.Test(src.Directory("/test")).Sync(ctx)
        return err
    })
    eg.Go(func() error {
        _, err := m.Lint(src.Directory("/lint")).Sync(ctx)
        return err
    })

    return eg.Wait()
}

func (m *Lala) Test(
    // +defaultPath="/test"
    file *dagger.Directory,
) *dagger.Container {
    // perform test
    return dag.Container()
}

func (m *Lala) Lint(
    // +defaultPath="/lint"
    file *dagger.Directory,
) *dagger.Container {
    // perform lint
    return dag.Container()
}

Without having to do all the src.Directory dancing to pass the arguments over... 🤔

[restive shore]

Testing #8805 - Github PAT support

[sharp zealot]

Today @distant geyser mentioned the problem Dagger Modules having a steep learning curve, and floated the idea of "single-file modules" as one possible solution. So I created an issue to discuss that. Would love to get your thoughts! https://github.com/dagger/dagger/issues/8808

[pliant path]

left my 2 cents, -> https://github.com/dagger/dagger/issues/8808#issuecomment-2446193699

[bronze palm]

hello, i new in dagger, try to understand how the call works to integrate it in some pipelines across my work ,
i do have the following error:" : prepare
13 : prepare ERROR [0.0s]
13 : ! unknown command "LibraryFetching.initialize_repo_and_submodule" for "dagger call"
"
i call the dagger using this: "dagger -m "./library_fetching/dagger/src/main" call LibraryFetching.initialize_repo_and_submodule , also try this option: dagger -m "./library_fetching/dagger/src/main" call initialize_repo_and_submodule
" same error, i use azure pipelines hope some one can give me a hand please

[dense canyon]

hey @bronze palm ! Welcome!

are you able to run those functions locally?

[bronze palm]

i call function "dagger call initialize_repo_and_submodule --source ."

this is my module:
"
import os
import dagger
from dagger import dag, function, object_type

@object_type
class LibraryFetching:
# No init method needed

@function
async def initialize_repo_and_submodule(
    self, 
    repo_url: str, 
    submodule_url: str, 
    submodule_path: str, 
    submodule_branch: str, 
    main_branch: str, 
    git_user: str, 
    git_pat: str
):
    "

got this eerror:

Module.initialize: Module! 1.9s
! failed to initialize module: failed to call module "library fetching" to get functions: call constructor: process "/runtime" did not complete successfully: exit code: 1
┃ ╭─ Error ──────────────────────────────────────────────────────────────────────╮
┃ │ Module “library fetching” doesn't define any top-level functions or a │
┃ │ “Library fetching” class decorated with @object_type.

[dense canyon]

what do you get if you call dagger functions?

that should list the functions available in your module and how to specify their name through the CLI

[bronze palm]

o got this:

[ashen spindle]

Error: response from query: input: container.from.withLabel.withExec.withDirectory.withWorkdir.withLabel.withExposedPort.withExec.asService.up resolve: failed to start host service: start upstream: service exited before healthcheck

How do I debug this, eg get the container logs?

[versed sedge]

When I ran dagger functions , i got some problems. how can i solve this issue? Anyone could help me figure out this.

is it network problem?

[sonic vessel]

What's the different between CPU Pressure (some) and CPU Pressure (full) ?

[elfin lark]

time="2024-11-03T09:17:17Z" level=error msg="solve error: process \"/runtime\" did not complete successfully: exit code: 1\ngithub.com/moby/buildkit/solver.(*edge).execOp\n\t/go/pkg/mod/github.com/dagger/buildkit@v0.0.0-20241010144035-f69c0798296a/solver/edge.go:912\ngithub.com/moby/buildkit/solver/internal/pipe.NewWithFunction[...].func2\n\t/go/pkg/mod/github.com/dagger/buildkit@v0.0.0-20241010144035-f69c0798296a/solver/internal/pipe/pipe.go:78\nruntime.goexit\n\t/usr/lib/go/src/runtime/asm_arm64.s:1223" client_hostname=MacBook-ABC.lan client_id=5tehhfq10qikvzjsblcqv6v56 function=up module=inbdm object=Inbdm session_id=bjjfb9jg0w9llmefifapnl4da spanID=a31d812459003a85 traceID=328d636eda1d010e6e7b4ecd79171de1

Can you please give me hint how to troubleshoot such issue? Using 0.13.7

[wintry prism]

When I ran dagger functions , i got

What's the different between `CPU

[ashen spindle]

Is daggerverse's crawling broken: https://daggerverse.dev/crawl/github.com/sai-php/sai ? This module installs locally

[dense canyon]

👋 adding @$commit at the end of the ref seemed to work.

https://daggerverse.dev/mod/github.com/sai-php/sai@d0380650a1a5ca0b0244ce34820a6b5cc99f6c1b

[dense canyon]

@wintry prism @warped canyon I believe this was possible in the past. Seems like we might have a regression where publishing modules without a @$ref is not possible anymore?

[warped canyon]

☝️ this is now fixed! thanks for reporting

[latent trellis]

Talked to a friend about Dagger today. One thing that made me think:

According to the teachings of the Church of Dagger, CI should ve treated as code.

If that’s true, you should be able to debug it.

When using modules that becomes much harder, since dependencies live in different repos.

If you own the repo and know the module, it’s probably easier.

But if you know nothing about it, except what you read on Daggerverse, how do you tinker with a module code?

If you know how modules work, you can find the source, copy it, replace the dependency, but that’s tedious.

Would be nice to have a vendor or replace command that downloads the module code to a local dir (.dagger maybe) and replaces the module ref in dagger.json (maybe adds a replace field, so the original reference can be preserved).

That would make tinkering with third-party modules easier.

Thoughts?

[sharp zealot]

Yes that has definitely been requested before... I agree a way to vendor and/or replace dependencies, would be very useful. How do you feel about creating an issue for it? 😇

[latent trellis]

Will do…I was on the way and wanted to write my thoughts down before I forget them

[sharp zealot]

Is there an end-to-end example of using interfaces anywhere?

The docs show how to define and receive an interface... But not how to implement and send one

@warped canyon @dense canyon I remember you guys helped me figure it out... I honestly can't get it to work again

You need 3 modules total, because the same module can't define the interface and send it?

[dense canyon]

https://github.com/marcosnils/dagger_iface_test

[sharp zealot]

sad face

(happy & grateful for the example.. just sad that it's so complex)

So 3 modules is the absolute minimum correct?

[dense canyon]

it should also work with two modules as well I think . I used 3 in my example because I wanted to separate the concerns of Platform, App and Pipeline.

[sharp zealot]

But if I only have 2, how do I get the .AsApp

[warped canyon]

I think 2 is what you need, I have another example

[dense canyon]

As long as you return a type that implements that interface, the AsApp will be generated automatically

[warped canyon]

https://github.com/kpenfound/dagger-modules/blob/main/secrets-provider/main.go

https://github.com/kpenfound/dagger-modules/blob/main/secrets-provider/providers/1password/main.go

[dense canyon]

so in my example, if the Platform module has a function that returns a type which implements App, then you'll get the AsApp helper function generated

[sharp zealot]

Yeah but isn't it generated in bindings for other modules? If the same module is implementing the concrete type, and passing it as argument, how does it call AsApp on its own type? Doesn't that require self-calls?

OK, so 2 modules is only if Platform module provides the interface + some possible implementations of it

[dense canyon]

yes, it's for other modules so that's why you need at least 2 modules to make it work

[dense canyon]

correct

[sharp zealot]

OK understood. Makes the 2-module scenario technically possible, but pointless.

(if the same module defines the interface and also its concrete implementation, the interface is just extra complication in the middle)

[warped canyon]

My implementation falls into the 2-module scenario doesn't it? or are you including another module that consumes the implementation?

[sharp zealot]

It's frustrating because I have the perfect use case for interfaces in front of me, it's elegant and will simplify a whole spaggheti bowl of code in our CI... but I can't use it

[latent trellis]

https://github.com/sagikazarmark/dagreleaser

No idea if it still works or not

It worked in march

[sharp zealot]

Looking at your code. I see:

1. Interface defined in github.com/kpenfound/dagger-modules/secrets-provider
2. Interface implemented in github.com/kpenfound/dagger-modules/secrets-provider/providers/1password
3. Interface passed as argument from .. ? Looking for that in your code github.com/kpenfound/dagger-modules/secrets-provider/example/go

[warped canyon]

https://github.com/kpenfound/dagger-modules/blob/main/secrets-provider/example/go/main.go#L32

[dense canyon]

@warped canyon in your example looks like you're not using the interface as arguments to functions IIUC

I think that's what Solomon is looking for

[sharp zealot]

But if he were, it would be from that third module

[warped canyon]

yes, that makes sense then

[dense canyon]

not necessarily Solomon, you can define a function in the secrets-provider module that uses that interface. same as I do here? https://github.com/marcosnils/dagger_iface_test/blob/main/ACMEPlatform/dagger/main.go#L19

I think (Erik will know better) the main limitation is that interfaces can be passed as arguments only in the module that defines them

[sharp zealot]

I think there are 2 main limitations... the first one is what you said

the second one is that a module that implements an interface cannot use it

I'm going to implement a workaround following @warped canyon 's pattern: a "standard library" of implementations.

In my case it doesn't really make sense, but it's better than losing the interface completely, so I'll create the "library" as a placeholder

will send the PR link in a minute so you understand the context (if you're interested)

thanks for the help guys

[sharp zealot]

@warped canyon @dense canyon any special tricks to make the implementation match the interface? Like the argument names have to be the same or something? No error or context allowed - or is it the opposite?

Oh damn it worked! It was indeed the argument names. They have to match

[warped canyon]

Yes! I have notes on this from when I wrote the code above. I need to find them to add to the documentation

[sharp zealot]

It makes sense when you think about how GraphQL works. It's just surprising from the POV of a Go developer comparing it to Go interfaces

[dense canyon]

yep, that's in the docs (https://docs.dagger.io/api/interfaces/)

[sharp zealot]

wording is not clear though. I'll open a quick pr tomorrow

[latent trellis]

https://github.com/dagger/dagger/issues/8876

[dense canyon]

💯 on this one. Would love to see this being prioritized

[restive shore]

Interesting thing I just came across. I have a function parameter failOnNonCompliant. The CLI doc shows the generated name is --fail-on-non-compliant. However, BOTH --fail-on-non-compliant and --failOnNonCompliant work in my case. Is that expected?

Totally found out by accident.

[dense canyon]

Yes, it's an easter-egg (?) we haven't addressed yet 😂

[restive shore]

Ok awesome! I don't think I mind if you didn't address it haha! But good to know I wasn't doing something I wasn't supposed to

[kind carbon]

It’s by design but better not depend on it 🙂

[restive shore]

Does the engine always try to pull the go toolchain? Or is it only when the go version in the dagger.json doens't match the engine go version? I'm getting CA cert errors when pulling the toolchain so trying to figure out if it's possible to coerce the engine to not pull the go toolchain.

[dense canyon]

hey Nipuna! IIRC you're using a custom URL for the Go module runtime in your dagger.json, correct?

[restive shore]

Hey Marcos! No I am not. My dagger.json looks like any other

[dense canyon]

gotcha. So the engine will download the toolchain if it doesn't have it cached. Seems like you've updated your engine version and when you tried running a pipeline it's trying to pull the toolchain again?

part of the engine build process embeds the specific runtime sdks as defined here: https://github.com/dagger/dagger/blob/e8db99875106f9d78873e7553be897b1d74e50bf/engine/distconsts/consts.go?plain=1#L28-L29

if you have a different version that the one specified there, the engine will have to then download build that specific runtime as it's not embedded in the engine by default

having said that, inbuilt SDK's don't support selecting a specific version as specified here: https://github.com/dagger/dagger/blob/e8db99875106f9d78873e7553be897b1d74e50bf/core/schema/sdk.go?plain=1#L128

so just wondering @restive shore which version you're refering to in the dagger.json 🤔

[restive shore]

Oooh!! My bad! I meant the version in go.mod. I've had differing versions between modules in a daggerverse repo. I was wondering if that matters when the engine decides which runtime version to use.

[dense canyon]

Nope, that's independent of the runtime

[restive shore]

Hmm.. ok this may be an issue for me because I'm doing a go mod tidy inside the container. Let me try to do a repro that I can share. My use case is linting my daggerverse. I'm doing the codegen using the dagger API and then before running golangci-lint I have to do a mod tidy or else the linter fails.

In either case, I should not be having CA certs issues but that seems to be a regression in the engine.

Sorry if this is kinda out of context. I have a thread here with more details about the CA certs issue. #1301297050624856136

[dense canyon]

this will have an effect at the module initialization level. Part of the module initialization is basically running go mod init and effectively a bunch of other stuff that will try to download the toolchain if needed. Having said that, are some of those go.mod versions higher than the current engine go version? since the standard GOTOOLCHAIN is set to auto in the SDK initialization phase (https://github.com/dagger/dagger/blob/e8db99875106f9d78873e7553be897b1d74e50bf/.dagger/build/sdk.go?plain=1#L136), it shouldn't have to download any toolchains since the go.mod of all the modules should be equal or lower than the current engine's go runtime version 🤔

I guess I need to understand better where and how you're currently doing that go mod tidy. Is the go runtime container that's actually trying to pull a different toolchain?

[restive shore]

That's a good point! I just checked and I'm running the go mod tidy within the golangci-lint (v1.61) container and that's on 1.23.1.

So makes sense that it's trying to pull the newer toolchain.

[dense canyon]

yep, it makes sense in this case and I also think I know why certs stopped working in the default go runtime image as well

@restive shore is there a golanci-lint image with 1.23.2 already?

[restive shore]

omg! please tell me it's an easy fix.. It's been eating at me for a while

[dense canyon]

that should unblock you in the meantime 🙏

[restive shore]

yes, and I just ran it and it didn't try to pull the newer toolchain

[dense canyon]

ok, so it wasn't the go runtime container all this time. It was your image of golangci-lint. 🙏

[restive shore]

Yes! I was using the official one and was a version behind. IDK why I concluded that the go mod tidy was running inside the engine runtime.

[dense canyon]

gotcha, np

[restive shore]

curious what the issue with cert propagation is

[dense canyon]

checking if my assumption is correct 🙏

[restive shore]

no worries! We can continue that discussion here - https://discord.com/channels/707636530424053791/1301297050624856136 when you are ready 🙂 I'm about to go to sleep anyways. GN and thanks for looking into it!

[dense canyon]

sure, np! have a good sleep, see you at kubecon!

[quick wind]

If I remove a repository (or subdir of a repository) I presume the Daggerverse removes that module at some point in the future?

[restive shore]

Is there any guidance on maintaining a private "daggerverse"? dagger publish won't work right? Are there plans to improve that? Now that PAT auth is working this is my next quest 🙂

[sharp zealot]

yes we have plans 😁

[restive shore]

Awesome!! In the meantime, my only option is to individually git tag the modules?

[sharp zealot]

yeah it's all works the same on private repos. Daggerverse just can't crawl and render them (yet)

[restive shore]

I am trying to figure out if I can come up with a CI process for the module "releases". I wonder if anyone else has come up with a Dagger module to do that. I'm thinking, on merge to main, it will have to detect if there were changes to a certain module (say module M) and increment patch and tag HEAD commit on main branch with M/v#.#.#. Just an idea. I'll also need some way to enable users to update to those new versions because dagger develop does not do it today.

[wintry prism]

Yep! Planned 🙏

[sharp zealot]

It might be easier to just not version them in that case. Effectively making main a continuous release, like a saas. Dependencies are always pinned to a commit so you don't lose any repeatability.

The git digest will carry as much information as the auto-incremented patch, with way less work

[restive shore]

That makes sense. That's similar to how we manage things with other shared libs

how do consumers get updates though? Do they have to dagger install every time?

There is a pin in the dagger.json which doesn't get updated unless I do a dagger install

[sharp zealot]

one thing we considered in the future, is an semver-compliant auto-versioning service that analyzes module API changes, and increases the patch, minor or major accordingly

[sharp zealot]

yes, at the moment that's how you do it. that way no surprises at run - dependencies are repeatable

[restive shore]

It's a bit tedious but I guess I could create make target to do bulk updates. It would be nice if dagger install supported a --all flag to update all deps in one go. If I manage a continuous release, that will become a use case.

[sharp zealot]

yes I think we need dagger update then no argument would mean "all"

[restive shore]

A question I know we'll have down the road is, "how do we force people update a certain dep if a critical vulnerability is detected". In the current model of users pulling in changes, that's not possible. We'll need a way to somehow blacklist a certain commit or force modules to get on a newer one.

[sharp zealot]

how do you handle it for eg. npm or go dependencies?

[restive shore]

Proxy via our own registry

[sharp zealot]

but Go dependencies are pinned. What happens when a Go project has a dependency digest in go.sum that you want to force update?

[restive shore]

good question, I can't go into more details here but happy to catch up on that later 🙂

[quick wind]

This feels like it wouldn't take much work? Is there somewhere I can 👍 this for priority?

[kind carbon]

https://github.com/dagger/dagger/pull/8839

[proven tusk]

Nice!! Took me a while to figure out dagger develop would update the whole thing.

[quick wind]

Is the process described here (https://docs.dagger.io/api/publish#publishing-on-use) typically instant? I've called a module from github with a new tag and it's not reflected

[ornate fern]

Is there a way to install the latest version of Dagger using Homebrew? I already have an older version installed, but running the brew reinstall dagger command didn't update it.

[spiral whale]

Is there a way to install the latest

[sharp zealot]

This is fun 🙂

[plucky ermine]

love this 🙂

[shadow notch]

This is fun 🙂

[latent trellis]

I had an idea about parameter validation and I opened an issue: https://github.com/dagger/dagger/issues/8981

[latent trellis]

This is a strange one:

Here is a dagger.json file:

{
  "name": "tests",
  "sdk": "go",
  "exclude": [
    "../../.direnv",
    "../../.devenv",
    "../../go.work",
    "../../go.work.sum"
  ],
  "dependencies": [
    {
      "name": "apko",
      "source": ".."
    }
  ],
  "source": ".",
  "engineVersion": "v0.12.0"
}

If I run dagger develop here (0.14.0), I get the following error

Error: dagger develop on a module without an SDK requires either --sdk or --source

Both sdk and source are set. If I remove or change the engine version, I still get the same. Any ideas why?

(Reproducible on all of my test modules here: https://github.com/sagikazarmark/daggerverse/)

(Doesn't happen with 0.13.1)

[deep hedge]

^^ I can take a look at this one.

[latent trellis]

I'm not sure if I'm doing something wrong or if it's a bug, hence I asked here first, but I can open a bug report if it's confirmed.

[deep hedge]

if it works with v0.13.1 and fails with v0.14.0, it could be due to https://github.com/dagger/dagger/pull/8557 (which was first released in v0.13.6).

[deft rain]

no, that just changes the default for init

shouldn't affect it if source is already set

i can repro this on @latent trellis's apko/tests

ohhh

i suspect this is https://github.com/dagger/dagger/pull/8575
apko has tests in it's exclude

i think this means that dagger.json for the child module is getting ommitted, so we actually don't see a dagger.json

[deep hedge]

wow, that was fast. thanks Justin.

[deft rain]

lemme quickly just confirm with a bisect

[deft rain]

yeah, first broken in v0.13.4, which is when that change was introduced

cc @kind carbon i suspect this isn't intentional? it seems like we're applying the dependency's excludes to the importer?

[deft rain]

opening an issue 🙏

https://github.com/dagger/dagger/issues/8984

[latent trellis]

That's very much on purpose, so dagger doesn't load any sources it shouldn't.

[deft rain]

yeah, your dagger.json definitely seems reasonable to me - i'm just a bit confused why we follow the exclude for a dependency when loading a module 🤔

cc @rocky harbor who's also been working on filesync things recently

[kind carbon]

It's intentional.

[deft rain]

so is there no way to exclude apko/tests when just importing the apko module?

[kind carbon]

Similar answer: #1298031968293879918 message

[deft rain]

When a module has dependencies, their include/exclude patterns are added to the module's own
^ this is the bit I don't understand

why do we do this instead of doing separate loads for each module?

i guess originally there's a potential performance penalty?

but i think https://github.com/dagger/dagger/pull/8818 should allay that

[kind carbon]

You'll have to ask @rocky harbor for that.

[kind carbon]

Yeah, I think that PR changes things, but not sure how it affects this particular case.

[deft rain]

I don't think it modifies this logic right now - that logic is in moduleSourceResolveFromCaller I think, and we haven't modified it

but i think we might be able to lift the restriction and have each module do it's own filesync - if the original version was performance that is

[sharp zealot]

If you are interested in how the dagger shell might impact the balance of convenience and portability in our UX: consider sharing your thoughts here: https://github.com/dagger/dagger/issues/8988

[latent trellis]

Where do I report issues for daggerverse?

Looks like default values are not generated for constructor functions.

Module: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/php@0af084118b214771286c548a63742219cf95d398#Php
Default value: https://github.com/sagikazarmark/daggerverse/blob/0af084118b214771286c548a63742219cf95d398/php/main.go#L39

[warped canyon]

Thanks, I'll have a look!

[bright beacon]

Hi All,
Any idea how can i load env variables in a dagger container
Im running codecheck using sonarCube in a self hosted dagger.
Here is how I am loading the env variables;

    let container = dag
      .container()
      .from('sonarsource/sonar-scanner-cli:11')
      .withDirectory('/src', config.source, {
        exclude: [
          '.dagger',
          '.nx',
          '**/node_modules',
          '**/dist',
          '**/coverage',
          '**/.reports',
        ],
      })
      .withWorkdir('/src')
      .withFile('.env.ci', config.envFile, { permissions: 0o744 })
      .withExec(['sh', '-c', '.', './.env.ci']);

And here is the error I'm getting in the gitlab ci log

38  :   [0.9s] | ./.env.ci: line 1: .: filename argument required
38  :   [0.9s] | .: usage: . filename [arguments]
35  :   Container.sync ERROR [0.9s]
35  :   ! process "sh -c . ./.env.ci" did not complete successfully: exit code: 2

I tried env loading with '.' and source, but both are not working.

[foggy stream]

Hey folks does anyone know how I can reduce the initial connect time? And why does using a from statement add 2s overhead if it's already pulled and here?

This is a consecutive run without changes.

[warped canyon]

Hey 2 questions

• what evironment are you calling dagger from in this case? e.g. mac + docker desktop?
• is there still a significant time on the .from if you specify a tag or digest?

[deft rain]

what's actually happening in the connect? can you expand that one a bit more

if you could share a link that would be helpful as well

[foggy stream]

Hey yes you assumed correctly it's my m1 mac + docker desktop.
In regards to the sha I've tried this but it seemed not to work with: base := client.Container().From("sha256:2d4e6732b03f221c2877d88f894284a67435d97b86477b4d3be7e34f6a633593")

Whereas docker run --rm -it sha256:2d4e6732b03f221c2877d88f894284a67435d97b86477b4d3be7e34f6a633593 works perfectly fine.

:latest is a tag? or what do you mean?

[foggy stream]

That's the breakdown on what happened:

I see that the "start engine" takes pretty long + "connect to engine" while this invocation was a consecutive one without any code change 😬

[deft rain]

sha256:xxx isn't a valid image reference - you need to do something like myimage@sha256:xxx (and make sure it's published to a registry somewhere)

docker run works for this, but yeah, this is slightly strange behavior in the first place imo 😄

huh yeah, connecting to engine really is taking that amount of time 🤔
start engine is expected to be quite slow the first time it's run, since it needs to pull and start the engine image - but on subsequent invocations, or when the target image is actually around it should be much faster

not really sure why connecting to engine takes so long though, does that line up with how slow it generally is @south helm @plucky ermine @warped canyon (as people who've talked about slow macos perf recently)

[foggy stream]

@deft rain If I try to run a local built image it does not work, is there something I miss?

Like I've built this image with: docker build . -t dagger-base

=> => writing image sha256:7e71c2102a4c4a938669b59750eaab2ef8be867353a849b0f546d340766545b2                                      => => naming to docker.io/library/dagger-base

Running it with docker works fine, but with dagger it fails: client.Container().From("dagger-base@sha256:7e71c2102a4c4a938669b59750eaab2ef8be867353a849b0f546d340766545b2")

Does not work with simple container tag as well From("dagger-base:latest")

[deft rain]

yeah, this currently isn't supported, you're right. there's a similar issue for this around integrating into the local container store: https://github.com/dagger/dagger/issues/8025

[warped canyon]

This is fixed now 👍

[warped canyon]

yeah what I was meaning here was to use something like ubuntu@sha256:278628f08d4979fb9af9ead44277dbc9c92c2465922310916ad0c46ec9999295 instead of ubuntu:latest because afaik :latest will be looked up every time whether it's been pulled or not, even with docker run

[drowsy basin]

After a long pause (due to personal issues and huge professional work load) I'm back on the modularization of the SDK in java. However I don't really know how to start. I looked at other SDKs for inspiration (in particular Typescript and Elixir) and I started to write the runtime module which from what I saw should expose 2 functions: moduleRuntime and codegen. I'm not sure what these functions should return. So if you have any tips or guidelines I'm interested 😅

[kind carbon]

Sure, I can guide you on that, but let's continue on https://discord.com/channels/707636530424053791/1310914793804402759. 🙂

[sharp zealot]

Yay @drowsy basin !

[alpine rampart]

Hi I m new here I search how I can run dagger with podman I get allways this message : ModuleSource.resolveFromCaller: ModuleSource! 0.9s
! failed to collect local module source deps: failed to load sdk: failed to load embedded sdk module "typescript": select: failed to create module from directory: select: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "codegen --output /src --module-context-path /src --module-name typescript-sdk --introspection-json-path /schema.json" did not complete successfully: exit code: 1

[sharp zealot]

Hello! Are you able to share a trace URL? You can set up shared tracing with http://dagger.cloud/traces/setup

[alpine rampart]

Ty a lot @sharp zealot you find there a print sreen of log

this the URL but I cant find how I can share this with others https://dagger.cloud/ilyhel/traces/5e0ccf2f530127fbcc7a65e10ee894fb

[alpine rampart]

this the URL but I cant find how I can share this with others https://dagger.cloud/ilyhel/traces/5e0ccf2f530127fbcc7a65e10ee894fb

[sharp zealot]

at the moment you can only share with other members of your org, or members of the support org at Dagger - which I am part of 🙂 let me take a look

[dense canyon]

@alpine rampart this is probably related: #1306706089643085874 message

[alpine rampart]

[alpine rampart]

yup the key is the --rootful should be true so I create a new podman machine and I enable Machine with root privileges 😉 ty a lot

[dense canyon]

Awesome! Glad that you managed to get it working

[robust stump]

Hello

Juste re-use dagger from long time ago, discovered all news stuff like modules/functions/services
I have a question on a use case
We have restriction on having docker loccally on our machines, so we think about dagger to remote some usage of container stuff
On php development with a local docker env, developpers mount local code in container on the webserver via binding volume
Any change on local FS is instantly propagated as it's the same data, so refresh browser see changes immediatly

With dagger, i can build this container remotely, and launch it with a service
And with up command, it port-forward the wevserver on my local machine, so i can browse

Is there a way to live sync my FS on remote running service with dagger, or schedule a sync every x sec for example ?

in our context, we have a local dagger cli, and a remote dagger-engine via tcp connection

[hazy turret]

Not yet. But seems they are making progress on it. The relevant issue to track and leave your feedback/support/request:
https://github.com/dagger/dagger/issues/6990

[robust stump]

the function for our apache service is like that for example :

func (m *Module) ApacheService(
    ctx context.Context,
    src *dagger.Directory,
) *dagger.Service {
    return m.Build(ctx, src).
        WithServiceBinding("mysql", m.MysqlService(ctx, src)).
        WithEnvVariable("DATABASE_SERVERNAME", "xxxxx").
        WithEnvVariable("DATABASE_PORT", "3306").
        WithEnvVariable("DATABASE_NAME", "xxxx").
        WithEnvVariable("DATABASE_USERNAME", "xxxx").
        WithEnvVariable("DATABASE_PASSWORD", "xxxx").
        WithDirectory("/app", src).
        WithExposedPort(8081).
        AsService()
}

The idea is to live sync the /app directory from src

oh thanx for link
don't find it when searching

[foggy stream]

Hey folks is it possible to run dagger inside a Docker image? (kind of a Docker in Docker appraoch)
I'm using a golang binary where I'm using dagger programmatically and connecting to it like that:

client, err := dagger.Connect(ctx)

I wonder If I can integrate it easily into our current stack by exporting and distributing that as docker image (where dagger is installed to)

[sharp zealot]

yes it's possible but it needs to be run as a privileged container, since dagger itself is a container runtime.

[foggy stream]

Thanks is there somewhere an example?

[sharp zealot]

Well the engine is always packaged as a container, it happens automatically every time you use the dagger CLI (it will download & run the engine container if it doesn't find one already running).

The unusual part is to bundle your tool along with it. Definitely possible but deserves some thought in how to do it best.

Actually if you build your tool into an image with the dagger and docker CLIs pre-installed and functional (eg. docker run works within the container) then you're good. It should all work when you run the container.

[sour citrus]

👋 I'm kind of stuck while creating modules in ruby. I guess I'm missing something obvious, but I could not find it...
This is based on this branch: https://github.com/eunomie/dagger/tree/sdk-ruby
I'm creating a ruby module (for instance dagger init --sdk=ruby modules/rubymod)
Then trying to call it (not everything is implemented, I'm just trying to wire everything first, then make it generic)
cd modules/rubymod; dagger call
The code is doing a dag.current_function_call.return_value(value: register.to_json) with register the module definition.
But I'm always getting a failed to initialize module: failed to call module "rubymod" to get functions: failed to read function output file: open output.json: no such file or directory
Any idea what I am missing?
(if that can help, traces are here: https://dagger.cloud/eunomie/traces/c6efcd3893ca16641627e5fed02c4145)

[kind carbon]

Ruby SDK

[warped canyon]

@mossy hazel in your demo, is there a way for the module to know if the engine has GPU enabled? basically to make the --with-gpu flag optional?

[sharp zealot]

I really really would prefer for us to just get GPU support enabled in all engine builds; before it becomes an entrenched best practice to check for which kind of build you're using. I'm scared of fragmentation. "This modules works only if bla bla bla" we had this problem with Docker and it was a PITA

[warped canyon]

100%, what I'm thinking is more like "is there a GPU available" vs "is this a specific flavor of engine"

[sharp zealot]

oh right! 👍

[mossy hazel]

I think there is a way yeah. I’ll add this to my improvement list.

[mossy hazel]

I don't think this is doable in the current state. Our base image is currently better with alpine or wolfi-base, with very few packages. Nvidia drivers require a bunch of cuda-related packages that are heavy, that are packaged for very few distros and are super heavy. AMD GPUs will need different packages, etc...

[agile saddle]

Yes, but there are APIs that allow OCI compatible engines to pull in the appropriate libraries from the host. Nvidia provides a mechanism called the nvidia-container-runtime which provides callbacks that the container engine uses to find these. For Intel and AMD, its a specific folder and set of devices files in /dev. If you also choose to map in the driver libraries, it often requires a glibc based image, but other than that its doable.

There is a open standard called the https://github.com/cncf-tags/container-device-interface which supports pulling these in.

[mossy hazel]

Yes, it was we use in the engine: install the right packages and invoke the hook when enabling the GPU support. Maybe we could scan /dev and automatically switch on the GPU support for Nvidia when we find one. However it would mean pulling the fat engine image whenever dagger sees an nvidia gpu whether you're using it or not. Some users may complain about that, I don't know.

But I think I found a clean workaround given we can't introspect the host for GPU support: https://github.com/samalba/dagger-modules/blob/7273f9528f91021cf0e41766349e1f3f9f5a90d7/nvidia-gpu/main.go#L78 - cc @warped canyon 🙂

[warped canyon]

nice!!

[viral pike]

Hi there, anyone with an opinion on which is the best way to periodically test all my daggerverse modules (e.g. with any new dagger version)?

[latent trellis]

Here is what I do: https://github.com/sagikazarmark/daggerverse/blob/main/.github/workflows/nightly.yaml

Basically: run a cron workflow with the latest dagger nightly build

[viral pike]

Cool! Thanks!

[sharp zealot]

Request for comments: https://github.com/dagger/dagger/issues/8235

[latent trellis]

Quick question:

If I want to make sure that an environment variable is unset (that I normally mount as a secret variable), is calling WithoutSecretVariable enough?

Because it doesn't seem to be:

dagger core container with-env-variable --name FOO --value bar without-secret-variable --name FOO env-variable --name FOO

Output: bar

(Haven't tried it the other way around)

It's a bit confusing that I have to call both (WithoutEnvVariable, WithoutSecretVariable), if I want to make sure an env var is unset.

[dense canyon]

@latent trellis but you're using with-env-variable and without-secret-variable in this case? You should be using with-secret-variable if you want to unset it 🤔

[latent trellis]

When I set the variable, I set it with with-secret-variable. When I unset it, I want to make sure that it's unset (regardless if I set it as a secret, or someone else set it manually in a container that I accept as an input).

(I guess that list bit was the missing information)

Most of my modules accept an arbitrary container to make sure users have a certain degree of control, but there are things that I want to be able to control in the module.

If I don't know if an env var was ever set (and how it was set), I need to call both without methods to make sure it's really unset.

[dense canyon]

ok, I see.. I do think it's nice to make WithoutSecretVariable to both unset the secret and the env var. WDYT @latent trellis @sharp zealot @kind carbon

[latent trellis]

I would probably deprecate WithoutSecretVariable in favor of (or make it an alias for) WithoutEnvVariable.

In my mind, WithSecretVariable is just a special version of WithEnvVariable (even if in reality it's not the case, the end result is similar)

[young schooner]

Issue DOCS-382 created.

DOCS-382 - document how to test modules with new versions

see discord link

Status

Triage

Assignee

Vikram Vaswani

Documentation · Dec 12, 2024, 9:03 PM

[warped canyon]

might be missing something with the 0.14.0->0.15.x upgrade. The following command errors in 0.15.x:

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service up

error:

"root" execution of the PostgreSQL server is not p
mitted.
The server must be started under an unprivileged u
r ID to prevent
possible system security compromise.  See the docu
ntation for
more information on how to properly start the serv
.

[wintry prism]

need --use-entrypoint:

dagger core container from --address postgres with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres as-service --use-entrypoint up

[warped canyon]

Hm ok. That works for my core repro but not my code 😭 back to debugging

[wintry prism]

can you show code?

[warped canyon]

Actually I had missed one of the services 🤦‍♂️

[manic lynx]

Hey! I created a dagger module named dagger-phpstan to give me the dagger call phpstan syntax ..

The plan is to add this dagger module onto my project' source code, so that I can do something like dagger call phpstan --source=. and immediately have the CLI functionality of this module exposed to my project's dagger functions list .. Therefore I can start using this from Jenkins/GitHub CI

Example scenario is

dagger init --sdk=php .
dagger install github.com/dragoonis/dagger-phpstan@30d7518bdf42c238940790110c6169cbb7d3dcc9
dagger call phpstan --source=. --php-version=8.4 --path-to-test=src

However, the issue is .. that my external module I installed doesn't expose any of it's CLI functions onto my app's CLI functions .. so now I'm a bit stuck?

Do I have to write glue code, so I need a local @func public function phpstan code on my local module that proxies to the dagger-phpstan module?

I'm expecting CLI goodies, but I didn't get any at all, and now a bit stumped on this ..

Thoughts? 🙂

[wintry prism]

@sharp zealot perhaps this is a limitation of the standard dagger CLI, eh? I had the same feeling in the past where I expected to install a module and "extend" my API right away. Guess that's what dagger shell delivers using module namespacing to avoid function name collisions.

[manic lynx]

Next - Dagger shell has "positional args" but when puttin stuff inside developer README documentation, it's really useful (and self documenting) to have named arguments with -- so that the developers understand what the values mean against which args

This doesn't work, it says "invalid flag"

dagger shell -c 'github.com/dragoonis/dagger-phpstan | phpstan --php-version 8.1 --source https://github.com/dragoonis/Sylius\#2.0 --path-to-test src'

hope this makes sense? would love to have standard CLI args with names, rather than positional-only flags

[sharp zealot]

we discussed that problem with Helder, unfortunately I believe it's all or nothing: either we have some positional args that are not available as flags, or we have no positional arguments. Given that choice, positional arguments are too valuable (in terms of keystrokes saved and command readability) to give up. If we somehow found a way to get the best of both, we would consider it for sure

[manic lynx]

understood .. it is a nice-to-have, and we will have to just solve the problem with additional README docs, to explain to developers what the heck these random characters mean 🙂

i.e:

8.1 . src

alernative is to do dagger -m and then we can specify the args with self-documented names --php-version=x --source=. --path-to-test=x

[sharp zealot]

Well there is .doc FOO to get full documentation on FOO

[manic lynx]

dagger shell is magic - neo himself entered the matrix and emerged with this cool syntax

[sharp zealot]

yeah, in the shell a modules dependencies are exposed as top-level commands in that module's scope.

So you from your module you can do phpstan --source=bla

[manic lynx]

Are there performance "slowdown" from using dagger shell -c versus dagger call -m ... ??

important to know, because developer's first experience of Dagger should be as fast as possible .. so they love it

[wintry prism]

what do you mean by last line?

Mixing dagger call and dagger shell style 🙂

yeah, in dagger shell we had no issue accessing deps.
in module foo we just called the dep and its function (both called phpstan, so a little confusing):
phpstan | phpstan 8.1 . src

But yeah, we'd like to have that phpstan | phpstan --source=. option, if it were possible

[manic lynx]

@wintry prism is daggerverse still broken (not syncing?) it's still not picking up my repos latest changes .. as it's still not appearing in the search box

https://daggerverse.dev/mod/github.com/dragoonis/dagger-phpstan@a3aeb30da60abd4a8662c7ba0efdc6a1b0d6c6d2

I changed my module function name from phpstan to analyze

[wintry prism]

I just merged a fix. It's deploying.

[manic lynx]

1. what is the TTL on the daggerverse UI for parsing repos and finding new changes? My repo has no webhook to Daggerverse, so daggerverse must be crawling it ?

2. any way for me to give it a kick? so daggerverse initiates a re-parse?

[wintry prism]

Should happen automatically when you call a module in a public repo. Let me know if you can see
https://daggerverse.dev/search?author=dragoonis

[manic lynx]

I'm looking for the new commit of 4310ae23ba242f78626d59c8d519fde9fc1d6631

for this change

https://github.com/dragoonis/dagger-phpstan/commit/4310ae23ba242f78626d59c8d519fde9fc1d6631#diff-95e0401c0daf96bb7efca3991c590e46a495a8c2f8bd2e6d46262d2220922218R23

UPDATE:
Commit made 11 mins ago - it's not picked it up

If I need to wait - that's okay - just wondering

a) what is the TTL for daggerverse to pick up my src changes on github

b) can I give it a kick? as I need to test this module on consumer projects with dagger install <ref> stuff

dagger install <new ref> fails at the moment (12 minutes since push)

[wintry prism]

It only gets indexed when called/installed via its public git url (e.g. dagger call) OR I just ran dagger publish from a clone of the repo to "kick" it 🙂

Hoping that works 🤞

Seeing it: https://daggerverse.dev/mod/github.com/dragoonis/dagger-phpstan@2478fd5672a018db14a3299abeb03effab56d138#Phpstan.analyze

[manic lynx]

OOOOH! I didn't know about dagger publish

This is how you kick it 🦵

TYVM Jeremy, i now have a "dev workflow"

@wintry prism if I already have dagger install done in my webapp repo .. how do I upgrade the current version of the dagger module to the latest one ?

like dagger update <module-name> ? or something

[wintry prism]

did you try to just dagger install it again?

without a sha

[manic lynx]

nope, but I can do that now and see

@wintry prism yes! I can confirm that dagger install <module name> does indeed bump the version to the latest commit hash - awesome, thanks for teaching me. I now understand the full dev lifecycle of module development/upgrading 🙂

Off I go to build more stuff🏃‍♂️

[sharp zealot]

maybe those should be optional arguments?

seems to me you can boil down those 3 required args to 1

[leaden pagoda]

does dagger call -m github.com/your-module phpstan --source=. ... not meet what you need then? i.e you need to have the module, but also some other things too, that may be niche or minor additions?

[leaden pagoda]

has there been any discussions around the DX of installing modules? dagger install [module] is what we have right now, but with the concept of packages, uninstalling and updating is also very common

• dagger module install
• dagger module uninstall
• dagger module update (as an alias for install, if it hooks into the same method)

the hash can sometimes be a little hard to follow too in the dagger.json, modules have a release but then we end up with the module git hash, so not always clear about "oh i have the tagged version 0.1.4", as an example.

[deft rain]

uninstall should already exist 👀

update is in progress: https://github.com/dagger/dagger/pull/8839

[leaden pagoda]

oops yeah, sorry i see this in the v15 release.

is grouping under a module something that could help group that functionality? maybe it wont be as visible though so not sure what others think of that idea, a bit like how we have dagger core

[obtuse glacier]

Is dagger shell the recommended way to use dagger right now?

[wintry prism]

It's new, experimental, and under development, but very fun and powerful.
It uses the same modules/functions as dagger call but is a bit less stable of an interface since under dev.

[sharp zealot]

yeah it will be the recommended way. For now it's unfinished like Jeremy said

but solid enough to be worth experimenting with, if you want to see where our UX is headed

[obtuse glacier]

How does something like https://github.com/shykes/gha translate to dagger shell with positional args?

[obtuse glacier]

Is there an escape hatch for accessing host info within dagger modules?

One example would be passing through all of the CI environment variables to the module, without needing topass them all as env vars.

[sharp zealot]

We're very cautious about that because we don't want to break the sandboxing which is key to repeatability and a scalable composition model. But we're working on adding convenience, just being cautious

[foggy stream]

Hey folks has someone exprience in running the dagger engine in k8s with gvisor?

I'm running into some issues with gvisor on minikube. Running without gvisor works perfectly.

exec /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml "$@"
time="2024-12-19T13:30:56Z" level=info msg="detected mtu 1500 via interface eth0"
time="2024-12-19T13:30:56Z" level=debug msg="engine name: dagger-engine-767fcb6bcc-v7nc9"
time="2024-12-19T13:30:56Z" level=debug msg="creating engine GRPC server"
time="2024-12-19T13:30:56Z" level=debug msg="creating engine lockfile"
time="2024-12-19T13:30:56Z" level=debug msg="creating engine server"
time="2024-12-19T13:30:57Z" level=debug msg="auto snapshotter: overlayfs is not available for /var/lib/dagger/worker/snapshots, trying fuse-overlayfs: failed to mount overlay: invalid argument"
time="2024-12-19T13:30:57Z" level=debug msg="auto snapshotter: fuse-overlayfs is not available for /var/lib/dagger/worker/snapshots, falling back to native: fuse-overlayfs not installed: exec: \"fuse-overlayfs\": executable file not found in $PATH"
time="2024-12-19T13:30:57Z" level=info msg="auto snapshotter: using native"

dnsmasq: cannot create netlink socket: Permission denied
dnsmasq exited: exit status 5
time="2024-12-19T13:30:57Z" level=warning msg="failed to release network namespace \"fwmzqjsw2j47cl30yev2cymiv\" left over from previous run: plugin type=\"loopback\" failed (delete): unknown FS magic on \"/var/lib/dagger/net/cni/fwmzqjsw2j47cl30yev2cymiv\": 1021997"
buildkitd: failed to create engine: failed to create network providers: CNI setup error: plugin type="bridge" failed (add): failed to create bridge "dagger0": "dagger0" already exists but is not a bridge

[strong ingot]

Still working on this one. cc @opal zephyr

[obtuse glacier]

Is there a pattern for error handing in pipelines? For example on failed tests, I still want to upload results and print stdout before throwing the error.

[warped canyon]

There is! Here's a basic example from the cookbook https://docs.dagger.io/cookbook#continue-using-a-container-after-command-execution-fails
does something like that work for your case?

[obtuse glacier]

Yeah, basic idea, but in a long pipeline with lots of potential errors

[edgy cape]

Hello there! I'm working to integrate the shell into Runme.dev and stumbled over some differences in stdout handling Shell <> CLI. This currently breaks the existing UI/UX integration. I believe I fixed the issue in this PR, but I could use some help to make sure I'm not causing side effects. I believe @deft rain, @kind carbon, & @sharp zealot have touched surrounding code. Help, please. https://github.com/dagger/dagger/pull/9258

[sharp zealot]

Is it possible for a function to export a directory to its local filesystem, then re-import it?

func (m Mod) RoundTrip(ctx context.Context, in *dagger.Directory) (out *dagger.Directory, err error) {
  if err := in.Export(ctx, "."); err != nil {
    return nil, err
  }
  return dag.CurrentModule().Workdir("."), nil
}

Is this 👆 supposed to work?

EDIT: it works 🥳

[sonic vessel]

What is the benefit of doing this? Just curious.

[leaden pagoda]

im also curious, is this 'cheaper' compute, than creating another tmp directroy and placing the contents there?

unrelated, has anyone wanted something for default secrets? I cant find anything in the docs, but i was looking for something like

// +default=(e.g env:MY_VARIABLE | cmd:op /grab/from/1password/secret | file:./secret.txt)
mySecret dagger.Secret

dager call do-my-thing // didn't need to specify the secret, since i have +default=

The example docs only show an example working with a plain string type, not a secret. I thought i'd give it a go but i couldnt get it to work.

[restive shore]

Trivvy scan failed (HIGH severity) on the latest dagger engine image 0.15.1. Dagger's CI should run this (or similar) scan before producing the image. Interestingly, the reason I'm doing this scan is because Dagger enables me to do it so easily on my local. So I added it to my docker image build+publishing workflow.

[strong ingot]

We do: https://github.com/dagger/dagger/actions/runs/12411261036/job/34648580631

The issue has been fixed: https://github.com/dagger/dagger/pull/9253

Fix will be out as soon as we ship the new release. No ETA just yet.

[sharp zealot]

Here's a fun one: execute commands on the host from the dagger shell 😛

sh() { container | from alpine | with-mounted-secret /output cmd:"$@" | with-exec cp /output /output2 | without-mount /output | file /output2 | contents; }

sh "pwd"
sh "ls -l"
sh "touch ~/hello.txt"
sh "open http://dagger.io"

[sharp zealot]

@sonic vessel @leaden pagoda I think no benefit, it was to work around an AI-specific issue (how to give files to the agent and receive files back) but I found a simpler way 🙂

[sharp zealot]

Default secrets are not yet supported, we're looking for the best way to design them without breaking repeatability

[leaden pagoda]

👍 i came across the secrets provider PR and was having a look through there

[sharp zealot]

at the moment that PR doesn't address defaults, but it would be a good opportunity to at least agree on their design.

[sharp zealot]

One approach I find promising is a .env-style config file, to keep environment-specific config out of the source code, without having to manually re-type it every time either

[obtuse glacier]

I'm trying to migrate some CI pipelines to Dagger, and I'm trying to determine the best architectural approach.

Currently, I see two options:

1. Create separate Dagger plugins for each task, running them as individual CI steps.

2. Use a single step with chained functions.

The first approach feels like it might not be utilizing Dagger's capabilities fully, while the second approach loses the clear logging and visual separation between steps that we currently have.

I'm considering a hybrid approach: using chained functions but exporting a bash file for each "step" with logs and error codes, then having CI run these scripts to maintain visual separation.

Is this a reasonable approach, or am I missing better alternatives that Dagger provides?

[strong ingot]

The Dagger alternative would be to connect to Dagger Cloud - https://docs.dagger.io/configuration/cloud/ - and then benefit from views (including logs) like this one:

[restive shore]

I am having trouble getting CA certs to work. I remember having a discussion with @dense canyon (and he seemed to know be on the trail of an issue) but I can't find that thread here. With 0.15.1 I can't run any dagger commands without getting these.

go: downloading google.golang.org/genproto v0.0.0-20220822174746-9e6da59bd2fc
/go/pkg/mod/google.golang.org/grpc@v1.68.0/status/status.go:35:2: google.golang.org/genproto@v0.0.0-20220822174746-9e6da59bd2fc: Get "https://storage.googleapis.com/proxy-golang-org-prod/3837
3f37be406883-google.golang.org:genproto-v0.0.0-20220822174746-9e6da59bd2fc.zip?Expires=1735077265&GoogleAccessId=gcs-urlsigner-prod%40golang-modproxy.iam.gserviceaccount.com&Signature=FqdeCwo
qZFzGNV%2BbHOOPGD374pzUoGczU%2Fx%2Bt%2BmiJ8GvISBqYqFvVWJ2Xj8EA5uzFHVAPkmclcxG2sMhoFUo3MOXc%2Fxjamnqqc3Rh%2BieInlEQ9BiiOiMy16bi3tBmE4fQdiya0n63t%2BvS2y1GO9w4mUXzt8Ymffaic8JO2Six6X8ZTIWOo%2FUdL
fTaLwkrwFAAZHtt%2FbppjMAhDeLSxIryw0%2F4%2BVqM7qx3GWVcNnH%2BuwMxxod%2Fikox3tEmk3%2Ftmqn1LH4WEfI4vy1nckmZQSRjYxL9fmzkPTXPLyIuoj7lBMuG5s6c5YQC0iY3YGQNtBQk3C9xn61RSrO2S1o1WwA7g%3D%3D": tls: faile
d to verify certificate: x509: certificate signed by unknown authority

CA certs are set up properly inside the engine image.

[sharp zealot]

👋 @thorn moat I'd like to discuss possible variations of the trace views, to optimize for different use cases.

• For my agent experiments I'm relying more on custom spans. For this use case the default shows way too much, it's like dumping debug logs in the middle of my own UI, with no way to turn it off
• The topic of a "frontend API" for Dagger keeps coming up. Simple questions like "how do I print this information to the user" has no simple answer. This is not strictly a tracing topic, but in practice, the design of our tracing system (and its influence on our UI) is a gateway to any other design in that area. Should custom spans be the official means of printing things to the user? If so, how do we 1) make them more visible, and 2) make them easier to develop against?

Also, do you happen to know what these plaintext spans are next to my custom spans? I seem to be triggering them but I have no idea why, or how to fix it

[thorn moat]

frontend API

[thorn moat]

guessing this is coming from a Secret.plaintext call being made on each API call for the OpenAI token. Not sure why it looks like a "naked" plaintext call with no parent/receiver, will look into that. Also seems like it should be beneath the other spans - possibly a code error there with ctx propagation?

[sharp zealot]

Oh I see. Yeah the OpenAI chat completion API (or at least the way I use it) doesn't map perfectly to the hierarchical nature of the otel API.. I'll try to make it fit more cleanly

[sharp zealot]

The problem comes from the fact that their API is 100% stateless, so as the state of the session builds up (client prompts, AI replies, tool calls, tool outputs), the client re-sends the whole history each time in a single API request that gets bigger at each round trip. Then the API response is processed, which leads to 🤖📖 (robot looks up knowledge) or 🤖💻 (robot runs a command). Then the next API round trip, etc

So I guess there's a missing custom span, "state sync" or "API round trip", and the plaintext would be a child of that

This is a good example of #1320920620007034953

[restive shore]

CA cert setup seems to be required for 0.15.1

[sonic vessel]

What is the best way to run dagger inside a dagger module? My current solution is start docker as service and run dagger binary through that service, it works but want to know it has a better way to do it.

[sonic vessel]

Found another way is to start dagger as a service:

defp with_dagger(container) do
    engine_toml = """
    insecure-entitlements = ["security.insecure"]

    [grpc]
    address = ["unix:///var/run/buildkit/buildkitd.sock", "tcp://0.0.0.0:1234"]
    """

    dagger =
      dag()
      |> Dagger.Client.container()
      |> Dagger.Container.from("registry.dagger.io/engine:v0.15.1")

    dagger_cli =
      dagger
      |> Dagger.Container.file("/usr/local/bin/dagger")

    dagger_engine =
      dagger
      |> Dagger.Container.with_new_file("/etc/dagger/engine.toml", engine_toml)
      |> Dagger.Container.with_exposed_port(1234, protocol: Dagger.NetworkProtocol.tcp())
      |> Dagger.Container.as_service(
        use_entrypoint: true,
        experimental_privileged_nesting: true,
        insecure_root_capabilities: true
      )

    container
    |> Dagger.Container.with_env_variable(
      "_EXPERIMENTAL_DAGGER_RUNNER_HOST",
      "tcp://dagger-engine:1234"
    )
    |> Dagger.Container.with_service_binding("dagger-engine", dagger_engine)
    |> Dagger.Container.with_file("/usr/local/bin/dagger", dagger_cli)
end

[sharp zealot]

@sonic vessel it depends what you're trying to achieve. The most lightweight is to run the dagger cli in a container, with nesting enabled:

1. install the module github.com/dagger/dagger/cmd/dagger

2. call that module's binary() function to get a build of the dagger cli. Copy that into a container

3. Call the dagger cli with withExec(experimentalPrivilegedNesting: true)

[sonic vessel]

it depends what you're trying to achieve

I want to create code generator module that create a dagger module with dependencies setup automatically. Your solution looks very interesting!

1. install the module github.com/dagger/dagger/cmd/dagger

Is that github.com/dagger/dagger?

[sharp zealot]

cmd/dagger is a smaller module with only the cli

[plucky ermine]

Dagger shell is a game changer for debugging these sorts of issues ❤️

https://github.com/dagger/dagger/issues/9263#issuecomment-2563140462

[obtuse glacier]

Would it make sense to expose WithContainerFunc, WithDirectoryFunc etc... to dagger shell?

[plucky ermine]

Hey! Can you share any more details, I am not sure I understand your question here.

[obtuse glacier]

Would love to be able to do something like this:

with_build() { $1 | with-exec "go","build"; }

container 
  | from golang:1.21
  | with with_build()

[obtuse glacier]

Also I don't think its currently possible to share With*Func's, but it would be really awesome for modules to be able to export them, I think it would be more flexible than using custom types for a lot of use cases.

[sharp zealot]

@obtuse glacier thanks to bash functions being late binding, and our bash parser being blissfully unaware of dagger, you can actually do:

with_build() { with-exec go build;}

container |
  from golang:1.21 |
  with_build

Jury is still out on whether that's a good thing or not 🙂

[latent trellis]

That feels a bit like core type extensions which were removed from language SDKs early on.

[sharp zealot]

yes but with no typing whatsoever 😁

it's almost like a macro

[leaden pagoda]

{
          "type": "command",
          "style": "powerline",
          "foreground": "#c2c0d2",
          "background": "#5d21df",
          "properties": {
            "shell": "powershell",
            "command": "dagger -m https://github.com/pjmagee/dagger-openweatherapi call --api-key=env:WEATHER_API_KEY use-location --name=London,UK with-fields --fields temp,unit as-string",
            "interpret": false
          },         
          "powerline_symbol": "\ue0b0",
          "leading_diamond": "\ue0b6",
          "trailing_diamond": "\ue0b4",
          "cache": {
            "duration": "30m",
            "strategy": "session"
          }
        }

[obtuse glacier]

Awesome Thanks! Is there a way to import shared/common shell functions?

[robust hare]

I'm developing an SDK module. I started on a fairly old version of Dagger and am working on catching up during the holidays. In v9, we had root-for which wasn't very intuitive so I understand why it was removed, but it was essential for me to be able to use module /foo/runtime as the SDK for module /foo/example by placing a dagger.json directly in /foo that is root-for the example dir, causing example's context to include the SDK.

Now, ideally that wouldn't be required, but it was because by the time the engine gets to the point where it reads the SDK value from example's dagger.json, it is no longer in a state where it considers anything as being relative to the host filesystem, and expects to find any relative module/sdk dependencies within the currently considered module's own filesystem.

The documentation currently doesn't say anything about inter-module dependencies within a monorepo, and (as far as I could find) doesn't clarify the relationship between the context, source root, and source directories of a module. Given that one of Dagger's main selling points is eliminating "push & pray", I expect there must be some better option than having to publish a module to GitHub in order to use it from another module without having to nest the former within the latter on my filesystem.

[sharp zealot]

👋 welcome back 😁

A few quick notes that hopefully will help unblock you until deeper experts come along:

• Modules can import other modules in the same repo. We generally encourage this and have made many optimizations for dagger to work well in monorepos. If there are gaps we will fill them.

• I believe the same applies to the sdk field, since it's allowed to reference any module as sdk. that should logically include local modules.

• a module's root directory is where the dagger.json is

• a module's source directory is where the source field in dagger.json points to

• a module's context is its enclosing git repository. If no git repository is found, it defaults to the module root

[sharp zealot]

Not at the moment; the recommended mechanism for sharing functions is with a module, to preserve a unified cross-language ecosystem. There's always the option of a bash sdk which converts bash functions as actual dagger functions 😁

What are your thoughts on this?

[robust hare]

Thanks, just having the git repo as the context in all cases is intuitive. Why is it that CurrentModule only gives access to the source directory, though? For any other module, the ContextDirectory is easily accessible. But the object returned by dag.CurrentModule() hides it.

[sharp zealot]

For any other module, the ContextDirectory is easily accessible.

Do you mean via default directory arguments eg. +defaultPath in Go? Or some other way?

[robust hare]

e.g. dag.ModuleSource("myref").ContextDirectory()

Asking because my runtime module needs

• codegen
• SDK project
• invoker / introspector program
• go module

It would be nice to be able to lay them all out side by side. But because the go module which needs to dag.Container().From("blah").WithMountedDirectory("bin", mycodegendir).WithExec([gen the code]) and the only way it can get mycodegendir is either publish/version/download the codegen tool separately or nest it (and the sdk and invoker) underneath the go module source so that mycodegendir = dag.CurrentModule().Source().Directory("codegen").

[sharp zealot]

Ah this is in the context of developing a custom SDK right? Then I believe accessing the whole context directory for the module you're building/codegenerating for, is not an issue.

You're right that a module can't just read its context directory directly. It has to receive a directory argument and set the argument's default path relative to the context dir, eg. /foo/bar. This is not relevant for making a SDK though.

I don't have enough experience with the SDK development process to help you with the specific invocation sorry. I just know the design allows for it 🙂

have you checked how other SDKs do it?

[robust hare]

accessing the whole context directory for the module you're building/codegenerating for, is not in issue
Yes, the ModuleRuntime and Codegen functions are both given modSource of the subject module.

The other SDKs are either built-in, or clearly outdated and don't work against the current engine (e.g. have "root" property in their dagger.json). Built-in modules get the built-in codegen passed to them through the SDKSourceDir constructor parameter.

[sharp zealot]

ah I think I understand your issue. Your sdk module needs content from other places in its repo?

in that case you can 1) make that other place a module and import it or 2) use the default arg trick I described above. probably in your module constructor

note, the builtin SDKs are still "regular" modules except for the Go sdk that is truly a special case

[robust hare]

Solution 2 won't work because that would require consumers of the module to have its dependencies in their context dir.

... and pointing that out makes me realize why modules don't get access to their context dirs. For git modules, you're probably pulling only the source-root from the repo. The context directory is sort of a subjective concept... So, I'd say I'm satisfied with the constraint now. I don't think that the overhead of using module dependencies is worth the benefit of laying out my source differently. But it's a good possibility for me to keep in mind, and I am curious exactly what it would look like to pull files from a dependency module.

That said, it seems to me like modules should at least have access to their own source-root. E.g. I should be able to have

moddir
  dagger.json { "source": "module" }
  foo.img
  module
    main.go

... and access foo.img from main.go.

[sharp zealot]

the context dir is always the "callee's" context. So it would work in your case. it's like module source but at the scale of the whole repo

to access your module root, use an argument of type directory and set default path. . is moduke root. / is module context

[robust hare]

Okay, everything is making sense now. Thanks, this works very nicely, and doesn't require me to have them nested.

[alpine elk]

Hello,
I don't understand how to use interface (https://docs.dagger.io/api/interfaces/).
Anybody have an example / cookbook about it ?

[obtuse glacier]

Not at the moment; the recommended

[latent trellis]

Here is an early example from last year: https://github.com/sagikazarmark/dagreleaser

[alpine elk]

Thank you @latent trellis . Your repository https://github.com/sagikazarmark/daggerverse is already my bible for a few days now.

[leaden pagoda]

dockerfile build used to allow stdout to use the dockerfile entrypoint by default. it seems since 0.10+ some changes were made and now I need to have a withExec and copy the entrypoint cmd. Seems like a bug?

ah. I can just provide empty withExec and set use entrypoint. it's an extra line I thought wasn't needed.

[obtuse glacier]

Is this supposed to work in the golang SDK?

ctr.File("report.xml").Export(ctx, "report.xml")

I'm getting this output and the file is not exported:

/scratch/report.xml

[deft rain]

are you using modules?

[obtuse glacier]

yes

[deft rain]

so, the export function is exporting to the container runtime of the module - not to the host directly

to actually export to the caller where the dagger command is being run, you need to return the File and then -o /scratch/report.xml

[obtuse glacier]

My use case is unit-test | export-report | stdout is it possible to do that?

[deft rain]

so you want to get the contents of the report.xml?

if so, you should just be able to return the file from your function, and the caller can do dagger call my-function contents

[obtuse glacier]

no, i want to export the file and then show the output of the test run

[sharp zealot]

Your options are:

1. return a custom struct with fields for both the output and report file (and anything else you want). Then the caller gets the object and further queries what they need

2. return only the report file, and rely on dagger's own log output to stream (and trace) the output of the test command

[rose cloak]

Our humble request to give some priority to the dagger watch feature as such it can empower all the developers to drive end to end devops lifecycle with dagger only
https://github.com/dagger/dagger/pull/9108
PLEASE PLEASE...
we also badly need this.

[spiral whale]

Do we have a way to control what we're showing in TUI or console logs. I tried to use some internal traces but didn't worked like I was expected. I'm having some dynamic directory operations in our monorepo and the logs for these operations spam my logs, especially in CI

[sharp zealot]

cc @thorn moat 🙂

A popular request recently

[spiral whale]

can understand, new version has way better UI but way less readable due to these logs. Especially if you're doing directory operations like me. 1.6k line of logs to reach actual run target :D. This is one of the shortest examples we have

[thorn moat]

Do we have a way to control what we're

[hazy turret]

I just love this addition 😉

[rustic thicket]

When writing a module that takes a directory argument, I'm often trying to determine if a file exists in that given directory before running anything in a container. Something like this:

func (m *Composer) Install(ctx context.Context, src *dagger.Directory) (*dagger.Directory, error) {
    // make sure there is a composer.json file
    f := src.File("composer.missing")
    if s, err := f.Size(ctx); errors.Is(err, os.ErrNotExist) || s == 0 {
        return nil, fmt.Errorf("missing composer.missing: err %w", err)
    }

    // if there is no composer.lock, log it but continue

    return dag.Container().
        From("composer:latest").
        WithMountedDirectory("/app", src).
        WithWorkdir("/app").
        WithExec([]string{"composer", "install", "--prefer-dist", "--optimize-autoloader", "--ignore-platform-reqs"}).
        Directory("/app/vendor"), nil
}

Unless, I'm missed it. It would be helpful to have a FileExists method on *dagger.Directory or *dagger.File to make that logic a little more succinct. Happy to file an issue but wanted to ask here first

[deft rain]

agreed that it would be useful 🙂 there's actually quite a long issue discussing this here: https://github.com/dagger/dagger/issues/6713

[thorn moat]

yeah, it's a simple ask but in terms of API it gets a bit existential since the exists call would be made against an object that is only ever constructed if the file does exist. The simplest way might be to just f, err := src.File("composer.Missing").Sync(ctx) - if that returns an error, the file doesn't exist.

[sharp zealot]

Since this came up, I took a minute to update that issue, to reflect the fact that there are at least 2 design options

• "option 1" is what I originally asked for.
• "option 2" is what I believe would be easier to implement in practice. And honestly I'm fine with it

[edgy cape]

Completed a MVP-impl to leverage the Dagger Shell to compose pipelines as Notebooks using Runme. Would love to get some feedback. Posted a demo video on this issue: https://github.com/dagger/dagger/issues/8990#issuecomment-2578650472

For background, this issue is related to make the UX integration in Runme Notebooks work.

[latent trellis]

Theoretically, would it be possible to gather traces of the same module across different projects on Dagger Cloud? (Let's put the privacy concerns aside for a moment)

If I wanted to see how my Go module performs across different versions, would that be feasible? (Obviously, it only makes sense to compare builds of the same project and then those numbers to each other depending on the version. That could provide information about performance)

[plucky ermine]

@latent trellis sometimes it feels like you are reading our minds because we were literally just talking about this last week.

I don't think its possible at this exact moment but we are working on some early ideas to address this level of visibility.

Can I loop you in get some early feedback?

[latent trellis]

Sure!

[latent trellis]

I love giving feedback 😂

[plucky ermine]

we love to hear it 😄

[latent trellis]

What's the latest on caching?

I had a few conversations with people about their CI pipelines. All of them rely heavily on caching of their respective CI systems.

This topic keeps coming back in a lot of my conversations about Dagger.

It's also confusing for some people since caching is a highlighted feature of Dagger, but it's problematic when running on ephemeral CI machines.

[latent trellis]

I started daggerizing the justfile in my daggerverse repo: https://github.com/sagikazarmark/daggerverse/pull/209

Hopefully it becomes something useful in the end helping others create and manage their own daggerverse repos.

[sharp zealot]

Does Directory.diff() track deletions?

What about files that are changed but neither added nor deleted?

[fallow slate]

i know there's several modules for dotenv loading but figured why not make my own lol
I don't think its much different than any of the others but will handle *.secret.env files where it'll load those ones as secrets

https://github.com/bardic/daggerenv

[dense canyon]

https://github.com/moby/buildkit/blob/ef0f82244dc5ece851d021c098e7dc09f655d460/docs/dev/merge-diff.md?plain=1#L97-L133

Does Directory.diff() track deletions?

yes

What about files that are changed but neither added nor deleted?

yes

[gilded ibex]

Hi, I have a very generic (and maybe dumb) question about Dagger.Container type (using Go SDK but not related to it strictly):
When I use this kind of chain

dag.Container().
From("alpine").
WithExec(...args)

I expected this to be equivalent to

docker run <image> <args>

However it looks to me that using an image with a specific entrypoint is not working properly if in WithExec I provide the arguments without repeating the entrypoint command.
WithExec appears more close to docker exec

[strong ingot]

Entrypoint is skipped by default. See the rationale behind it: https://github.com/dagger/dagger/pull/7136

FYI https://pkg.go.dev/dagger.io/dagger#ContainerWithExecOpts

[strong ingot]

There are a bunch of related changes here, including how this affects service containers: https://github.com/dagger/dagger/blob/main/CHANGELOG.md

[gilded ibex]

with 2 cents you solved all the problems I had in my code

[gilded ibex]

Am I wrong or the Go example here is outdated?
https://docs.dagger.io/features/services/

Should not it be (in Go)

WithExec([]string{"python", "-m", "http.server", "8080"}).
AsService()

[strong ingot]

Good catch, thank you for sharing it: https://github.com/dagger/dagger/issues/9373

[gilded ibex]

One last question.
Is it possible to create a dagger.Directory or a dagger.File just by a path name string?

I cannot find a reference to that. All the examples so far just pass directory through a call, transforming the path string into a directory/file into the function body like magic

[sharp zealot]

It's a convenience from the CLI. Under the hood it calls host().directory()

Your dagger function is sandboxed so can't access files on your host on their own, they need to be passed in by the client

[gilded ibex]

I see, so it is not possible to do smt like dir := dagger.Directory{} // <- some host path inside the code itself.

I guess this is by design because actually everything at runtime is mounted within the Dagger container itself that then internally is handling and spawning containers. Somehow dagger container is working like a DockerInDocker service, to make it more clear to me...

[sharp zealot]

yes, dagger itself is a container orchestrator and runtime. Your functions are executed as containers, and can call the dagger api to orchestrate more containers

so everything is sandboxed by default

[gilded ibex]

But I can still download remote files and clone git repos, so I already have a way to circumvent that

[sharp zealot]

yes absolutely, and you can take directories, files, containers, service endpoints as arguments to your function too

[solar juniper]

Hi folks, I am pretty new to Dagger. I have a mono repo with a bunch of custom modules (using go sdk), I am wondering if there is a single command I can use to regenerate the /internal pkg for each of these modules or if I will have to script it? I tried dagger develop with a few variations, but the internal package doesn't get generated unless I execute it directly in each module's root.

[sharp zealot]

That is something I would also love to have. For our own CI we developed a dagger module to help 🙂

[solar juniper]

I was mainly looking into documenting for my team so they could start developing and running Dagger locally. A simple script might just do the trick! Thank you 🙏

[sharp zealot]

I think dagger develop --all or something like that would be really nice btw

[gilded ibex]

on a similar topic I also wondered if it exists the ability of dagger.json to handle multiple source modules. I am creating a sort of personal daggerverse and for each example I have the following structure

├── basic
│   ├── LICENSE
│   ├── dagger.json
│   └── hello-dagger
│       ├── dagger.gen.go
│       ├── go.mod
│       ├── go.sum
│       ├── internal
│       └── main.go
├── keytx
│   ├── LICENSE
│   ├── dagger.json
│   ├── key-tx
│        ├── dagger.gen.go
│        ├── go.mod
│        ├── go.sum
│        ├── internal
│        └── main.go
└── git-zero
    ├── LICENSE
    ├── dagger.json
    └── git-test
        ├── dagger.gen.go
        ├── go.mod
        ├── go.sum
        ├── internal
        └── main.go

which is extremely repetitive, but maybe I am really working on an edge case

[leaden pagoda]

modules may be at different versions of dagger, i think the isolation is important

I did think of something which could be a nice feature that may be abstractly related to you though, i thought a dagger functions call should be able to know about all child modules that the command was called from, think of a mono repo with various modules, maybe even written in different languages - e.g a front end, with its own build and ci in typescript, a backend in python, with python dagger module for api/deployments of the API but you may want the ability to have a sort of higher level dagger command, which understands/captures a larger context

[sharp zealot]

In the dagger shell interface (under development) we're giving a more prominent place to dependencies, so that they're more user-visible.

So in the context of a given module, that module's dependencies are exposed to the user as utilities you can call and compose

Example in the current shell syntax:

⋈ .use github.com/dagger/dagger
github.com/dagger/dagger ⋈ .deps
Available dependencies:
  - alpine
  - dagger-cli
  - dirdiff
  - docusaurus
  - elixir-sdk-dev
  - go
  - graphql
  - helm
  - php-sdk-dev
  - ps-analyzer
  - python-sdk-dev
  - releaser
  - shellcheck
  - version
  - wolfi
⋈ alpine | container | terminal
dagger / $

[restive shore]

This is great! I was planning on doing something similar. I like what your Just file does but I didn't want to force yet another tool on the devs.

[rain hedge]

I have a broad and maybe noob question. I would like to have spent some time on the PoC of Dagger earlier but could not find it. I would like to prioritise this again in Q1.
What I noticed is that there are rarely functions/modules developed via TS. Does this mean that Go is more stable for Dagger?

[dense canyon]

hey there! All the official SDKs (Go, Python and Typescript) have the same level of feature parity. The reason why most modules are written in Go I'd say is that most of the Dagger team are Go devs and that Go and python are generally more "devopsy" languages.

Having said that, there are a bunch of modules in the Daggerverse and every now I then I see some TS modules floating around. @brisk tapir is a hardcore TS module author for example

[strong ingot]

cc @upbeat herald

[upbeat herald]

Go SDK is known to be faster than the TypeScript SDK but there's no stability differences between the 2 since they depend on the same API.

There were some slight differences between Go and other SDKs, for example, Go has had interface suppport for more than a year but now others SDKs such as TS and Python now have the same capabilities so you can use your favorite language to code your dagger module 🙂

[rain hedge]

Thanks all - I thought it was a noob question but appreciated for the clarifications! I ll try both and share my feedbacks soon

[sharp zealot]

@sour citrus I would be very interested in your thoughts on the SDK development experience, based on your work on the Java SDK so far.

[sour citrus]

Sure, I'll write a few words about it. Basically I did a lot by reverse-engineering the different SDKs (ts, php most of the time) and the generated go code.
And one of the aspect that was not that easy is to understand exactly what is the expected output (files and directories) of the runtime part.
I'll compute all the notes I took on the topic.

[sharp zealot]

I saw another developer suggest that the SDK API be broken down into more fine-grained calls, with a more cleanly decoupled lifecycle. Does that make sense to you?

[sharp zealot]

Is there any hope of dagger interfaces supporting two-way callbacks?

Module A defines two interfaces, say "app" and "platform". One takes the other as argument (for callback)
Module B implements the App interface
Module C implements the Platform interface

Can A orchestrate B and C calling each other via the interface?

And how to do that without circular dependencies?

[wintry prism]

I've never seen this error message. Was trying to pass local dir as workdir via dagger call

error: parse selections: parse field "recorder": init arg "workdir" value as dagql.DynamicOptional (DirectoryID) using dagql.DynamicOptional: decode "Directory" ID: failed to decode base64: illegal base64 data at input byte 0

func New(
    // Working directory for the recording container
    // +optional
    workdir *dagger.Directory,
) Recorder {
    if workdir == nil {
        workdir = dag.Directory()
    }

[warped canyon]

I haven't seen it either. Do you get the same error if you call the argument something else?

[wintry prism]

I've never seen this error message. Was

[wintry prism]

https://github.com/shykes/x/pull/2

[sharp zealot]

This caught my eye, could be a nice primitive for a computer use / AI agent module 🙂 https://x.com/atinux/status/1882426945562153014

Sébastien Chopin (@Atinux) on X

An open source headless browser rewritten in Zig designed for AI and automation.

• 9x less memory footprint than Chrome
• 11x faster execution than Chrome

Very exciting times.

https://t.co/dc6XrnrvwM

X · Jan 23, 2025, 1:55 PM

[restive shore]

The standard --progress=plain is still too verbose. It gets very crowded when there are multiple module dependencies. I'd almost want something that shows stdout of my immediate withExecs with the ability to set custom "spans" as labels. As a dagger power user, even I can barely understand what's going on and I can see others being very confused by looking at it. This is important because it's the only type of output possible on the CI runner. In my case Jenkins.

[deft rain]

agreed 🙂

tried to do some clearing up in https://github.com/dagger/dagger/pull/9434

but we also need some way of letting users pick what to view - there's an issue for discussion in https://github.com/dagger/dagger/issues/9374 (feedback appreciated!)

[restive shore]

this is a good start! happy to see the discussions happening around this!

[solemn basin]

Hello, one question about caching. I have following Dagger pipelines:

from typing import Annotated

import dagger
from dagger import dag, function, object_type, Ignore, DefaultPath


@object_type
class Blueprint:

    @function
    def build(self, context: Annotated[dagger.Directory, Ignore([".dagger"])]) -> dagger.Container:
        return (dag
                .container()
                .from_("eclipse-temurin:23-jdk-alpine@sha256:2cb34eaa1bb3d1bdcd6c82c020216385ca278db70311682d8c173f320ee4f4c4")
                .with_workdir("/src")
                .with_mounted_directory("/src", context)
                .with_mounted_cache("/src/.m2", dag.cache_volume("maven-3.9.9"))
                .with_service_binding("docker", dag.docker().engine())
                .with_env_variable("MAVEN_OPTS", "-Dmaven.repo.local=/src/.m2")
                .with_env_variable("DOCKER_HOST", "tcp://docker:2375")
                .with_exec("./mvnw clean package".split())
    )

    @function
    async def container_image(self, context: Annotated[dagger.Directory, Ignore([".dagger"]), DefaultPath("/")]) -> dagger.Container:
        build = self.build(context)
        build_dir = build.directory(await build.workdir())
        return dag.container().build(build_dir)


    @function
    def verify(self, context: Annotated[dagger.Directory, Ignore([".dagger"]), DefaultPath("/")]) -> dagger.Container:
        return self.build(context).with_exec("./mvnw verify -DskipTests".split())

When I call dagger call build --context=. and then dagger call verify --context=. in second call the build steps get reexecuted and not reused from cache. Why are the build steps not being reused from cache in subsequent verify?

[quick wind]

Is there a page/section in the docs that describes when Dagger will fetch an image vs use a local image? I'm sure I'd read that using the image digest would result in Dagger using the pulled image and not fetching it again?

[sharp zealot]

If you specify the digest, Dagger (via buildkit) will skip looking up the latest version of the tag.

Either way, for a given digest, whether to use the local copy or download is independent of whether you specify the digest. If the data for that digest is in cache, it will always be used. Otherwise, it will always be pulled

[quick wind]

So having that image digest locally means nothing, it has to be fetched and cached first regardless

[sharp zealot]

Skipping the lookup is an extra optimization, to save the few 3-digit milliseconds you'll spend on that lookup even when the layer is in cache

[quick wind]

Then once it's cached, no more fetching

[sharp zealot]

Once it's cached, no more fetching BUT there will still be that ref lookup unless you specify the digest

This is exactly the same behavior than in vanilla buildkit

you can try with docker build in a dockerfile for example

[quick wind]

All good, I'm using the digest

[dense canyon]

@deft rain any ideas if this could be related to the cache volume namespacing invalidations we've been observing? I'm running this locally and seems like the dagger call verifycall invalidates everything after the with_mounted_cache step

[deft rain]

it's definitely not the service getting invalidated? dag.docker().engine() is constant?

do you have a trace?

[dense canyon]

I've just removed both the service and the cache volume and still getting invalidations

let me get you the traces

[dense canyon]

Initial build: https://v3.dagger.cloud/marcos-test/traces/7754214c795fa1f26209da20b237e1fc
Follow-up verify: https://v3.dagger.cloud/marcos-test/traces/35b276ed99d01a134b46d1ca9cc4a933

Code:

from typing import Annotated

import dagger
from dagger import dag, function, object_type, Ignore, DefaultPath


@object_type
class Blueprint:
    @function
    def build(
        self, context: Annotated[dagger.Directory, Ignore([".dagger"])]
    ) -> dagger.Container:
        return (
            dag.container()
            .from_(
                "eclipse-temurin:23-jdk-alpine@sha256:2cb34eaa1bb3d1bdcd6c82c020216385ca278db70311682d8c173f320ee4f4c4"
            )
            .with_workdir("/src")
            .with_mounted_directory("/src", context)
            # .with_mounted_cache("/src/.m2", dag.cache_volume("maven-3.9.9"))
            # .with_env_variable("MAVEN_OPTS", "-Dmaven.repo.local=/src/.m2")
            .with_exec("./mvnw clean package".split())
        )

    @function
    async def container_image(
        self,
        context: Annotated[dagger.Directory, Ignore([".dagger"]), DefaultPath("/")],
    ) -> dagger.Container:
        build = self.build(context)
        build_dir = build.directory(await build.workdir())
        return dag.container().build(build_dir)

    @function
    def verify(
        self,
        context: Annotated[dagger.Directory, Ignore([".dagger"])],
    ) -> dagger.Container:
        return self.build(context).with_exec("./mvnw verify -DskipTests".split())

as you can see from the verify step, the mvn clean package should have been cached 🤔

[deft rain]

the mounted directory definitely isn't changing either?

but yeah, then i don't think it's going to be the result of a cache volume

[dense canyon]

nope, it's the same dir

you can see from the traces that the directory is the same

[deft rain]

lol i can't load your trace, it paniks

[dense canyon]

nope, I've removed the volume and still invalidates 😬

@deft rain I think I know what's happening

it's the famous 2nd to last thing we saw at kubecon remember?

I've opened an issue about that

[deft rain]

https://github.com/dagger/dagger/issues/8955

this one?

[dense canyon]

yes

[deft rain]

huh okay

i'll go play around with that, i hadn't remembered that

[dense canyon]

because verify is adding a new step to the pipeline

so package which is the second to last, gets invalidated

lol

IDK how that crossed my mind

but just verified and it seems to be that. Added a noop exec at the end

[deft rain]

crap

welp 😛

yeah, i'll go splunking then

[dense canyon]

@deft rain lol, found another variation of the issue above. The problem seems to also manifest within the scope of each function

for example in my code above: If I add another step after the mvn package in the build function, then mvn package gets cached when calling verify correctly. However, if I leave build as-is but I add another step at the end of verify, mvn package will still get invalidated.

I'll add this to the issue

[dense canyon]

Hey there! this seems to be realted to this issue: https://github.com/dagger/dagger/issues/8955

A stopgap to avoid hitting this cache invalidation is to add another no-op step after the mvn package one. Let us know if that works

[meager grove]

Is there a way to retrieve a secret from a dagger function call like the following without bleeding credentials all over the place:

export token=$(dagger -m $DAGGERVERSE/gcloud@main call container \
       with-mounted-secret --path=/mnt/$SA_PATH --source=file:$SA_PATH \
       with-exec --args=gcloud,auth,activate-service-account,--key-file=/mnt/$SA_PATH \
       with-exec --args=gcloud,auth,print-access-token stdout)

Or.. any general recommendation on how to handle passing a secret from one dagger module to the next without coupling them?

[south helm]

random dagger-for-dagger GHA pipelining challenge: im trying to send notifications to discord when a certain build fails. that's easy enough with @strong ingot 's notify module. the hard part is that ideal content of that notification is a link to the dagger cloud trace for the current call... anybody have some prior art for dagger functions self-linking their parent-root cloud trace? cc @deft rain ... this would be some generically useful functionality that i'd prolly upstream into that tiny module

[solemn basin]

Thank you! I searched the issues, but found nothing. Good to know. I will keep an eye out for the fix. You guys are doing great work!

[deft rain]

goooood question, i can't think of any prior art, but it might be available through OTEL? if you can find the trace id in there

[strong ingot]

Looking forward to that PR!

As for the Dagger Cloud trace info, my first thought is @thorn moat . My second thought is @chrome pilot .

[dense canyon]

sure, np. Let us knwo if the stopgap proposed above works for you or if you have any other questions 🙌

[dense canyon]

@south helm each with-exec has a TRACEPARENT env variable which has the traceID there. dagger shell -c ".core | container | from alpine | with-exec env | stdout". That what some user have been using to get the ID to construct the dagger cloud URL to use it somewhere else. Does that helps?

[south helm]

yeah i know it's in there, but i don't wanna spin up a container to get it, it should be accessible in module code... maybe i can reverse engineer based on sdk codegen & execs maybe? is it possible today to add custom otel spans and whatnot to module code? like i know you can wire up your userspace running-in-an-exec code to our otel stack, but can you wire up the module code in between?

[dense canyon]

haven't checked but if you do os.Getenv("TRACEPARENT") from the module's code you should also get it? Since the module's code is running withing the context of a span also?
mostly assuming here.

[south helm]

lol we are of the same mind ```func (b *Bench) Traceparent() string {
return os.Getenv("TRACEPARENT")
}

and it is indeed set

idk why i assumed the module code didn't have OTEL set up, bad place to start from

it does have extra bits tho which is kinda weird: 00-5db51a3dca984fab7d2a4b880c19ade7-8e1137c6bf094be0-01 but the actual trace id is 5db51a3dca984fab7d2a4b880c19ade7

easy enough to parse out but im still gonna see if i can get otel to do it for me

[south helm]

func (b *MyModule) Traceparent(ctx context.Context) string {
    spanC := trace.SpanFromContext(ctx).SpanContext()
    if spanC.IsValid() {
        return spanC.TraceID().String()
    }

for anybody who finds this via search

[strong ingot]

We are unlikely to find it in this haystack. Any chance you could add it to GitHub somewhere? This seems a good place to me: https://github.com/dagger/dagger/tree/df40da5a61c787ef321f503287dcde92222279d8/core/docs

Maybe @bleak nest can suggest a better place?

[south helm]

tbh what really needs documentation is basic OTEL usage in module code, like it should first be documented how to get a tracer and add new spans in each sdk language, then we have an additional example of pulling the parent traceid without creating a span if we really want to, but it's kinda a special case that'd follow just fine from docs that say "hey, dagger is providing otel config to module code, all you gotta do is use it like this"

[bleak nest]

For the first idea, we already discussed adding a recipe for creating spans to the cookbook but ultimately decided to wait for the higher-level API. Context in https://github.com/dagger/dagger/pull/9301 and #t-documentation message. Now to answer @strong ingot's question, if we wanted to add this to the docs, a good place would be the cookbook (possible recipe title: "pull a parent trace ID without creating a span"?). But again, maybe this will be part of the higher-level API and so we wait for that? /cc @thorn moat

[thorn moat]

Adding an API to get the current trace ID would be pretty easy yeah

[south helm]

tq i was looking for this but reading issues instead of PRs like a dummy

tbh i'd question how useful the traceid-without-a-span is for normal users... it's pretty specific to modules that externalize call-state to other platforms, hence wanting to just put it in notify as DaggerCloudTraceLink(ctx)

[thorn moat]

true, all use cases i know of are just for generating a cloud URL

[south helm]

or a datadog/prom/honeycomb url in the hypothetical case you're exporting dagger traces to a third party provider

[solemn basin]

I used a different solution than the one proposed. Basically I understood that I do not need mounted directory, just a directory. Using the directory, instead of mounted_directory fixed the issue with caching. It intuitively makes sense that mounted directories break the cache, at least from my perspective. I can not say if it technically makes sense for them to do so though.

[sharp zealot]

That's very surprising to me, as with_mounted_directory is a read-only bind mount. It shouldn't affect caching any differently than with_directory

[dense canyon]

it's basically this. The main different is that with_mounted_x is not read-only. It's still a rw overlayfs mounted in the underlying container with the difference that the changes made there do not become part of the image container's FS

[sharp zealot]

right 👍

[warped canyon]

Question about cache volume filesystems

TLDR with npm cache, npm has to copy a huge number of small files to a projects node_modules directory. So generally a copy from a with_mounted_cache to a with_directory

In this example, the copy takes roughly 40 seconds. However if I do the same operation from a terminal session, it's almost instant.

My only guess is that there's some kind of difference with the filesystems when you're in a terminal. Any way I can get that same performance from the with_exec??

[sharp zealot]

If I wanted to hack self-calls myself, with a raw GQL query, how would I do that?

[sharp zealot]

something like: dag.WithGraphQLQuery(querybuilder.Query().Select("load<MyType>FromID").Arg("id", CURRENT_ID) but how to get my current id?

[deft rain]

i don't think it's plumbed through

theoretically you'd add it as an environment variable when you call WithExec in modfunc.go

the current id is available in dagger as dagql.CurrentID(ctx)

[meager relic]

Is there any way to disable caching? Either at the engine level or perhaps on a per-command level?

I'm running into issues with a build that is slowing to a crawl and crashing

My current theory is that there is a lot of directory manipulation, which appears to have significant performance impacts. And I think a large portion of that comes from Dagger having to re-compute digests on every step.
If there was some way to tell dagger "don't even attempt to read/write from cache, just execute", I'd be interested to see what kind of performance gains are possible

[tired pasture]

If I remember well the documentation, you can just add a timestamp in an environment variable to your docker image and this will invalidate cache.

[leaden pagoda]

I do vaguely remember 1 example in the cookbook within the docs for cache invalidation.

[restive shore]

Has anyone else run into this? I get this when an external module (in another repo) tries to copy a file local to that repo. External module is 0.15.2 and my local module is 0.15.3. There is no secret involved here so that's puzzling. I saw this once earlier and it went away (not sure how or why) but it popped up again when I upgraded my local module to 0.15.3.

! failed to add client resources from ID: failed to add secret from source client wg21a6nhnlgkro4qogf2t12fo: secret xxh3:a1b22fd50fdb4b95 not found in other store

I'll try to reproduce with a separate module

[dense canyon]

I recall seeing that in the past. Would be awesome if you could get a consistent repro Nipuna 🙏

[restive shore]

trying.. This one is tricky because it's across module repos 😅 and involves file import as well as secrets

[restive shore]

Has anyone else run into this? I get

[sharp zealot]

Is the pattern format for Directory.glob documented anywhere?

I'm finding it hard to match even simple things, like "all files anywhere with a .txt extension"

[kind carbon]

**/*.txt doesn't work?

[sharp zealot]

It only matches subdirectories I believe (won't match ./foo.txt)

[kind carbon]

⋈ directory | with-new-file foo.txt "bar" | glob '**/*.txt'
foo.txt
⋈

[sharp zealot]

!!

Oh I think it's because I'm confusing with .gitignore format which is different

(hence my question about documenting it)

[warped canyon]

since I can't use dag.Host() in a module, is there a way to run a server in code and expose that as a service?
What I wanted to do was do something like http.Server in go, and return dag.Host().Service({Backend: 8080, Frontend: 8080})

I could move the server to it's own go package and run that in a go container, but i'd rather not if there's an alternative to dag.host().service()

[sharp zealot]

I think we should re-expose Host, assuming it's safely sandboxed. The only issue is user confusion (eg. maybe we should call it something else 🙂 , but it's practical to have it

dag.Self() ? dag.Client()? I feel like we've had this conversation many times over the years 🙂

[warped canyon]

Sandbox? Runtime?

[sharp zealot]

No because it has to work both in the context of an external client, and a client inside a function runtime

[warped canyon]

true

[sharp zealot]

that's what makes it tricky

[leaden pagoda]

dag.host() was confusing to use/understand at the start, within a function itself

feels like it went against a dag input -> output (cli being the out/end)

i think when i came along, this was being removed, or did get removed - but i surely remember seeing examples of it from the old docs

can also lead new users down a path of not understanding the code' -> 'upload -> engine do things -> back to client -> cycle

dag.WhenRunOnYourTerminal... kidding

[sharp zealot]

@leaden pagoda yeah for sure if we re-enable it, we also need to rename it, if not right away, at least very quickly as a follow-up

The main problem is the name host implies that your function can reach outside the sandbox, which it can't. Its' actually reaching out to its own runtime container 🙂 Which is sometimes useful (like what Kyle is trying to do) but not at all the same thing

[leaden pagoda]

'context' comes to mind naturally, what is 'my context' - is the context a client? or the 'server' where its running? :S

[sharp zealot]

The current pain is that when you do need it, it's for niche but legitimate reasons, and it's weird to forbid it when the engine is 100% capable of it

[sharp zealot]

yeah context is definitely one we considered, unfortunately it got taken to mean "the git repository where the current module lives". But maybe we should have helf off and not exposed that meaning of the word to the user. I always have to explain anyway, "this means the git repo containing your module"

plus there's the Go-specific problem that "context" has its own meaning in Go..

[leaden pagoda]

yeahhh, that was fun to learn being a new go person, mostly to learn dagger 😛

would it be correct to say the runtime is a container that the dagger engine session has created? also with dagger run (the old way, i think was ABOUT to start this way of using dagger) but then modules came out

could it not somehow have engine within the name

Or Session

hm

SessionHost, bleh, could still be interpreted as the users host if they dont understand the sandboxing

[sharp zealot]

the closest thing really is Client

the Dagger API supports client callbacks - eg. the API server can ask the client for things, like its files, its secrets etc.

[leaden pagoda]

i dont understand this part, is there a doc / diagram on that bit - .Host() only makes sense to me in the old way still and if kyle wants it to use within a module, then its still only ever running from the engine

[sharp zealot]

there is no good diagram of this, but we will make one soone I promise

When we introduced Dagger Functions, we didn't change the Dagger API or how the API client works. But we added a new way for you to develop an API server that can extend the Dagger API with new types and functions --> that's Dagger Functions.

In theory you can write such a server without needing a client (for example if you write a dagger function that just returns "hello", that function doesn't need a Dagger API client). But in practice, 99.99% of Dagger Functions (server code) also need to call the Dagger API, and therefore they need a client.

So now there are 2 different places you may need a Dagger API client: 1) from an outside program running on the host, and 2) from inside a Dagger Function which itself is implementing an API server to extend the engine's API

When we named dag.Host(), there was no difference between "the system where the client is running", and "the host system where the dagger CLI was called". But when we introduced Dagger Functions, suddenty those 2 systems can be different.

There is actually no API to access the host. Only an API for the client to give the engine access to its own system.

So with all the above 👆 in mind... Reasonable names that make sense in all cases, might be 1) dag.Client() 2) dag.Self() 3) dag.Context() 4) dag.Environment()

EDIT: added Environment as an option

[leaden pagoda]

oh, and i forgot the old way IS still supported, so because its NOT a module in all cases when Host() exists on the dag, doing dagger run and then runing some custom code, .host() ... i think im half way there

[quick wind]

https://github.com/dagger/dagger/issues/9005 This feels like it would be a fairly quick addition? Any chance of it making it's way into an upcoming release?

[deft rain]

atm, it's not actually particularly trivial - right now, we rely very heavily on LLB to perform file operations

something like https://github.com/dagger/dagger/pull/9395 will help lift this restriction - but it's sadly probably not going to happen for the next release

[subtle plinth]

With the new secrets providers, is it possible to use a default value to load secrets within nested modules; or do we need to fully load the root invocation and pass secrets about?

[sharp zealot]

No defaults yet but that's the next step.

My favorite design for defaults is 1) .env support + 2) allow default env var always scoped to .env

[subtle plinth]

Can you explain 1 + 2 a bit more? Is there an issue for this to subscribe to?

[sharp zealot]

will do today

[leaden pagoda]

https://d2lang.com/ well this looks juicy

unrelated to dagger sorry 😛 just we were talking about diagrams previously

someone from work showed me this, i havent used it yet

[sharp zealot]

Here's a d2 Dagger module 🙂 https://daggerverse.dev/mod/github.com/shykes/x/d2

Now you're back on topic 😛

[sharp zealot]

@chrome pilot 👋 https://github.com/aluzzardi/daggerverse/pull/20

I needed this to make my demo module simpler 🙂 Contributed a few chores while I was at it

[leaden pagoda]

is watch still a thing coming? I was thinking about d2 having a .d2 file and re-render - but it wouldnt work unless there was a way for dagger to monitor file/directories and somehow push them changes into the running session

[leaden pagoda]

d2 probably isntthe best example of usefulness with a watch cmd, im using vscode and its all i need - but it did make me think of watch or other more complicated things where you didnt want all those things installed and running locally like hugo changes or even docusaurus?

[sharp zealot]

watch is definitely planned but it's a pretty major feature, so we don't have a date for it yet

[leaden pagoda]

adds a lot of complexity too

like, how are services handled? continued service but somehow only updates files?

[sharp zealot]

right, it's tricky (but possible to implement). will be fantastic once it's done: a universal cross-platform watcher 🙂

it will have to restart the service though

just nothing else (including the forwarding proxy in front, for tcp connection continuity)

[quick wind]

https://docs.dagger.io/configuration/engine#security

The two code samples are the same (for engine.json)?

[wintry prism]

looks like first one should be

{
  "security": {
    "insecureRootCapabilities": "true"
  }
}

Will make an issue to investigate and fix. Thanks!!

[wintry prism]

@deft rain I wonder if JSON Boolean of true or String ”true” is called for 🤔

[leaden pagoda]

so i took this as inspiration and made one similiar 😄 turns out png and others needed more dependencies (urghhhh) and then i was learning more about d2, you can reference 'other d2 files' and pull them in, kind of like.. libraries

you know what would be cool, some kind of CLI spec, like an openapi document, or a json specification that describes (a bit like how we have CLI prompt completion) how a command line tool is used, if that existed and became 'popular' a lot of clis could probably have code generation wrapped for dagger

does this sound too loonyl?

ai agent -> go read the docs of a tool, generate json schema that describes the api from their documentation
ai agent -> read json schema schema, generate a dagger module for the cli tool using dagger primitives, like secret, file, directory, stdout, stderr that best suit the 'cli tool'
ai agent -> test the dagger module and check that each function returns a success status code and no errors to ensure the generated cli module is 'most of the way there'

[sharp zealot]

seems doable!

[dense canyon]

Seems to be mapped to *bool here https://github.com/dagger/dagger/blob/beea56ea5908fe9ca650e0de1e0d947c651ffef6/engine/config/config.go#L210 so it shouldn't be a string

[wintry prism]

https://github.com/dagger/dagger/pull/9546

[tired violet]

Hey, a small suggestion on the dagger cloud UI V3 : a tooltip on hover of the step status (pending / running / cached ...) might be useful (retrospectively it seems obvious but it took me some time to figure out that ♻️ meant that cache was being used)

[deep hedge]

#maintainers message - i raised the same request a couple of days back. Glad to know there are other folks who may find this addition to be useful

[edgy cape]

Been experimenting with the notion of a Dagger Shell notebook as part of the native Runme integration now available in Runme v3.12.

These notebooks render to Markdown, however, they've got a .dag file extension. Brand new .dag files always default to dagger shell. Here's a little clip https://www.youtube.com/watch?v=IKQQFVmB4UI.

Any feedback is welcome (updated this issue accordingly: https://github.com/dagger/dagger/issues/8990).

[dense canyon]

Noted. Thx for the feedback

[dense canyon]

Fixed Mikael, we've added an html title to the element so the typical hover text appears 🙏 . Thx again for helping us improve the product 🙌

[spiral whale]

is it possible to call modules from go code directly using dagger client?

[deft rain]

https://github.com/dagger/dagger/pull/9531 we're working on it now 😄

what's funny is we've had almost this exact question about 3 times in different places in the last couple days lol 😛

clearly @upbeat herald is working on the right things

[upbeat herald]

The target date is next release but it will be experimental indeed, it will also requires to wrap your execution inside dagger run in order to serve dependent modules inside the session.
That will be the only limitation for now, but we'll solve it at some point

[spiral whale]

does this mean my app will be run in sandbox environment as well right. At least for now

[upbeat herald]

I think dagger run let you access your host, local network etc, so I'm not sure

[deft rain]

yeah, the code runs on your host

so you can access all the hosty things

[sharp zealot]

Can you explain the need for dagger run? Since SDKs exec dagger session wouldn't that take care of it?

[upbeat herald]

I had a talk with Helder and he told me the same, and that's true, I will try to move the loading logic inside the withEngine function, so it should be global for dagger session/run

[subtle plinth]

Secret Default Values

[rose cloak]

Hi Dagger Team,

Could you please take down these modules from Daggerverse?
https://daggerverse.dev/mod/github.com/StaytunedLLP/dagger_modules/denodeploy@169f67ceecbc5ad823913e74842ba658512af679
https://daggerverse.dev/mod/github.com/StaytunedLLP/dagger_modules/git-diff@169f67ceecbc5ad823913e74842ba658512af679

Thank you.

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/9584

[obtuse glacier]

Does dagger do something different with logs?

With dagger I get no output from this:

From("nginx").
WithExec([]string{"nginx", "-t"}).
Stdout(ctx)

With docker run:

docker run nginx nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[spiral whale]

hey I'm experimenting concurrent ci checks with custom traces at the moment. Is there a way group executions in my function under my custom trace?

This is a little bit chaotic to understand what happen:

https://v3.dagger.cloud/gale/traces/99b23e6b7de499cde96ef8497d844beb?span=10ecd269b2b63dd9

[rose cloak]

Hey dagger team, please help. thanks

[sharp zealot]

@rose cloak are these your modules?

[rose cloak]

Yes

[rose cloak]

We have 4 now, but 2 with old url which will not work...as we renamed repo from dagger_modules to Daggerverse so, that's it

And so we want to remove mentioned 2 with old urls.

https://daggerverse.dev/search?q=staytuned

[sharp zealot]

OK @rose cloak I forwarded your message internally, we don't have that feature in Daggerverse yet but we can do it on a case-by-case basis.

[rose cloak]

alright, thanks a lot.

[wintry prism]

yes! My team will make it happen @rose cloak 👍

[wintry prism]

done! ✅

[rose cloak]

Thanks a lot.

[bright beacon]

Hi All
While building and publishing a multi-arch image from my CI to gitlab private registry, only the image tag being pushed.
Error on hovering the image: Invalid tag: missing manifest digest
On investigating more i found it could be because of the buildx and private registry compatibility issues. ref: https://anju-chaurasiya2012.medium.com/debugging-docker-buildx-invalid-tag-missing-manifest-digest-c4896bd55beb#:~:text=The “Invalid tag%3A missing manifest,Buildx process and registry interactions.

Has anyone faced this issue before?
Is there a way to set up build-x variables like provenance in dagger?
Thanks in advance!

[sharp berry]

Hey folks, general question I was wondering is how one would go about migrating from Jenkins with Groovy to Jenkins with Dagger, but maintaining stages and steps. Everything i've seen seems to lead to a single stage build

[velvet jolt]

hi

how can in use a local image with the python sdk ?

.from_("pythonflask") does not work and try to pull from the classic registry

does i have to mount a local registry ?

thanks

[kind carbon]

Hey, not from local docker engine. See https://github.com/dagger/dagger/issues/6411. You need a working registry endpoint.

[velvet jolt]

dam, if you have time, i will teach the basics of devops soon, and sometimes the proxy it's very complicated in the public administration, so if one day you make this possibility it could be nice. But thanks for the product.

(and your time 🙂 )

[kind carbon]

You can spin one up locally with https://hub.docker.com/_/registry

[velvet jolt]

yep that s what i will do i think 🙂

it will also teach the registry concepts for the students

[kind carbon]

Fyi: https://github.com/dagger/dagger/issues/5235

[velvet jolt]

thanks

[plucky ermine]

Check out the example in this repo that shows multi stage builds. Let me know if you have any questions

https://github.com/levlaz/snippetbox/blob/main/Jenkinsfile

[velvet jolt]

thanks

[pliant flame]

Hi all, one question, we are using dagger Golang SDK for building multi platform images. For one of the windows image, we had to support not just platforms (amd & arm) but also the OS versions for each platform. But we don't see the option for OS or OS Version in Golang SDK.
Any thoughts on this issue?

[restive shore]

This is what we do. If you are on Jenkins, you can't get away from groovy. Maybe once the --progress=plain output becomes cleaner.

[latent trellis]

Ran into this today after upgrading Dagger in my daggerverse:

! failed to decode module config: unmarshal module config: unmarshal module config: json: cannot unmarshal object into Go struct field alias.sdk of type string

It didn't take much time to figure out the problem, since I was upgrading Dagger, but I suspect this could happen when someone installs a module on a newer Dagger version.

Would it make sense to run some sort of version check first?

[deft rain]

just to check, which version was this to? this broke on v0.15.4, but i think we should have fixed it for v0.16.0: https://github.com/dagger/dagger/pull/9578

essentially, the sdk field became a map instead of a string - but when updating, there were some scenarios when the engineVersion wasn't also bumped, it might be that case here

[deft rain]

there should be a version check applied, older dagger versions should refuse to load a newer engineVersion than they are - curious how you got to that state

[latent trellis]

I updated dagger (and engine version) in my modules, but forgot to update the Dagger version in my GHA workflows.

Here is the workflow run: https://github.com/sagikazarmark/daggerverse/actions/runs/13457014473

[deft rain]

oh

[latent trellis]

My guess is, it couldn't unmarshal the config

[deft rain]

yeah, okay, we do all the unmarshalling first

before the version check

[latent trellis]

yep

[deft rain]

yeah, okay fuck 😛

cool, will fix that

sadly can't fix it for already released versions

[latent trellis]

I think it's specific to versions where the config is incompatible with a previous version

sdk -> sdk.source

[deft rain]

yeahhh you're right, will keep looking into it, but bleh, not sure it's a particularly cut-and-dry fix

gonna go grab lunch first 😄

[deft rain]

okay, worked this out: https://github.com/dagger/dagger/pull/9671

because it's an engine fix, already released versions will still get that incredibly annoying error, but at least from this point forwards we'll be able to catch this

[restive shore]

hey there, I had a module (golangci-lint) that did the dagger module codegen inside a module to run golangci-lint on it. It's not working on 0.16.1. Looking inside, I don't see the internal folder being included. I am getting the generated code with AsModule().GeneratedContextDirectory(). Is there a new trick to get the fully codegen'ed module now?

Here's my code that does this part

    for _, module := range ci.Modules {
        codegen := source.AsModule(dagger.DirectoryAsModuleOpts{
            SourceRootPath: module,
        }).GeneratedContextDirectory()
        scanSource = scanSource.WithDirectory("", codegen)
    }

[restive shore]

so I took a hint from this and updated my code - https://github.com/dagger/dagger/blob/cafeac4a8244ae9a407dbf6f5251acaa3cb1bc04/.dagger/main.go#L92. Now I see all the codegen but everything in dagger.gen.go is "undefined", hence golangci-lint fails.

[rocky harbor]

so I took a hint from this and updated

[velvet jolt]

how to make that dagger don't use the latest tag on a imge when making a pull ?

thanks

[leaden pagoda]

do you have a code snippet of what you are doing

[tulip root]

I just released Go Orb v0.1.0 we use Dagger for CI https://github.com/go-orb/plugins/blob/main/.github/main.go

We use Dagger for having isolated (no go.work) environments as well as multi processing.

In the past we used the following command a lot dagger call update --root=. source export --path=.

[sharp zealot]

We really need to change the dagger init template to include some state in the custom type

like a few fields in the object

[latent trellis]

Personally, I'd like to see the examples gone from init. Or an option to make them gone. My first step after creating a module is getting rid of all of them. I realize it may help people getting started with Dagger, but it's just noise for the rest of us.

What I'd like to see added, is a default empty constructor. I always add one.

[leaden pagoda]

init --starter-demo and if you don't include that you get completely empty module?

[sharp zealot]

I kind of like getting a skeleton, saves me time, but agree it could have less example content by default

[warped canyon]

am I remembering correctly that a dependency of my module can't use ExperimentalPrivilegedNesting in it's own execs? Is there a way for my module to say "i trust this dependency to use ExperimentalPrivilegedNesting"?

[deft rain]

nope, there's no protection, you can just do it 🙂

but generally, the nesting should only allow accessing the top-most non module

i think

[warped canyon]

haha, ok I remembered wrong 🙂 For some reason my dependency was still looking for a docker socket when executing the dagger binary, even with that set

[deft rain]

ohhh

hm, it shouldn't do that

it should be able to find it 🤔

[warped canyon]

I thought so too. I had it working fine when it was in the top level module, which is why my mind went to that

Actually nvm, its not working at top level now either. I broke something all on my own 🤣

not every dagger exec had privileged nesting set

[sharp zealot]

My DX wishlist for AI agent dev:

• Cross-module type export. We really need it. I know there is a risk of dependency matrix from hell, but there are possible mitigations - they are worth the effort. Modules are too cumbersome for true agent composition otherwise.

• Better laziness model. The current rule "returning an object is always lazy; returning a scalar is always sync" - is too heavyweight and mysterious. When building more amibitious modules, it gets realy confusing

• Send object fields as struct fields. I know I was against "leaking" the concept of fields vs. functions in the API, because I liked that GraphQL didn't differentiate. But it has leaked anyway, and the current DX makes it very painful to get a struct from another module. If I call a function that returns an object, and that object has 10 regular fields (not functions) - I should get all 10 fields for free in a single query.

@warped canyon do you agree? 👆 anything else?

[warped canyon]

Yep those sound like the top 3. Also self calls and maybe functions as args

[sonic vessel]

About dagger cloud, found from the link in thread #elixir message. I click to open the trace and it says:

input: trace code: 241, message: Memory limit (total) exceeded: would use 24.20 GiB (attempt to allocate chunk of 4230305 bytes), current RSS 28.80 GiB, maximum: 28.80 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (while reading column Branch): (while reading from part /var/lib/clickhouse/disks/s3disk/store/e05/e05bfed6-46b5-4f32-a5e6-03e646c23941/all_0_7454735_77055_11624065/ in table telemetry_2024_02_28.`.inner_id.52c57aac-77b9-4ce8-b01c-142e1c542e12` (e05bfed6-46b5-4f32-a5e6-03e646c23941) located on disk s3disk of type s3, from mark 108 with max_rows_to_read = 8192): While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder)

[wintry prism]

this one right? https://v3.dagger.cloud/dagger/traces/42a0e112dbc4fc053dfca2b7dfdb0644
seems okay now

[sonic vessel]

It worksnow. 🙏

[sharp zealot]

I got one yesterday too btw

[sharp zealot]

DX wishlist

[restive shore]

Hey all, I am seeing a insane amount of data when on --progress=plain with v0.16.2. Seems like a regression?

This seems to be the biggest culprit

[wintry prism]

@thorn moat 👆

[sonic vessel]

@wintry prism I hit the Memory limit error again. It will works if I keep refresh the page. 😂

From from this trace https://v3.dagger.cloud/dagger/traces/89c156efa3d6b3faa5d3dd56580ec5ca

[bleak nest]

I don't have a link atm but I've also seen this a couple of times. I'll make sure to save the link next time and add to the report

[restive shore]

works in local breaks in CI

[sour citrus]

I was wondering if it has already been discussed the idea to have a multi-types output from functions?
For instance I'd like to both export a directory and print a string. With agents that could be interesting, like "give me the result (the directory) and some explanations about why you did that". The default output is often too verbose, and to return multiple types could be a way to let the module handle that.
In go that would be something like ```go
func (m *Module) Refactor(ctx context.Context, src *dagger.Directory) (output string, dir *dagger.Directory, err error)

In other languages, that might be harder, except if that's in a specific response type.

[deft rain]

we've talked about it in this context: https://github.com/dagger/dagger/issues/8421

where you want errors with more info

[sour citrus]

Interesting, that's a very close use case

[fossil barn]

Hello, I'm working a dagger integration in a demo project and I stucked with this error. If I create a Service function, I can run dagger call nginx-service up --ports=8085:80 it start the service and I can connect to it without issue, but if I create a dagger call run-nginx-service with the following code, it looks like it is running, but I cannot connect to it.

    # Runs ok with `dagger call nginx-service up --ports=8085:80`
    @function
    async def nginx_service(self) -> dagger.Service:
        """Returns lines that match a pattern in the files of the provided Directory"""
        service = (await self.nginx_container()).as_service(use_entrypoint=True)
        return service

    # I cannot connect to local port with `dagger call run-nginx-service`
    @function
    async def run_nginx_service(self) -> None:
        """Returns lines that match a pattern in the files of the provided Directory"""
        service = await self.nginx_service()
        await service.up(ports=[dagger.PortForward(80, 8085)])

The traces in the console are the same for both executions. WhatI'm missing. Thank you in advance for your feedback

[wintry prism]

Hello, I'm working a dagger integration

[white hill]

Hello awesome @Daggernauts
i am trying to get dagger to publish to a Zot Registry for a POC . the export works, it writes the images locally . But i need the publish to work with Zot . Here my example :

docker run -d -p 5009:5000 --name zot ghcr.io/project-zot/zot:latest

registryAddress := "localhost:5009/go-hello-world:latest"
// Build options to publish to Zot Registry
publishOptions := dagger.ContainerPublishOpts{
    MediaTypes: dagger.ImageMediaTypesOcimediaTypes,
}
imageRef, err := finalImage.Publish(ctx, registryAddress, publishOptions)

stucking in this error : 2025/03/03 15:12:50 Error exporting image: input: container.from.withFile.withEntrypoint.publish failed to export: failed to push localhost:5009/go-hello-world:latest: failed to do request: Head "http://localhost:5009/v2/go-hello-world/blobs/sha256:0ffa5471c7887130be574a89502e3c5817b6ae7d1d5a763b37f1f3b834b3883b": dial tcp [::1]:5009: connect: connection refused

[plucky ermine]

Zot

[sharp zealot]

Module development challenge 🙂

[restive shore]

What is the recommended Go module to use? Is it the one in the dagger repo? The one by @latent trellis (0.9.0) is featured in daggerverse.dev but not the one in the dagger repo (0.16.2)

[latent trellis]

Am I supposed to answer that? 😂 I mean, I’m certainly biased.

All I can say: I stand by my Go module

[restive shore]

haha! IDK who should answer it lol

I am having to build my own so I'll take inspiration from both.

[wintry prism]

https://tenor.com/view/grandpa-both-confused-the-simpsons-gif-11891642

[obtuse glacier]

Is it possible to get live logs from a dagger pipeline?

[plucky ermine]

What do you mean by that? Can you give us an example?

[obtuse glacier]

    cli.Container().From("alpine:latest").
        WithExec([]string{"echo", "hello world"}).
        Stdout(ctx)

This only gives me the logs once its complete, I'm trying to be able to live stream the logs

[restive shore]

Is there a way to get the git info from a module pulled remotely? For example,

dagger call -m github.com/myorg/mymodule@pull/123/merge --help

I'd like to get the PR target, source, branch name etc. All of which dagger should technically have access to?

[heady pewter]

TL;DR; What’s the best idiomatic way to make some host configuration available to the engine runtime.

I am testing dagger to replace our entire buildchain (including CI/CD). As the final step of my journey, I’m trying to deal with is pulling and pushing stuff (libs, container images, etc…) from/to private registry.
Most of what we’re using can leverage providers sdks for authentication which would be really nice to be able to leverage inside the dagger runtime.
However they often require some sort of host config to be accessible (e.g. to leverage Workload Identity on a GKE cluster). Afaik dagger runs inside a containerized environment, where those config are not available. If I try this, I will receive an error, despite the same code running if using bare go on my host:

creds, err := cloudcreds.DetectDefault(&cloudcreds.DetectOptions{
        Scopes:           config.GCRScopes,
        UseSelfSignedJWT: true,
    })

because the google credentials library cannot find the required configs in the dagger engine runtime. Which totally makes sense to maintain the isolation.

I found some workarounds, like skip WithRegistryAuth and let dagger use your host container runtime configuration. However it can quickly become limiting. It comes short on GitLab k8s runner, with dagger engine configured as a DaemonSet. I’m gonna test binding a service account and modifying the deployment with an init container using the credentials helper, BUT:

1. I feel this is not gonna work, especially if dagger engine uses the host container runtime
2. It binds the runtime configuration to the engine deployment, which feels meh to me.

If we had a way to make the configuration (auth management included) fully dagger, this would make our buildchain really robust, standalone, clean and maintainable. Everything we do lack with our current system (yaml templates).

Sorry for the long post, happy to give more details as needed!

[sharp zealot]

Yeah you definitely don't want to mess with the engine container itself, it will make your life miserable down the line (breaks separation of concern between app and infra layers).

We have lots of plumbing for either 1) injecting client-side configuration (eg. your client machine's google cloud credentials) into the dagger sandbox contextually, or 2) fetching credentials dynamically from another dagger function. I think a combination of those two approaches should help.

For 1: when passing an argument of type Secret via the command-line, there are conveniences for choosing a "secret provider" by using a URI format:

• env://... -> load secret from env variable (default)
• file://... -> from a file
• cmd://... -> by executing a command
• vault:// -> get from hashicorp vault
• op:// -> get from 1password

-> We plan on adding more providers in the future

These are all resolved in the context of the client machine

Separately, I believe for native docker pull/push specifically (Container.from(), Container.publish(), Container.build()) the client system's docker config is used transparently. So for example, if you already have a valid docker config on your client machine, possibly via calling google cloud auth helpers, dagger will automatically use those.

We have a similar capability for ssh (this is how Dagger can load modules from private git repositories, it just picks up your system's ssh config, including exotic PAT/certificate configurations)

[restive shore]

Is there a way to get the private git auth into a container? The engine gets it from the host but I would like to use that same auth inside a container. Is that possible?

[sharp zealot]

Not yet but I believe it is planned? cc @void widget

[restive shore]

One more question, what's the equivalent of running up from the CLI? I tried returning the following but it didn't work. I can't access it from my host.

Up(ctx, dagger.ServiceUpOpts{
        Ports: []dagger.PortForward{{Backend: 8443, Frontend: 8443, Protocol: dagger.NetworkProtocolTcp}},
    })

[sharp zealot]

It's because you're exposing it to the current client - ie your module's runtime container

For sandboxing reasons, a function can't bind a port to the host's network stack. It's similar to eg. Directory.export. you have to return the service, and the top-level client calls up

Example for running dagger docs server from source; dagger -c 'github.com/dagger/dagger | docs | server | up'

[restive shore]

understood. makes sense.

[void widget]

Mmmh, just wanna make sure we're talking about the same thing -- you can mount the socket inside a container and pass it around. This is actually discussed on a docs improvement right now ahah:

• https://github.com/dagger/dagger/pull/9770/files#diff-1930eef6f3d16cd04006f36c7b76983ac1916d8d375bab5e946ae9aaa0903d00R212-R217
• https://github.com/dagger/dagger/pull/9770/files#diff-d09b6c7ade995e6fd330a4aa0593b0765364d8b14141268a4b00cd5f4053026dR1-R18

[void widget]

And, you could pass the PAT as args of that dag.Git() too ? But it would mean passing them around from the main module's arguments

[restive shore]

I am not sure it's the same thing? 1. I am using PAT, not SSH

2. Here's the scenario.

• I am mounting a directory which is a git directory into a container
• I want to be able to do a git fetch -a from within that container. But that requires credentials

So, I guess my question/request is, since dagger engine already has git creds, why can't they be passed down to the container?

Docker auth is passed through this way afaik, because docker in docker works without any additional auth

[void widget]

Mmmh, are we talking about a top-level module or any module ? It's also related to the security-by-design model of Dagger (same docs): https://github.com/dagger/dagger/pull/9770/files#diff-a207fece057659ec21e006a5557c2249e8cb086e2b4da6f04d613848e3d6535fR12-R19

--
oh so an .WithExec that does git fetch -a -- as the engine properly requested the credentials -- the security-by-design is still maintained, but in that given step, basically it shall be passed down? I see, it does make sense

[restive shore]

Right! At that point the engine already has the creds. Allow me to use them. Otherwise I'll have to request the user pass their PAT to the module.

[restive shore]

it can maybe be integrated with this also? https://github.com/dagger/dagger/pull/9730

[restive shore]

Question for @thorn moat , right now, the TUI doesn't show image download progress unless I zoom in. Is that intentional? I remember it used to show that. The reason I ask is, it kind of feels like the progress is hung for a while till docker images (especially large ones) are pulled. Is it possible to show that progress and hide it?

[thorn moat]

yeah that's a total gap in the UX at the moment, i want to fix it but have never had time to go back to it. It's been like that ever since the switch to OTel

not sure what you're referring to by 'progress' at the moment though

[restive shore]

all the HTTP GETs

[thorn moat]

from what i can tell the various lower-level image pulling spans end up showing up at whatever call site unlazies the fetch, which is even more confusing than usual

yeah ok

[heady pewter]

Thanks! I’m gonna try making something with Secret argument. However there’s still the issue of ttl of that secret for my use case. If generating the secret on the client with Google Workload Identity, the token is short lived (this is the very purpose of that). This should not cause any issue for my specific use case, but could hit us in case of long running functions (multiple build and push at once). Being able to generate the token directly from a dagger function at the right time would be more robust.
However I’m not quite sure this would be technically possible. If I understand correctly, in my use case (engine not running on the same host), if generating the token inside a Dagger function (rather than generating locally and passing as a Secret), this would require the engine itself to have the correct IAM binding. The engine being shared by multiple jobs, this is not ideal in terms of security. We would quickly have to give it a wide set of permissions to be able to handle all different jobs.
A gcp Secret provider would be useful, but if I understand correctly it would not solve the issue but just make the UX easier. I guess it would still be generated prior to sending the context to the engine, and still face possible expiration when actually needed.

When you say

the client system’s docker config is used transparently
In the context of distinct hosts for client / engine, would the client you’re talking about refer to the engine, or to the actual caller?

[sharp zealot]

note: secret providers are executed on the CLI side but they are called lazily on demand

not sure about caching if the secret is called multiple times in the same session. I believe token refresh is perhaps a known issue? @dense canyon @rocky harbor

[sharp zealot]

When you say
the client system’s docker config is used transparently
In the context of distinct hosts for client / engine, would the client you’re talking about refer to the engine, or to the actual caller?

actual caller (on a per-session basis, so: multi-tenant)

[rocky harbor]

I don't think any of the secret providers support token refresh but @chrome pilot @warped canyon could confirm

[heady pewter]

Ah! Well that solves the problem for container images then. Runner job with a docker auth helper is quite idiomatic on our side, we have to do it for all tools not supporting GAR/GCR natively (basically any tool but kaniko 😅)
I’ll dive deeper in that secret provider, this would still be useful for dependency management and cie

[warped canyon]

Correct, nothing built in for token refresh today, but I don't think a ttl would be too hard to add

[heady pewter]

Me again (sorry I'm a bit spammy). Now that container auth worked (thanks @sharp zealot 🙏 ), I'm playing with the secret provider to solve the package management (go module for now). I've succeeded using cmd secret provider (with different commands between local and k8s with workload identity enabled).
I'm trying to make the UX more user friendly by providing default value depending the engine environment but I'm unsure how.
This

 func New(
    ctx context.Context,
    // +optional
    // +defaultPath="/"
    src *dagger.Directory,
    // +optional
    gcptoken *dagger.Secret,
) (*Srs4Handler, error) {
        // ... constructor stuff
    if gcptoken == nil {
        gcptoken = dagger.Connect().Secret("cmd://gcloud auth print-access-token")
    }
        // ... more constructor stuff

would run the command on the dagger engine itself instead of the host. I faced a 127 so I do think this confirm run from the engine. Which makes sense since the Connect returns an instance of engine client. Is there a way to reproduce what dagger call --gcptoken=cmd://"gcloud auth print-access-token" funcname would do (run the command on the cli)?
Sorry if that's a dumb question, I did not have time to dive deeper in the code/sdk doc yet, but I'll do my homework asap.

[sharp zealot]

It's not possible for a dagger function to specify default secrets; we want to add convenience but need to be careful of not breaking sandboxing & portability.

We're looking at natively supporting .env or something equivalent so that you can easily pass configuration profiles around

[heady pewter]

Ok copy, that was just pure UX anyway, this does not block me. I'm not gonna die having to write the arg for now. Still light years better than

[sharp zealot]

wait you have an "old man yells at gitlab" emoji that's hilarious 😂

[heady pewter]

Wait that's the dagger server nah? But anyway our slack has you covered too

[restive shore]

Do you want me to open an issue to track this?

[manic lynx]

Hey!

Anyone around to hop on the #911305510882513037 channel and answer some quick fire Dagger questions for me? it's about custom types, and trying to understand some golang as well.

This is the module I'm using as a reference: https://github.com/sagikazarmark/daggerverse/blob/c964ee26f982c4db0282523cd06f75ecb7e1102f/kafka/main.go

https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/kafka@c964ee26f982c4db0282523cd06f75ecb7e1102f#Kafka.singleNode

I'm super lost now

[sharp zealot]

@manic lynx can't join a call at the moment, but happy to try and answer async here

[manic lynx]

So .. here is my module: https://daggerverse.dev/mod/github.com/dagger-php/phpstan@59986887100c53a772fe63d2c7ae1d62bbb4132a#Phpstan.analyze

I'm trying to build up some command line arguments for the main withExec()

https://github.com/dagger-php/phpstan/blob/phpstan-options/src/Phpstan.php#L58

so the main with command line call is like this

vendor/bin/phpstan -v analyse --memory-limit=2G --level=6 ./app

memorylimit and level are 2 arguments I want the user to be able to set, before running the main analyze() function here, https://github.com/dagger-php/phpstan/blob/phpstan-options/src/Phpstan.php#L41

How could I do this, from a syntax perspective (the main application calling this module)

This is the main app syntax I was trying to go with

dag()->phpstan()->memoryLimit('2G')->limit('5')->analyze('8.4', $this->source, './app');

analyze() returns a Dagger Container, but when I call analyze() the data set from the memoryLimit func and limit() func are "lost"

when I call limit() it sets state on the class, https://github.com/dagger-php/phpstan/blob/phpstan-options/src/Phpstan.php#L34

but on analyze() that $this->options arg is empty array.

It lost the state ...

Am I making sense?

[warped canyon]

based on what you're describing, it sounds like the commented out bit

$this->options['memory_limit'] = $limit;

return $this;

is what you'd need? I'm not as familiar with php syntax

[manic lynx]

Ignore memoryLimit() function please ... it was like that before, but it was empty, so I was hacking around .. look at level() function please,

https://github.com/dagger-php/phpstan/blob/phpstan-options/src/Phpstan.php#L32

EDIT: refresh the page, I reverted those commented out lines and it's back to normal now.

[warped canyon]

Got it. Yeah on the chaining side that looks good to me

[manic lynx]

If you were building this in golang, what syntax would you use to set memoryLimit state and level state, before calling the main analyze() function that does all the from()/withExec()s?

It's kind of like setting global state on the dagger object (the module) and then referring to that in a future dagger functoin ( analyze() )

Is this menat to work ?

[warped canyon]

it would be like

func (f *Foo) WithBar(bar string) *Foo {
  f.Bar = bar
  return f
}

which looks like what you have

[manic lynx]

Would you make this a dagger function?

[warped canyon]

python for example https://github.com/kpenfound/dag/blob/main/workspace/src/workspace/main.py#L82-L90

[manic lynx]

ok, that's a yes. let me annotate this with dagger function. one minute.

my module -

https://daggerverse.dev/mod/github.com/dagger-php/phpstan@59986887100c53a772fe63d2c7ae1d62bbb4132a

the sample "copy command" shows this

dagger install github.com/dagger-php/phpstan@59986887100c53a772fe63d2c7ae1d62bbb4132a

but I want to install from a specific branch phpstan-options

https://github.com/dagger-php/phpstan/tree/phpstan-options

how can I dagger install a module, but specify a branch on the dagger install command ??

I tried this: dagger install github.com/dagger-php/phpstan@phpstan-options but it didn't work. Is it supposed to work like this ?

[sharp zealot]

Yes that should work, assuming phpstan-options is a branch of that repo

[manic lynx]

ok trying again

Okay dagger install worked..

dagger install github.com/dagger-php/phpstan@phpstan-options

{
  "name": "laravel-2",
  "engineVersion": "v0.16.1",
  "sdk": {
    "source": "php"
  },
  "dependencies": [
    {
      "name": "phpstan",
      "source": "github.com/dagger-php/phpstan@phpstan-options",
      "pin": "603d69c82931f153611553439baef9934dfcc0d3"
    }
  ],
  "source": "dagger"
}

I just pushed some more commits to my github repo branch .. what's the command to update to the latest commit on my branch?

This errors out: dagger update github.com/dagger-php/phpstan@phpstan-options

[sharp zealot]

Should not error... You can also try dagger update, or dagger update phpstan

[manic lynx]

ok

@sharp zealot @warped canyon I got it to work 😄

I'm cleaning up stuff, prepare for sexy syntax

@sharp zealot @warped canyon

So I made an "arg builder class"

[warped canyon]

super nice 🤩

[manic lynx]

Yea, the DX is important .. this is internal to the module itself .. ☝️

this is how the end-users now call the module, from their applications

same DX super clean and sexy

[restive shore]

I am trying to do something simple but I have to do it in a convoluted way. I have a directory (git) and it's checked out to a specific branch. I want to fetch the main branch for the same dir/repo. The way I'm doing it is creating a new container that has gh cli and git and doing gh auth login and then adding additional config to the GCM to store my creds and then doing a git repo fetch origin branch. Seems like an awfully roundabout way given that I can easily do a dag.Git().WitAuthToken()... But there is no way to fetch a branch. Maybe I'm missing something simple.

[manic lynx]

Here is the internal module code, in the end I called it PhpstanArgBuilder

with a private makeCliArgs function .. that should do the trick 🙂

Thanks for the help @warped canyon and @sharp zealot

EOF - good night 🙂

[warped canyon]

I might be missing what you're asking, but wouldn't it be dag.Git().WithAuthToken().Branch(branch)?

or you want both branches fetched in one directory?

[restive shore]

Yes, both

It's for Sonar. It needs the reference branch to be fetched to compare

My above request for @void widget is related to this. I shouldn't even need to provide the auth token as the engine already has my local git context. But For now, I'm trying to achieve this with the token.

maybe I need to figure out the pieces I need from the .git dir and merge them? IDK

[warped canyon]

yep makes sense! In that case what I've been doing is basically what you said before. Get in a container with git and do a fetch

[manic lynx]

Is it possible to do dagger install without specifying any commit hashes or branches? and it'll default to main latest commit?

I'm working on the README and trying to come up with something simple that won't ever really change

[restive shore]

But this doesn't work without creds.

I guess I end up having to do the full config inside the container

[warped canyon]

yep defaults to head!

[warped canyon]

ah right I haven't run into that myself.. maybe there's a way to attach the ssh socket or something?

[restive shore]

Not ssh 😦 PAT

[warped canyon]

yeah 😬 i think what you described is probably how I'd do it too then

[restive shore]

Things are so much easier on the open Internet 😅

[warped canyon]

indeed 😅

[manic lynx]

I'm still here @warped canyon @sharp zealot

I've been wired into improving the DX on this module, to the best possible, and I've wrote an entire comprehensive readme file 😄 😄

How's the readme looking? anything else that is "standard" that I'm missing?

https://github.com/dagger-php/phpstan/tree/phpstan-options

I want to keep the left module, but remove the right module, from the daggerverse .. how can I delete it ?

[void widget]

please 🙏

[void widget]

pinging @wintry prism for that

[restive shore]

https://github.com/dagger/dagger/issues/9858. Hope I incuded enough info. Happy to provide more details, or test.

[restive shore]

Very excited for this! https://github.com/dagger/dagger/pull/9860 (dagger develop --recursive). I am guessing this makes it worthwhile to create a top level module that imports all sub modules for a "daggerverse" repo. Right now the modules are their own bubble in the folders. Is that the idea? Or am I mis-reading this.

[deft rain]

lol that was quick

uhhh I had not considered this

to do multiple disconnected modules, I'm not actually super sure what we want to happen

[restive shore]

today I run a command like this find . -type f -name "dagger.json" -execdir sh -c 'dagger develop' \;. Which is super slow and keeps getting slower when we add more modules. At some point we may stop updating everything at the same time? I like keeping them in sync for now.

I'd rather use a dagger module to do this but I can't tell it to modify my host folder in code because of the sandbox

[latent trellis]

I do this in my daggerverse repo: https://github.com/sagikazarmark/daggerverse/blob/main/justfile#L67

It's not superfast, but it's managable.

[manic lynx]

Curious what you are specifically keeping in sync with dagger devlop

[restive shore]

That's the only way to upgrade a dagger version of a module afaik. If I wanted to bump a version from 0.16.2 to 0.16.3 I have to run dagger develop within the module

[restive shore]

I am not familiar with justfile syntax. Isn't that doing the same thing my find command is doing?

[latent trellis]

I guess it does

[latent trellis]

It updates module dependencies in go.mod file (for Go modules) and updates the minimum Dagger version to the latest. It's useful if you are relying on features introduced in a newer version (forces consumers to update as well instead of randomly breaking).

[manic lynx]

Cool Cool, thanks

[meager relic]

I've been working on "migrating" my Nx monorepo to Dagger.

I'm not ditching Nx, but I've gotten burned by them a couple times because caching isn't as deterministic as I'd like, and it is very easy to accidentally use local files in ways that don't reflect the final build. I'm still happy to use it locally (its much faster, see stats at end, but even in a world of uber-optimization, it is simply faster to re-use existing workspace than continuously spin up virtual environments) but I want a more robust system that can trade speed (to an extent) in favor of correctness when it comes time to do the final review/merge.

Dagger CI seemed like a perfect answer to the deterministic build part.

I wanted to share some of my experience with Dagger so far (both the good and bad).

[restive shore]

IDK if this is specific to 0.17.0, but it took 4.5 mins at finding module configuration the first time I ran it for one of the modules in my daggerverse repo.

after the first run it's almost instant. It wasn't downloading any images either, I zoomed into max verbosity.

[restive shore]

hey @thorn moat , sometimes in the TUI the seconds tickers are beyond the screen width and can't be seen. It makes it feel like the TUI is hung.

[thorn moat]

there should probably be a global timer - that way you can at least see that one still incrementing

and/or I can do some sort of wrap-around thing and put the timer on a second line

[restive shore]

I think a global timer would suffice

[warped canyon]

if I return a list of []MyObject, how can I chain on that in shell? is there no way to select an element?

We have docs for Array arguments but not return values

[restive shore]

is there a way to increase verbosity in shell?

and also, is there a way to re-load a module when I make changes? Right now I exist and re-run shell

@thorn moat , my shell is still showing werid colors

This is in input mode. I think it's just the area which is focussed

[sharp zealot]

yes you can press + and - during execution

[restive shore]

ah so by going into nav mode.. but my execution ends so fast I don't have a way to get to nav mode before it ends

[sharp zealot]

normally if you press escape you can navigate up to previous spans, and then press + I think? Or you can set the verbosity before running the command

[restive shore]

Or you can set the verbosity before running the command
This is what I'm looking for. Adding -vvv didn't work (this is in the dagger shell)

[warped canyon]

Yeah you should be able to enter nav mode and change verbosity before or after running the command. After will still increase it for the previous commands

[restive shore]

hmm, the TUI is still a bit awkward to navigate. I usually have trouble getting to the info i need. As you said I can increase the verbosity, but with the bad highlight I posted above it's straining on the eye. Granted, it's probably a highlight from my theme (everforest). i switched to nordfox and it looks better. Still I think I may be happier without any highlight or a way to disable it when running shell

[restive shore]

^^ this would help with iterating immensely

I could have sworn I saw a .load. Did that get removed?

[sharp zealot]

I think it's called .refresh now?

[thorn moat]

eh yeah might just need to get rid of it and have some other focus hint, too much variance between themes :/

[restive shore]

Is there a way to "load" a module into dagger shell? I started the shell in the root dir, but my dagger module is in a subdir. Do I have to start dagger shell in the subdir instead?

I can type modulename and it pulls in the module, but it doesn't set that module as the context. So I have to prefix all commands with modulename

[upbeat herald]

dagger -m ./<mod-dir> should work

-m selects a module to load, it can either be a git ref or a local path

[restive shore]

gotcha, so you removed the ability to set the module context from within the shell? What's the reason? Seems useful

[kind carbon]

Use .cd modulename.

[restive shore]

oh! that works!

[kind carbon]

There's .cd, .ls, and .pwd. They're just hidden so it gives us more time to think about the DX.

[restive shore]

gotcha

[kind carbon]

You can .cd into any path, it's just tied to module load like direnv. I.e., if you cd into a directory that's in a module, it will load it.

Note that when you use Directory/File arguments (as a path), it will be relative to your .pwd.

[restive shore]

makes sense.

[sour citrus]

Is there a way to say "I want to delete a cache volume"?
I experimented about dev envs using dagger. So at some point I open a terminal, let the user do what they want. But then I'd like to get back the modification made to the files.
My understanding is Terminal can't do that. You can't get the modifications, and the trick is to use a cache volume (maybe there's a better way?)
So I did that:

WithMountedCache("/terminal", dag.CacheVolume(uuid.New().String())).
WithExec([]string{"cp", "-r", "/workspace", "/terminal"}).
WithWorkdir("/terminal/workspace").
Terminal().
WithExec([]string{"cp", "-r", "/terminal/workspace", "/out"}).
Directory("/out")

And that's works 🙂
But I'd like to remove the cache volume, because that's just a temporary thing that should not exist outside of this function. Is that possible?

[crude gazelle]

Referencing this page: https://docs.dagger.io/api/custom-types

~~The examples show making a call of dagger call dagger-organization members url

The snippets have a property called members but no method to get it.

If I'm understanding this correctly, do Go, Python and Typescript SDKs implicitly add getter functions for public variables?~~

EDIT: Ignore me, I've looked at it more now. It does seem to be the case... Something else on the PHP SDK ToDo list then 😅

[restive shore]

in dagger shell, how do I stop execution of a dagger command midway? if I go into nav mode and press q it quits the shell altogether.

[warped canyon]

it should be ctrl + c (from input mode), but I've found sometimes certain individual execs aren't cancellable and you have to wait for that to finish before the function call gets cancelled

[restive shore]

hmm, i tried ctrl + c and it didn't work. Let me try again

huh, ok I'm seeing something strange. When I press ctrl + c it sends a context cancelled and the progress stops. But when I re-run the same command, the log doesn't progress, it re-starts the previous (stale) spans. So it's off screen for me, I have to navigate to see them.

[dense canyon]

👋 if you mv the data away of the cache-volume instead of running cp, that will effectively free it up. You can additionally run any rm commands you find necessary.

[blissful shore]

Hey all, I was interested in adding an option for enabling extended thinking in Claude.

The change itself is pretty trivial, the Thinking attribute in anthropic.MessageNewParams in the llm_anthropic.go would need to be supplied similar to the Anthropic SDK example.

I'm just not quite sure what would be the best way of allowing a user to enable Claude with thinking. What is the best way to pass an argument on whether they want to enable thinking, and the max tokens they'd like to use?

[sour citrus]

I'm trying to have interaction with the user while calling dagger functions. As a kind of real example, think about a command that will open a PR based on a local repository the user provides. An LLM for instance can prepare the PR body based on the different commits.
But I'd like the user to review and edit this body.
I might be wrong but I haven't seen a way to prompt a user.
So I tried by hacking around terminal:

func (l *LocalAgent) Write(ctx context.Context) (string, error) {
    return dag.Container().
        From("alpine:3").
        WithMountedCache("/workspace", dag.CacheVolume(uuid.New().String())).
        Terminal(dagger.ContainerTerminalOpts{Cmd: []string{"vi", "/workspace/file.txt"}}).
        WithExec([]string{"mv", "/workspace", "/out"}).
        File("/out/file.txt").
        Contents(ctx)
}

It almost start, the editor opens, I can write anything I want. But the exist doesn't work and I got a panic.
Before I dig deeper in this aread, is there a better way to do that?

[sharp zealot]

This is how I would do it too 🙂 I'm surprised that you get a panic. What's the error?

[sour citrus]

This is how I would do it too 🙂 I'm

[latent trellis]

I'm trying to diff two directories coming from different sources, but I get this error:

cannot diff with different relative paths: "/" != "/work"

If I hack the paths and put the original directory under a work subdirectory, fetch it and run diff on that, it works.

Is this a known limitation or a bug?

[deft rain]

🤔 yeaaa, it's a limitation, the hack you mention is the way we workaround it too

but honestly, we should probably apply it transparently

[latent trellis]

Thanks for confirming. That's what I did for now.

[blissful shore]

Could I bump this in case someone has any thoughts? Or is it better to open an issue? 😄

[thorn moat]

An issue would be good! 🙏 The request makes sense, just have to figure out how to propagate that config

[restive shore]

I didn't realize this was the case before but I got this feedback from a code review. I have a function named WithOsUtilities. That translates to
with-os-utilities in the CLI and WithOsUtilities in the SDK. However, the Go naming convention is WithOSUtilities and linters will throw a warning on it.

So if I change my function to WithOSUtilities, the resulting CLI is with-osutilities and SDK is WithOsutilities. CLI is awkward but manageable but the SDK generated function still doesn't adhere to Go convention.

Is there a trick I'm missing? Or is it a design decision/limitation?

[latent trellis]

I recall opening an issue about function naming

[sharp zealot]

@void widget @muted plank 👋 I'm with someone from e2b, can you share with me the url of your e2b "local emulator" thing? Is it open source?

[sharp zealot]

🙏 🙏 🙏 https://github.com/dagger/dagger/issues/10064

[restive shore]

I found the related issue and commented on it. I hope it can get prioritized - https://github.com/dagger/dagger/issues/7091. I didn't find one from you but if you have context to add, please do so.

[latent trellis]

I'm trying to figure out if it makes sense to mount default cache volumes in reusable modules or if they should always be supplied by the caller.

For example: my Go module currently mounts both module and build cache by default with an option to disable them in the constructor and a way to provide custom cache volumes.

Previously, this proved to be challenging with Dagger Cloud caching, because concurrent jobs (ie. concurrent dagger calls) stepped on each others toes and the last one in always overwrote cache volumes with the same name.

I know that there were plans to namespace cache volumes per module, though I'm not sure how that would solve this particular problem. Also, I know the distributed cache is disabled at the moment, but I assume there will be a new solution introduced at some point.

Given all that, would it make sense to keep default cache volumes risking concurrent jobs may interfere with each other or the user should be forced to set cache volumes manually (when necessary?) to make sure cache volumes are unique per project?

[deft rain]

so cache volume namespacing is definitely a thing now

so you can set your own names without fear of overlap

the overwriting problems is definitely still a very real thing - currently there's not really a way to get different behavior. we have discussed previously that you might somehow be able to extend a cache volume and provide your own cache "merge" logic?

but yeah, this is still a bit of an uncharted space, a lot of the work @rocky harbor and me are doing right now is around moving more of the cache logic out of dagger into buildkit itself, so we can actually control these kind of things - instead of having to rely solely on upstream

[latent trellis]

How does that work in practice?

I'm mostly interested if module A calls module B and module C calls module B, will the cache volumes created in module B be namespaced according to the module hierarchy?

[latent trellis]

Do you have a sense of how the final solution is going to look like? Which direction looks more likely: users will always have to name their cache volumes uniquely or there will be some additional, per project namespacing? Trying to figure out which direction to go with modules, what should be the default.

[sharp zealot]

@latent trellis so far the most promising architecture seems to be to separate volumes and layer cache as distinct problems.

• Volumes are best handled at the cluster level. They may get generalized to support external storage backends such as nfs, sshfs etc (there's an active issue for that)

• layers on the other hand can be replicated globally, majority ongoing R&D by @deft rain and @rocky harbor to solve that problem once and for all, once that's done it will unlock a myriad options, from DIY cache export that actually works, to magicache v2 in Dagger Cloud that blows away the performance of v1. BUT all of that is layers only

My recommendation when making modules today, is to create cache volumes by default, and allow for optionally overriding your own via constructor args. whatever changes roll out later, they are still months away and will necessarily have good backwards compatibility or easy migration from that pattern, since it's the dominant pattern today

[latent trellis]

Thanks!

That's what I'm doing today. The tricky part is when the module has a dependency on another module with cache volumes.

The "nice" solution would be passing module B to module A, but that's not possible today.

I either expose module B's constructor arguments as well or just pass in a container. :/

To clarify: module A is golangci-lint, module B is go.

[sharp zealot]

I forgot to add that we already added a hook for infra operators to customize cache volume storage in the engine image. will add more things like that when possible

[sharp zealot]

Mmm you mean you want 2 modules to use the same cache volume for their respective work?

[latent trellis]

The opposite: I want to make sure that two instances of the same module use different cache volumes.

The problem I often have is two parallel jobs overwriting the same volume.

Or had...

But I guess that falls under the case where people should specify their own cache volumes.