daggernauts

[sharp zealot]

also can’t ctrl-C out, I have to kill my term

related message by @rocky harbor : #1167558851395076177 message

[dense canyon]

Cache flushing happens when the engine is stopped. Don't think that's the reason

[dense canyon]

Oh wait, it could be telemetry flushing.. cc @chrome pilot .

[wind coyote]

Sorry for the really dumb question but does dagger mod use work across SDKs yet? Could I write a HelloWorld module in Go and use it in Python?

[sharp zealot]

yes 🙂 also it’s mod install now

also: definitely not a dumb question

[wind coyote]

Cool! Imma try that out now 🙂

[sharp zealot]

I haven’t tried it yet, let me know how it goes!

[wind coyote]

Do I need to build from git or is this in the latest published binary?

[sharp zealot]

good question, I think it should all work in 0.9.3

[wind coyote]

good question, I think it should all

[latent trellis]

What do people do to update modules these days? What's the dagger mod equivalent of go get -u?

[sharp zealot]

Personally I just re-run dagger mod install FOO

[warped canyon]

should we have dagger mod install bump all dependencies with no arg?

[latent trellis]

One thing I often do with dependencies is bump them all and fix any issues that come up.

Based on the last couple days, here is my biggest (current) pain point with Dagger with regards to integrating it into a GitHub Actions pipeline: basically, I can't use env vars.

I want to use GHA env vars instead of passing arguments to my command. It's just easier, they are already there on GHA (I don't have to pass them as parameters).

Related to that, currently everything seems to be a parameter to a function. I recall seeing structs with tags as potential parameters, but now everything is a parameter as far as I can tell (even if they are optional). That's not a great experience. Even though the difference is semantic, I don't enjoy writing that code.

[sharp zealot]

I’m thinking we could encode that pattern in a Dagger module 😀

[sharp zealot]

Do you have an example of a CI config that illustrates this problem, that we could use as a baseline to discuss improvements?

[latent trellis]

Sure: https://github.com/sagikazarmark/demo-cloud-native-rejekts-na-2023-dagger/blob/main/ci/ci.go#L20-L26

[latent trellis]

Also related: #1170467071684001833 message

[latent trellis]

Here is how I call it from GHA: https://github.com/sagikazarmark/demo-cloud-native-rejekts-na-2023-dagger/blob/main/.github/workflows/release-dagger.yaml#L25

For most of those I could just use these env vars: https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables

[sharp zealot]

@obtuse lion you ask, Zenith delivers 😄 https://twitter.com/solomonstre/status/1721399519835725852

[obtuse lion]

wow, that works? You are just squashing the layers on top of each other?

[sharp zealot]

Yeah it uses the magical llb MergeOp under the hood

Honestly I didn't expect it to work so easily, and with so little additional code

Next I want to apply the same pattern to installing actual APK packages. Same thing apko does - download and merge packages without any exec - but with MergeOp. The hardest part there is figuring out how apko gets the dependency graph, and downloading the package data. Plus probably platform management, that's always universally a PITA

Neat little detail @obtuse lion , in my command I pass image refs as arguments to --image, but the function doesn't actually take a string as argument: it takes a container. It's just the CLI dans encodes the string argument as a Container().From() for convenience. So you can wrap the same function and pass it any container object you want as overlay, and it will work the same

[obtuse lion]

That's pretty slick. so this is what your wolfi looks like in AML. To be clear, I have no idea if this AML thing is a good idea, I'm just fooling around. But you notice in the Container function how funky and terrible the looping is, since this is a purely functional language. I'm trying to figure out a better syntax short of just doing reduce

Base: function Config {
    return: {}
}

define Config: {
    overlays: [dag.Container]
    packages: [string]

    WithPackage: function Config {
        args: package: string
        return: self + {
            packages: self.packages + [args.package]
        }
    }

    WithPackages: function Config {
        args: packages: [string]
        return: self + {
            packages: self.packages + args.packages
        }
    }

    WithOverlay: function Config {
        args: image: dag.Container
        return: self + {
            overlays: self.overlays + [args.image]
        }
    }

    Container: function dag.Container {
        init: dag.container().from(baseImage)

        for i, overlay in self.overlays {
            if i == 0 {
                ctr: init
            } else {
                ctr: prev.ctr.withDirectory("/", overlay.rootfs())
            }
        } else {
            ctr: init
        }

        if len(packages) > 0 {
            return: ctr.withExec(["apk", "add"] + packages)
        } else {
            return: ctr
        }
    }
}

[sharp zealot]

First version of a basic docker-compose module:

(
  export DAGGER_MODULE=github.com/shykes/daggerverse/docker-compose
  dagger call example services name
  dagger up example service --name db up
  dagger shell --entrypoint=/bin/bash service --name db container
)

[obtuse lion]

Is the docker socket available to the module?

[sharp zealot]

No, it's a pure Dagger reimplementation. Of course compatibility won't be as good that way, but I want to see how far I can get

[obtuse lion]

oh interesting, there's so little code I just assumed it was still using docker under the hood.

[sharp zealot]

Ha ha 🙂 Well it doesn't do much yet... But yeah the impact-to-LoC ratio is looking pretty good

Here is the logic for using the image or build fields: https://github.com/shykes/daggerverse/blob/main/docker-compose/main.go#L123-L142

[obtuse lion]

The squashing with wolfi doesn't seem to fully work. It will fail with dagger call --progress plain -m github.com/shykes/daggerverse/wolfi base with-overlay --image cgr.dev/chainguard/go with-overlay --image cgr.dev/chainguard/git with-package --name bash container

[sharp zealot]

Huh. I guess something in those overlays breaks the base image's ability to call apk:

$ dagger shell -m github.com/shykes/daggerverse/wolfi base with-overlay --image cgr.dev/chainguard/go with-overlay --image cgr.dev/chainguard/git container
# apk                                                                        
Segmentation fault   

goes away if I remove the 2nd overlay from the command

Oh, looks specific to the git overlay

[obtuse lion]

fwiw, I'm finding the "container" method pattern your doing there useful. Basically don't return containers everywhere, but keep it as an internal state and only expose it when asked for.

[obtuse lion]

I wonder if there is a ldconfig post install hook

[sharp zealot]

Replacing git with curl, for example, works:

dagger call --progress plain -m github.com/shykes/daggerverse/wolfi base with-overlay --image cgr.dev/chainguard/go with-overlay --image cgr.dev/chainguard/curl with-package --name bash container

[obtuse lion]

I think it's because go is using glibc and git is using musl. Stupid musl

yeah, switch to git:latest-glibc and it works

that's super cool

[strong ingot]

FYI in case dagger mod publish doesn't work for you: https://github.com/dagger/dagger/issues/6066

[deft rain]

@obtuse lion not sure if you worked around this, but i've got an open pr to actually fix this so you can do what you're suggesting 🙂 https://github.com/dagger/dagger/pull/6057

[obtuse lion]

I really logically struggle with method chaining. My personal experience is that the benefits of fluent apis are very niche. A style that should be very sparingly used. I’m finding with dagger that I’m writing a lot of Setters (WithX). The authoring side is a lot of boilerplate (in my AML sdk I removed the boilerplate and had the sdk generate it, that was nice) and then on the consumption side I feel it’s equally verbose. Lots of WithX calls. So specific for this setter use case I wonder if there something could be done. Like I’d like to just create and object from a struct that already has most fields set.

In my first pass at my aml sdk I actually tried some hack where I tried to translate a variable assignment into a WithX call. But I felt I was probably going to much against the grain so I dropped that.

[latent trellis]

FYI I'm gonna play with replacing toplevel With* functions with From*:

• FromVersion
• FromImage
• FromContainer

Lower layers may still use With*, but IMO this signals better that you are initializing something. Don't know if this is a good idea or not.

[obtuse lion]

I do think WithX is a strong existing pattern seen in builder APIs. So I immediately understand what With means. I'm actually not really sure what I'm asking for but if you look at solomons wolfi https://github.com/shykes/daggerverse/blob/main/wolfi/main.go#L15 the Config here is mostly a data structure and it just seems like a lot of effort to create all the WithX method and then a lot of stuff the consuming side to set some fields

[latent trellis]

I found that the WithX pattern makes a lot of sense when subsequent calls depend on each other. For example: WithContainer().WithSource(src).

But that's my biggest issue with the current flat API style found in most modules. I could do something like this: WithContainer().WithSource(src).WithContainer()

That's why I started experimenting with a little bit different style: https://github.com/sagikazarmark/daggerverse/blob/main/spectral/main.go

[obtuse lion]

So in my AML sdk I made WithX methods auto generate. So all of that code becomes the following in AML

let baseImage: "cgr.dev/chainguard/wolfi-base"

Base: function Config {
    return: {}
}

define Config: {
    Overlays: [dag.Container]
    Packages: [string]

    Container: function dag.Container {
        for overlay in self.Overlays {
            ctr: prev.ctr.withDirectory("/", overlay.rootfs()).withExec(["ldconfig"])
        } else {
            ctr: dag.container().from(baseImage)
        }

        if len(Packages) > 0 {
            return: ctr.withExec(["apk", "add"] + Packages)
        } else {
            return: ctr
        }
    }
}

So I made that simple and cool to write. But the consumption of this still requires. base().withOverlays(....).withPackages(...). Which seems verbose. I don't know how to fix this. Just complaining.

Does zenith support graphql input types? Not sure if that name is correct, but basically just a struct. Maybe that would help me.

Like I could then do base(&ConfigInput{ Overlays: ... , Packages: ...})

[sharp zealot]

You can pass a struct I think, there are also optional arguments

[obtuse lion]

I'm going to attempt this... in my SDK create a convention that generate a method NewX so I can write NewContainer({from: "x", exec: "...", entrypoing: "..."}) and that just translate to a bunch of WithX calls. I know ordering matters, but if I really care about order I can do subsequent WithX calls after the fact.

[deft rain]

only inline structs are currently supported - having arbitrary structs supported as inputs means we need arbitrary module types to be IDable which is a bit of a pain (but only a temporary limitation)

[sharp zealot]

I know you know this, but the WithX pattern is just one way to do things, it’s not mandatory nor is it necessarily the best pattern in every case, we’re still exploring what’s possible.

[obtuse lion]

Yeah, it's just the style of the core API so I want to do whatever is idiomatic. I'm also trying to convince myself to like it, because honestly I'm just not a fan of fluent apis, so I know my initial reaction is to not like it, but there may be great value I'm overlooking.

[sharp zealot]

zenith now has docs! Thank you @bleak nest 🙏🚀 https://docs.dagger.io/zenith/

[obtuse lion]

Would it make sense to have a default command for dagger shell dagger up? Meaning dagger shell with no args would do the same as dagger shell shell-default and dagger up up-default. Like I like the general utility of docker-compose where you don't know anything about the project and you just type docker-compose up and it does the default thing for you.

i haven't played with up yet. Hopefully soon as a start to natively integrate acorn. There's some obvious use cases I see between dagger and acorn.

[rocky harbor]

Yeah I felt the need for this too recently. Only hard part is how to support configuring the default. Could be a special name (i.e. DefaultShell), could be in dagger.json, could be an annotation on the func in languages that support that (not go…).

dagger.json might be most flexible

I suppose it’s also a bit awkward in that technically the shell, up, call etc concepts only exist in the cli. So this would be on the border of what should be in our api and what shouldn’t

[obtuse lion]

I really wonder if passing around Directory is a more powerful primitive than passing around container. What id like to do is have a directory that represents my workspace. And then I can pass that directory to things like “go build” “make”. But I see an issue with this in that I really want to pass a directory + a subdir. So that the directory returned from “go build” returns is persistent as a subdir in my parent workspace directory. Sorry if that’s confusing. Im still trying to wrap my head around maintaining state between multiple logical steps.

[sharp zealot]

If you want a point of comparison, I carry 2 directories in https://daggerverse.dev/mod/github.com/shykes/daggerverse/supergit

[latent trellis]

Generated help text with multiline godoc:

┃ Available Commands:                                                                                                                                                                                           
┃   container      Return the default container.                                                                                                                                                                
┃   exec           Run a Go command in a container.                                                                                                                                                             
┃ By default it falls back to using the latest official Go image with no mounted source.                                                                                                                        
┃ You can use --version, --image, --container and --source to customize the container.                                                                                                                          
┃   from-container Specify a custom container.                                                                                                                                                                  
┃   from-image     Specify a custom image reference in "repository:tag" format.                                                                                                                                 
┃   from-version   Specify which version (image tag) of Go to use from the official Go image repository on Docker Hub.                                                                                          
┃   with-source    Mount a source directory. The container will use the latest official Go image. 

Code:

// Run a Go command in a container.
// By default it falls back to using the latest official Go image with no mounted source.
// You can use --version, --image, --container and --source to customize the container.
func (m *Go) Exec(args []string, version Optional[string], image Optional[string], container Optional[*Container], source Optional[*Directory]) *Container {

[spiral whale]

I'm not able to publish a dagger module with local dependency. I'll create an issue today but just wanted to share it here as well. How can I publish module with local dependencies to daggerverse?

I wasn't able to load a module from github.com/aweris/gale/daggerverse/gale@1dee85484cbfba12c609ba114839977547e94c7a. Check the logs below for details.

get name: input:1: git.commit.tree.directory.asModule failed to create module from config: failed to get runtime: failed to install deps during go module sdk codegen: failed to get module dependencies: failed to get dependency mod from ref "../repo": failed to get config file from path "/repo/dagger.json": lstat daggerverse/gale/repo/dagger.json: no such file or directory

[rocky harbor]

Im not 100% sure but the issue could specifically be that the local dep is not under the other module, i.e. its path starts with “..”.

That’s still a bug, it should work and the issue will be appreciated, but i know we already have modules published that have local deps like “./dep” and they work as expected. If that’s an option for you then that could workaround (if this all correct)

[spiral whale]

hmm, I'll update the path with "./" and try it again then 🙏

[latent trellis]

We need a daggerverse badge 😄

[spiral whale]

https://github.com/dagger/dagger/issues/6080 🙏

[sharp zealot]

I think this is related to my duck typing request..

ie. it should be possible to call dagger shell, dagger up etc against any object with the right signature

[sharp zealot]

tmate is fixed 🙂 Thanks @rocky harbor for tracking down the bug (see #1170141459782062141 ).

The following now works:

# Run tmate in a simple container
$ dagger shell -m github.com/shykes/daggerverse/tmate tmate

# Run tmate in YOUR custom container (replace 'ubuntu' with your image ref) :)
$ dagger shell -m github.com/shykes/daggerverse/tmate tmate --base ubuntu

@glass zephyr this is... amazing 🤯 https://tryhelix.ai/dagger

[sharp zealot]

Doesn't seem to work on my Mac though - I'm guessing it's because I don't have an Nvidia GPU 😢

Would it make sense to have a fallback to unaccelerated?

[glass zephyr]

Yeah Linux + Nvidia only right now, @mossy hazel was saying it's hard to get Mac GPUs working inside the Docker VM. Can definitely look into fallback CPU mode though!

[rocky harbor]

Maybe we need a module that takes another module and executes it on paperspace/other remote gpu environments? Or maybe that would be an SDK? worth more thought

[glass zephyr]

Yesss

[obtuse lion]

when I create a module called Foo I get a dag.Foo() method generated. Can I adds args to that method somehow? So I can have dag.Foo("bar")

[obtuse lion]

There doesn't seem to be a way to initialize the default state of a module. Basically a constructor. The issue right now is that I have a field defined foo on my module and I want to do dagger call foo and get that field. It will be nil, there's no way to initialize the field that I can see.

[obtuse lion]

as a workaround I just created a function Self that just returns itself and it will have the fields initailized. Obviously thats no good.

[spiral whale]

Is the foo your entry point of the module? If it is your command would look like ‘dagger call -m foo bar’. And bar would be a function can return foo

[obtuse lion]

in this situaton foo is field on my module

[spiral whale]

Hmm, it would be ugly but if you name your field bar and create a function foo with default fallback value it would work

But it is ugly as hell and you would ado have bar in your module

[obtuse lion]

is the path dag.Host().Directory(".") for a module different between modules. Like one modules calls another, are they differently scoped?

[obtuse lion]

this seems to be the case. I ran into this because I though I would just default a module to use dag.Host().Directory(".") if the caller didin't specify one. That didn't work 😦

Is it possible to do a host bind mount? I though container.WithMountedDirectory would do this, but didn't seem like it

[sharp zealot]

#daggernauts message

[sharp zealot]

basically I agree 100% @obtuse lion 😁

[obtuse lion]

I can't seem to get directory or file export to work. It's weird, it returns true but I'm not getting a file written anywhere 🤷‍♂️

[obtuse lion]

I must be missing something obvious. I would expect this to copy the "/usr/bin/" directory to the host at "./test"

package main

import (
    "context"
)

type Test struct{}

func (m *Test) ExportTest(ctx context.Context) (bool, error) {
    return dag.Container().
        From("alpine:latest").
        Directory("/usr/bin").Export(ctx, "./test")
}

[sharp zealot]

zenith modules are fully sandboxed. So the “host” is actually the module’s containerized runtime.

[obtuse lion]

How do I get stuff out?

[sharp zealot]

dagger download against a function that returns a directory, file or string

[obtuse lion]

Hmmm, I have to think about that. I still struggle with the big picture of how all this plays together. It’s a very different model from “traditional CI”

[sharp zealot]

I understand, we’re still working it out also.. The constraint here is that having functions be truly sandboxed is a killer feature that helps the whole platform scale beyond simple pipelines. So we’re building the rest of the UX around that, and accepting that it comes at a price, ie. having to pass explicitly things that less sandboxed tools can do implicitly for you

[spiral whale]

The execution time for the generated GitHub actions modules has improved with the release of version 0.9.3. However, I still face an initialization time of around 11 seconds. I am exploring ways to further reduce this initialization time, perhaps by lowering the number of layers involved.

[rocky harbor]

The execution time for the generated

[spiral whale]

Interesting to see, just a small DX change effect couple of seconds of my execution time

• Single function:

dagger call run --repo kubernetes/minikube --branch master --workflow build --job lint --token $GITHUB_TOKEN --runner-image ghcr.io/catthehacker/ubuntu:js-latest sync

...

⧗ 1m3.1s ✔ 236 ∅ 46

• WithX pattern applied

dagger call run --repo kubernetes/minikube --branch master --workflow build --job lint with-token --token $GITHUB_TOKEN with-runner-image --image ghcr.io/catthehacker/ubuntu:js-latest sync
...
⧗ 1m7.4s ✔ 247 ∅ 48

Both of the execution times are best results after multiple re-try with full cache. Because of the size of the DAG of gale, every extra element significantly affects time.

[rocky harbor]

We don't support bind-mounts because they rely on the dagger client and the server being on the same host. We don't want that to be a hard requirement and we also generally speaking don't want to add APIs that only work in certain client<->server setups.

We have talked about use cases that would require more continuous syncing between host<->container rather than an import at container start and export at end. E.g. here: https://discord.com/channels/707636530424053791/1169591512133292093

Not exactly the same as a bind mount of course but would help fill certain gaps, not sure if it's applicable to you

[rocky harbor]

Interesting to see, just a small DX

[restive shore]

How do I handle dependency upgrades for modules? For example, my dagger module is using dagger 0.9.1. How do I upgrade to 0.9.3? I am running dagger CLI 0.9.3 and running dagger mod sync does not bump the version of engine. Should dagger mod support that?

[spiral whale]

Bumping dagger version for modules.

[grizzled basin]

sweet - just created my first dagger module. sure its a tiny as thing, and was/is simple as, but - a whole new world opens.

[dense canyon]

you've been dagerized! Welcome to the Zenith club 🕵️‍♂️

[grizzled basin]

If I'm not mistaken - it works as "modules all the way down" (much like turtles), and even your top level pipeline needs to be a module - since they all need a dagger.json etc. Something I think I asked about awhile ago, but wasn't quite succinctly mentioned as such - which may have been me just not grokking it then, or not actually clearly mentioned in a way such as: "previously had a program with main() which used dagger.Client.... - now you start with a module and use dag.Blah...

[sharp zealot]

yes that’s right 👍

[grizzled basin]

I like Turtles.

[restive shore]

New request (and blocker) from me! I can open an issue if this is not tracked already, We are using github enterprise (cloud SaaS) and I am creating and pushing modules to that. I can't seem to use (or install) a module directly as it would require a GH PAT that is authorized to access my enterprise instance. I would need this support to start building out modules within my company. Right now I can only use modules by pulling down the code and running dagger call. I guess dagger could honor a GITHUB_TOKEN set in the local environment? This would be kind of related to the GOPROXY , http_proxy pass-through requirement to the dagger session. On top of that I think for Go I'll also need support for GOPRIVATE. Tagging @rocky harbor as he's aware of the GOPROXY requirement.

[obtuse lion]

I think it's a good decision to not allow bind mounts. They are very tempting drug, so best to abstain.

[obtuse lion]

I'm creating the one dagger module to rule them all (queue meniacal laugh)

[spiral whale]

https://tenor.com/view/mic-drop-im-out-king-minion-king-of-gifs-despicable-me-gif-16642244

please replace GIFS to Modules 😄

[spiral whale]

What is the exact meaning of the 50 and 3 in this summary from TUI progress bar ⧗ 4.95s ✔ 50 ∅ 3 ?

[latent trellis]

Cache hit/miss?

(Just guessing)

[spiral whale]

it makes sense, my first guess is node results for DAG. I'm just trying wrap my head around how to measure size of the DAG for my executions

[obtuse lion]

I can't seem to return a type from my module that is defined in a different module:

I have module foo that is ```
package main

type Foo struct {
}

func (m *Foo) Test() string {
return "hi"
}
and then module bar that is
package main

type Bar struct{}

func (m *Bar) Test() *Foo {
return dag.Foo()
}

And then when i can `dagger call -m ./bar test test` I get the following error

┃                            
┃       | ...                
┃  2036 |                    
┃  2037 |                    
┃  2038 |         """        
┃  2039 |         test: Foo! 
┃  2040 | }                  
┃  2041 | extend type Query {
┃  2042 |         bar: Bar!  
┃       | ...                

[spiral whale]

You can return types defined by other modules directly. Transitive dependencies are not included when you're using a module. What you can do is forward your call to Foo module with a wrapper from Bar like:

func (m *Bar) Test() string {
    return dag.Foo().Test()
}

[deft rain]

Hm, this looks like a bug to me actually - I can't think of why this shouldn't work off the top of my head.

[spiral whale]

From my understanding Foo and Bar are different modules. Since Foo is a transitive dependency, I'm expecting when we call dagger call -m ./bar test test generated client code should not contain any code related to Foo. Did I understand wrong how transitive dependencies work?

However, maybe it's possible to include types from transitive dependencies if they exposed as part of the module like in this example

[deft rain]

ok this is actually the same module id-able thing i've been working towards 😄
there's no reason this can't work - but you can't do it at the moment, because Foo doesn't have an ID. essentially, Bar's view of Foo is a graphql query - returning that doesn't really make a hugo amount of sense. instead, what we need to be able to do is get the ID of Foo, and then we can return that.

the graphql schemas being generated are just a sympton here - yes, they're wrong, but actually this is only one little part of it, the go codegen is also wrong.

[quick wind]

Quick one - is --help still the ideal/preferred way to list functions on a module, or has/will it be replaced by functions? There was some discussion of functions in late October but nothing since?

[spiral whale]

Thanks for the explanation. I don't think why I always assumed that's part of the dependency isolation. But what you describe makes sense. Do we have an issue for this id-able thing to track?

[deft rain]

I don't actually know! 🤔
Personally, I tend to use --help, since it's easier to change when exploring around - I think both will probably stay around for some time.

[deft rain]

not yet i don't think - i'll throw something together before the end of the week though, i'm still kinda working out what needs to be done 😄 i've not actually poked through this part of the code yet

[sharp zealot]

I personally would like functions to work more like —help. In its current form (dump all types & functions) I don’t find it as useful. My preference would be to keep both, and improve functions. But open to suggestions! Do you have a preference @MB?

[quick wind]

Not particularly. --help dumps a load of other help info, so a command distinct from --help that just lists functions would probably be useful, but ultimately either will work

Python module question - I get an error if I type a parameter as Optional[str] - unsupported union type: typing.Optional[str]. Are optional parameters not supported yet, or is there another way I should be handling these?

[restive shore]

New request (and blocker) from me! I can

[obtuse lion]

So I thought this was maybe because I was trying to return the module itself essentially, so I create a subtype in the other module and tried to return that and it still didn't work. Basically

func (m *Child) Test(ctx context.Context) *ParentThing {
    return dag.Parent().Thing()
}

Which returns Error: failed to get loaded module ID: input:1: host.directory.asModule.serve failed to install module schema: schema validation failed: input:2039: Undefined type ParentThing.

Error: failed to get loaded module
                                  
      | ...                       
 2036 |                           
 2037 |                           
 2038 |         """               
 2039 |         test: ParentThing!
 2040 | }                         
 2041 | extend type Query {       
 2042 |         child: Child!     
      | ...                       

[deft rain]

ugh yes - i tried this as well. both cases don't work until we can do ids as above, though it surprises me that we get completely different error messages.

[sharp zealot]

could GraphQL fragments help somehow? Came to mind when you said “this is just a graphql query, can’t return it”

[deft rain]

i don't think so - the way module calls are actually implemented, they're passing json data on disk, not actually being queried by graphql.

i actually think ids might be closer than i thought (though don't quote me on that)

i have a super basic implementation here - the test calls id on a module type Foo and later loads it back using fromFooFromID https://github.com/dagger/dagger/compare/main...jedevc:dagger:module-id-types#diff-3f816215b584a031057b440089ceb90784c3d4257816d7facfa3dcec14fe0a94R1330-R1372

[sharp zealot]

I'm having trouble running anything from the conference. I'm on tethered 5G, not awesome internet but not terrible. I don't know if the problem is purely bandwidth. Has anyone seen this before?

dagger mod init --name slim --sdk go
▶ init
  ✘ connect ERROR [26.96s]
  ├ [26.96s] starting engine

⧗ 26.98s  ✔ 1 ✘ 1
Error: new client: failed to run container: Unable to find image 'registry.dagger.io/engine:v0.9.3-gpu' locally
v0.9.3-gpu: Pulling from engine
aece8493d397: Pulling fs layer
a11284da28c6: Pulling fs layer
b28787532fca: Pulling fs layer
497effb46b5b: Pulling fs layer
0cf9322dd313: Pulling fs layer
ab0aae127b74: Pulling fs layer
9929d6ed75c9: Pulling fs layer
870e75346456: Pulling fs layer
201d71a1425c: Pulling fs layer
6d196cf407a7: Pulling fs layer
c3b3bc27bbee: Pulling fs layer
0b797bf69e96: Pulling fs layer
f2e81b007aa2: Pulling fs layer
2123046f69d0: Pulling fs layer
a5dff998be19: Pulling fs layer
c81ad70c95fd: Pulling fs layer
c35fd4d183fe: Pulling fs layer
c9acb41b4525: Pulling fs layer
b02de4eaf552: Pulling fs layer
4f4fb700ef54: Pulling fs layer
944600922bbe: Pulling fs layer
007d54b18118: Pulling fs layer
201d71a1425c: Waiting
0cf9322dd313: Waiting
ab0aae127b74: Waiting
ebc4dd5ed41d: Pulling fs layer
03b0a20e8b32: Pulling fs layer
2123046f69d0: Waiting
6d196cf407a7: Waiting
c9acb41b4525: Waiting
b02de4eaf552: Waiting
a5dff998be19: Waiting
c81ad70c95fd: Waiting
4f4fb700ef54: Waiting
944600922bbe: Waiting
9929d6ed75c9: Waiting
870e75346456: Waiting
ebc4dd5ed41d: Waiting
03b0a20e8b32: Waiting
f2e81b007aa2: Waiting
497effb46b5b: Waiting
0b797bf69e96: Waiting
c35fd4d183fe: Waiting
aece8493d397: Download complete
a11284da28c6: Verifying Checksum
a11284da28c6: Download complete
aece8493d397: Pull complete
a11284da28c6: Pull complete
497effb46b5b: Verifying Checksum
497effb46b5b: Download complete
0cf9322dd313: Verifying Checksum
0cf9322dd313: Download complete
b28787532fca: Verifying Checksum
b28787532fca: Download complete
ab0aae127b74: Verifying Checksum
ab0aae127b74: Download complete
9929d6ed75c9: Download complete
870e75346456: Verifying Checksum
870e75346456: Download complete
b28787532fca: Pull complete
497effb46b5b: Pull complete
0cf9322dd313: Pull complete
ab0aae127b74: Pull complete
6d196cf407a7: Verifying Checksum
6d196cf407a7: Download complete
9929d6ed75c9: Pull complete
870e75346456: Pull complete
0b797bf69e96: Download complete
f2e81b007aa2: Verifying Checksum
f2e81b007aa2: Download complete
c3b3bc27bbee: Verifying Checksum
c3b3bc27bbee: Download complete
2123046f69d0: Download complete
a5dff998be19: Verifying Checksum
a5dff998be19: Download complete
c35fd4d183fe: Verifying Checksum
c35fd4d183fe: Download complete
c81ad70c95fd: Verifying Checksum
c81ad70c95fd: Download complete
c9acb41b4525: Download complete
b02de4eaf552: Verifying Checksum
b02de4eaf552: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
944600922bbe: Download complete
007d54b18118: Verifying Checksum
007d54b18118: Download complete
ebc4dd5ed41d: Verifying Checksum
ebc4dd5ed41d: Download complete
201d71a1425c: Verifying Checksum
201d71a1425c: Download complete
201d71a1425c: Pull complete
6d196cf407a7: Pull complete
03b0a20e8b32: Verifying Checksum
03b0a20e8b32: Download complete
c3b3bc27bbee: Pull complete
docker: failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header.
See 'docker run --help'.
: signal: killed

I was able to reproduce it with docker only. Seems related to the gpu support:

$ docker pull registry.dagger.io/engine:v0.9.3-gpu
v0.9.3-gpu: Pulling from engine
aece8493d397: Pull complete 
a11284da28c6: Pull complete 
b28787532fca: Pull complete 
497effb46b5b: Pull complete 
0cf9322dd313: Pull complete 
ab0aae127b74: Pull complete 
9929d6ed75c9: Pull complete 
870e75346456: Pull complete 
201d71a1425c: Pull complete 
6d196cf407a7: Pull complete 
c3b3bc27bbee: Pull complete 
0b797bf69e96: Extracting [==================================================>]  1.048MB/1.048MB
f2e81b007aa2: Download complete 
2123046f69d0: Download complete 
a5dff998be19: Download complete 
c81ad70c95fd: Download complete 
c35fd4d183fe: Download complete 
c9acb41b4525: Download complete 
b02de4eaf552: Download complete 
4f4fb700ef54: Download complete 
944600922bbe: Download complete 
007d54b18118: Download complete 
ebc4dd5ed41d: Download complete 
03b0a20e8b32: Download complete 
failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header

Confirming that unsetting _EXPERIMENTAL_DAGGER_GPU_SUPPORT fixed my dagger issue. FYI @strong ingot @slow sand @rocky harbor

[kind carbon]

Python optional params

[slow sand]

I have been trying out GPU support using the same image with no issues but I should try the GPU Flag with dagger mod init to see what happens.
could be a network error?

[slow sand]

could you try docker image prune ?

[viscid totem]

Hi all, I'm playing around a little bit with the new module system. Locally everything is working well, but when running against a remote dagger engine (_EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://...) I'm having the following error:

$ dagger call container-echo --string-arg message
✘ load call ERROR [2.37s]
├ [2.37s] loading module
┃ Error: failed to get loaded module ID: input:1: host.directory.asModule.serve failed to load module types: failed to call module "hello" to get functions: failed to get function output directory: process "/usr/local/bin/codege
┃ --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1                                                                                                              
✘ serve ERROR [0.41s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [0.40s]
┃ runc run failed: unable to start container process: error during container init: error mounting "/run/buildkit/buildkitd.sock" to rootfs at "/.runner.sock": stat /run/buildkit/buildkitd.sock: no such file or directory         
• Engine: e8c0169dde5b (version v0.9.3)
⧗ 2.67s ✔ 15 ∅ 5 ✘ 3

Perhaps I'm missing something, any ideas?

[sharp zealot]

What version is your remote engine?

FYI @strong ingot @warped canyon @wintry prism dagger mod publish does not work for me.

$ dagger mod publish -f
✔ dagger mod publish -f [0.59s]
┃ publishing github.com/shykes/daggerverse/slim@7e20a402acd5ed08013095278f30a3633b20c
┃ eb8 to https://daggerverse.dev                                                     
┃ published to https://daggerverse.dev/mod/github.com/shykes/daggerverse/slim@7e20a40
┃ 2acd5ed08013095278f30a3633b20ceb8                                                  
• Cloud URL: https://dagger.cloud/runs/072e191c-f54c-4bd7-80cc-12ea63edab28
• Engine: fe056d690118 (version v0.9.3)
⧗ 1.76s ✔ 3

--> Looks like it worked?

But if I look in daggerverse.dev: no trace of github.com/shykes/daggerverse/slim which is what I published

$ dagger version
dagger v0.9.3 (registry.dagger.io/engine) darwin/arm64

Follow-up: when I try to publish it manually in the web UI, I get this

@strong ingot 👆

[wintry prism]

Will take a look when at airport.

[spiral whale]

the link is working from your console log

also, I saw it in daggerverse.dev did you fix your issue?

[sharp zealot]

I suspect that there are inconsistencies between instances. One of the instances sees it, the others don't. We recently rolled up multi-instances, with load-balancing, database replication etc.

might be caused by session stickiness

Not visible on mine

[restive shore]

I don't know if related but daggerverse.dev intermittently doesn't work for me.

[sharp zealot]

Not visible from a different browser either

[sharp zealot]

What issues are you seeing?

[restive shore]

It's working right now but IIRC it fails to resolve DNS

It didn't work earlier today. I'll post the error when it happens again

[sharp zealot]

@restive shore if this started within the last 30mn, it might be related to @wintry prism trying to troubleshoot the issue I reported, and restarting instances.

Otherwise, no

[restive shore]

It's been happening for a while

I can also see your slim module on daggerverse

https://daggerverse.dev/mod/github.com/shykes/daggerverse/slim@7e20a402acd5ed08013095278f30a3633b20ceb8

[sharp zealot]

Does it show up in the list, in the daggerverse home page?

[spiral whale]

I just tried from different locations after I read multi-instance and I can see your module on the daggerverse.dev

didn't able to replicate the issue

[restive shore]

yes it's the 3rd from the top for me

[sharp zealot]

That confirms my suspicion that instances are out of sync. I am still not seeing it anywhere on the list

and search comes up empty

[spiral whale]

it seems so

[sharp zealot]

Navigating directly to the module URL: https://daggerverse.dev/mod/github.com/shykes/daggerverse/slim causes Proxy error: EOF

[spiral whale]

https://daggerverse.dev/mod/github.com/shykes/daggerverse/slim@7e20a402acd5ed08013095278f30a3633b20ceb8 can you click this one?

[spiral whale]

same for me but I guess the link you shared is not complete.

[sharp zealot]

I tried the pinned link too, same error

[spiral whale]

☝️ the link i just shared is the redirection link when I click the module from home page

[sharp zealot]

http://daggerverse.dev/mod/github.com/shykes/daggerverse/slim@7e20a402acd5ed08013095278f30a3633b20ceb8

--> same error for me

[spiral whale]

it is interesting one, it works for me

[sharp zealot]

I cleared my browser cache just in case, but doesn't change anything. This doesn't seem to be session stickiness. Maybe IP stickiness. But that is weird too because I have moved from 5G tethering at the airport, to wifi in the plane. So presumably completely different IPs.

Oh it might be my DNS resolver cache

Or, it might be geo-location

Can you run dig daggerverse.dev please?

[spiral whale]

yea that could be geo-location. it makes more sence

dig daggerverse.dev

; <<>> DiG 9.10.6 <<>> daggerverse.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8427
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;daggerverse.dev.        IN    A

;; ANSWER SECTION:
daggerverse.dev.    54    IN    A    66.241.125.177

;; Query time: 50 msec
;; SERVER: 192.168.0.1#53(192.168.0.1)
;; WHEN: Fri Nov 10 00:59:28 CET 2023
;; MSG SIZE  rcvd: 60

[sharp zealot]

I remember Gerhard mentioning something about having instances in Chicago for Kubecon

Same IPU for me

Let's move to a thread 🙂

[sharp zealot]

Raw DX feedback: having the ID field be reserved continues to trip me up. My use case:

• I am developing a docker engine package (building from what we created for the docker-clim module with Kyle & Kyle.
• I have a type Image which represents an image in the Docker Engine
• To populate my Image, I unmarshal the output of docker image list --format ''{{json .}}" directly into the struct.
• Unfortunately, that JSON object has a field called ID...
• Even more unfortunate, that field is critical to my needs, so I can't skip it.
• Therefore I have to add much complexity, to manually copy fields rather than just unmarshalling them wholesale
• I can't use json: tricks to unmarshall ID into another field name... because the Dagger SDK also uses json:

One idea would be for the Go SDK to stop interpreting json: tags, and instead define our own struct Tags

That way I could control 1) how the DaggerSDK loads my fields, while 2) separately controlling how the Go json package marshals/unmarshals my fields in my own logic

Note: this is a mild inconvenience, not a blocker

[sharp zealot]

Success! A docker-slim module appears 😁

[viscid totem]

version v0.9.3

docker run --rm -d \
    -v $(pwd):/etc/dagger \
    -v $(pwd)/dagger-data:/var/lib/dagger \
    --privileged \
    -p 1234:1234 \
    registry.dagger.io/engine:v0.9.3

Running another example with dagger run against the remote engine is no problem, that is working as expected

[sharp zealot]

Huh, it's specifically with one module, and specifically with a remote engine?

Any chance you could publish that module so I can try to reproduce?

[viscid totem]

It is actually the example module generated by dagger mod init --name hello ---sdk go
I ran the init against my local engine (without the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable) and then swiched to the remote engine.
Now I just tried dagger mod init ... again in an empty directory with the remote engine, resulting in the same error

$ dagger mod init --name hello --sdk go
✘ generatedCode ERROR [0.45s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [0.15s]
┃ runc run failed: unable to start container process: error during container init: error mounting "/run/buildkit/buildkitd.sock" to rootfs at "/.runner.sock": stat /run/buildkit/buildkitd.sock: no such file or directory                                                     
WARNING: no LICENSE file found; generating one for you, feel free to change or remove license="Apache-2.0"                                                                                                                                                                      
• Engine: d7ffee39a419 (version v0.9.3)
⧗ 2.65s ✔ 16 ∅ 1 ✘ 2
Error: failed to get codegen output entries: input:1: host.directory.asModule.generatedCode failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

[wind coyote]

Ive been playing with zenith a little bit more. Couple questions:

1. Is it possible to use private github repos for modules?
2. Are there plans to be able to use modules inside of pipelines without having to create a module within an application repo? I.e, platform engineering creates a module that application developers use in their pipelines.

[sharp zealot]

1. Not sure it’s possible today, but we absolutely plan on supporting it. It should be mostly a matter of UX to pass the credentials

2. Yes, absolutely. We designed for this pattern from the ground up. It is a first class citizen.

[wind coyote]

Is the second a thing yet? I was only able to do cross modules.

[deft rain]

aha yes, i have an identical issue 🙂 unfortunately, i think the only way to resolve this is to not use encoding/json and to have a custom json encoding implementation (which is bizarre right? i just want to use a different struct tag field, and we'd need to fork the package. mayyybe we could have some fun auto-code generation to just find-replace on the json package, and create it like querybuilder?).
i tried something funky at some point to have the auto-conversion to a new struct type done automagically, but it felt really clunky, added tons of code and is also super hard to cover all the edge cases (things with multiple levels of indirection, like pointers to slices)

[restive shore]

I opened a thread for this the other day if you want to follow 🙂 https://discord.com/channels/707636530424053791/1171809848057401375

[restive shore]

For this, you'd have to use the dagger CLI and call your module. If you are looking to programatically call a module from non-zenith code, it's not possible yet but there is an issue open for it - https://discord.com/channels/707636530424053791/1170399581218029658

[sharp zealot]

Yeah you would do something like dagger call -m github.com/acme-platform/daggerverse/webapp deploy —source=.

[wind coyote]

it's not possible yet but there is an issue open for it

Ah, yeah, that's what I thought. NBD. I'm still super happy. Can't wait.

[sharp zealot]

@wintry prism sorry, moving back over here

[obtuse lion]

As I’ve been trying to use zenith to build a pipeline I’ve found the most useful construct is to define a step in a pipeline as a function (which hangs off a module) as having a required arguement of a container which then returns a container. Then optionally other args can be defined on the function. This makes it very reusable. Also the same style that works for steps works for building containers for publishing.

It’s a very simple construct but extremely reusable. With this style it’s trivial to define a GitHub action style workflow. Dagger provides all the steps (ideally from the ecosystem) I just need to stitch them together into a workflow.

[quick wind]

Can you link an example?

[spiral whale]

I'm using a similar setup for actions/runtime to link github actions marketplace actions together inside of the dagger pipeline. I was going to similar thing. It's good to have someone have same approach as me. Did you able to workaround id-able problem while stiching steps together or is it still missing?

[restive shore]

Doesn't that make it tough to call your functions via dagger CLI? You'd have to stitch together modules and expose entrypoints via a custom module right?

[obtuse lion]

The id-able workaround was to just use container since I can pass that around everywhere. And it ends up being fine. Before I had an object called Workspace that had a directory and env fields. But I found out that structure wasn’t enough and the container actually worked better. Because it has a root dir, work dir, and env. The container maps perfectly to the state maintained between gh actions which is just the VM really. So instead of each action mutating the VM it mutates the Container.

[spiral whale]

Hmm, it makes sense. In my case, actions make container modifications during executions using logs or env files. Those do not persist between container executions and processing them outside of the container creates overhead. I guess I need to find easy way to move those changes between steps

[spiral whale]

Using module to stitch a new pipeline together is expected use-case for me. The purpose is create a new pipeline with Github Custom Actions. I'm planing to separate cli optimized calls and module optimized call to keep thier DX is optimal for usage. But I'm still discovery phase for it

[restive shore]

Understood and makes sense. I just wanted to point out that container in-out pattern makes it harder to consume those functions from the CLI so if that's a pattern that's promoted we have to be clear it's not for CLI consumption.

[spiral whale]

🤯 Today, while refactoring gale, I broke something in the container. After struggling to debug the issue, I just experimented with dagger shell and it executed my workflow and connected to the shell when the pipeline failed. This is the best feature of the dagger so far 😄 Now it's so much easier to debug issues with gale

dagger shell --focus=false run --repo kubernetes/minikube --branch master --workflow build --job build_minikube --token $GITHUB_TOKEN sync

[obtuse lion]

is somebody trying to wrap github actions in dagger modules?

[sharp zealot]

Yes 🙂 There is a wrapper tool co-developed by @spiral whale and @chrome pilot , which will generate a Dagger module wrapping the specified Github Action.

[sharp zealot]

Under the hood it's a convenience wrapper around Project Gale by @spiral whale . You don't need to generate code for a module, you can invoke gale dynamically. The generation gives you the convenience of mapping the Action's inputs and outputs schema into GraphQL types (and from there, native Go, Python types)

See also: #github

[obtuse lion]

this is working much better than I thought it would...

[ruby night]

You and Kyle did all the work there... Thank you guys! Didn't expect we'd have an opportunity to do this during KubeCon. Learned a lot about Dagger modules. Working on the updated version. Quick question... is it common to have module parameters that point to config files? is it an anti-pattern? what if you have a lot of module parameters?

[rocky harbor]

This weekend I used Zenith to finally tick something off my TODO list that's been there over a year: https://daggerverse.dev/mod/github.com/sipsma/daggerverse/apk@d709dd631e176025ea042dc03fdc90197f4b12ba

That module uses Chainguard's go-apk package to take a list of alpine packages, turn it into a toplogically sorted DAG of package metadata and then uses dagger to download the packages, unpack and assemble them into a container image where each layer corresponds to one package in the DAG. With a little more polish, this gets us close-to-optimal caching for building alpine images. And hopefully in time we can take the same idea and extend it to other packaging systems.

Funnily enough, the last few hours of my life have been addressing some vulnerabilities that trivy started flagging in our dev engine images. The last part of that involved some alpine packages that were fixed a few days but weren't getting picked up because our CI was still using the cached result of the apk add ... commands we run. Pretty much the perfect example of the problems that apk module and similar will fix; it allows you to get all the latest packages for alpine without compromising on caching! Just need to finish our conversion of CI over to Zenith

[sharp zealot]

Finally!!! 🥳🥳🥳 I actually tried last week and got completely lost in the apko codebase. Can’t wait to incorporate this into my wolfi package

[rocky harbor]

Yeah it turns out the package we want specifically is really this one: https://github.com/chainguard-dev/go-apk, which is what apko+wolfi use internally.

The code I pushed there is really rough but pretty neat in how little there is to it. Definitely feel free to just copy-paste as I'm not sure when I'll have time to go back and polish more

[sharp zealot]

Wait I just realized this also gets us closer to solving the “dockerfile over-caching” problem. I didn’t think that would be possible without changing the core.. Even better news then 😀

[rocky harbor]

Yeah exactly, just a matter of squashing each package system with modules like that now 🙂

[sharp zealot]

I guess the part that requires core changes would be to optimize even further and cache as much of the runtime execs as possible? (the design discussion we intentionally shelved for now)

[rocky harbor]

Yup, and btw I thought about it again and think I may know how to implement your idea around the function code itself making a call w/ its cache keys and the engine deciding if it should keep running or not. It requires some upstream buildkit changes (need to be able to query its cache), but probably within reach

[sharp zealot]

I remember there were a few competing designs for this, I’m not confident mine is the best, first step (when we decide to unshelf) would be to review our options probably

[rocky harbor]

I remember there were a few competing

[quick wind]

Are there any plans re: host environment variables? There's some discussion from mid-October (#daggernauts message) about passing them as parameters but for longer lists of env vars (some of which are optional and require further code to test and set) that feels quite clunky.

[obtuse lion]

Don't post-install hooks from package managers completely destroy this? Or do you assemble all the packages and then still do a post install run after everything?

Feedback for daggerverse. I went to this page https://daggerverse.dev/mod/github.com/sipsma/daggerverse/apk@d709dd631e176025ea042dc03fdc90197f4b12ba and spend a good 30 seconds looking for a link to github source. I felt stupid when I figured out it was the big github logo. But it's not intuitive I guess. Not the first time I've done that

[thorn moat]

haha, yeah I agree, the nav needs more iteration beyond "pick an available corner and put a button there"

I also want to make it so when you go-to-definition and land in the client code, there's a comment linking to the backing implementation

[obtuse lion]

Very important feedback. Daggerverse is a long and difficult word to type. I'd hate to see a d9e.

[wintry prism]

Let's just leave off the final 'e' hint and use: dag8
😆
with alternate: dag♾️

[sharp zealot]

I’m thinking it will end up integrated into the main site (maybe)

@obtuse lion but the real solution is to make daggerverse.dev your first autocomplete result on ‘d’

[obtuse lion]

you need to coin a term like dmod so I can then have github.com/ibuildthecloud/dmod/hello-world Just think of the efficiency gains.

[sharp zealot]

maybe just “verse”?

[deft rain]

https://tenor.com/view/firefly-serenity-kaylee-fry-river-tam-jewel-staite-gif-4816271

[rocky harbor]

Don't post-install hooks from package

[grizzled basin]

Can zenith use private repos for modules? (yet?) - think I saw something in here awhile ago about that not currently being possible?

[wind coyote]

IMO not currently possible unless something changed within the week.

[restive shore]

Not possible yet. Hopefully soon! I have a thread open regarding the same - https://discord.com/channels/707636530424053791/1171809848057401375

[azure tartan]

Doing some local testing of lambdas with docker images and the aws_lambda_rie; wondering if that's a nice candidate for a zenith module?

[dense canyon]

👋 not sure if it has been brought up but is there a way to differentiate between input and output Files/Directories in function arguments? I was trying out a module that expects an output-dir argument as a Directory and I ran it from my home which ended up in dagger trying to transfer all my homedir into the function context.

[dense canyon]

Well... thinking a bit more about it, I'd assume that output files/dirs should be specified as strings instead of the native Dagger Directory or File types?

[restive shore]

Yeah that makes sense, you'd only want the path string and not the dir. Even if you do import the directory and manipulate it (I didn't know it was even possible), you can't export it without running dagger download. Having to call two different commands (call vs download) for exporting files/dir makes it a bit awkward I wonder if that DX can be better.

[sharp zealot]

Input vs output directories & files

[rocky harbor]

@spiral whale (continuing conversation from DMs since I think there's probably wider interest) for this PR: https://github.com/dagger/dagger/pull/6102

Was the part you were interested in using specifically the ability to use objects from other modules as input args + return types? Or just the ability to use your own module's types as input args?

Doing some local testing of lambdas with

[signal perch]

Howdy! I'm trying to create a zenith module for depot to build images. I could use some advice on how provide module flag options.

I have an example (https://github.com/goller/daggerverse/blob/main/flags/main.go) with two different "styles" to provide options. I'd love some advice about which way (both?) to do.

Style1 seems better for dagger mod use users while Style2 seems better from dagger call users.

I've created an example dagger mod use for both styles here: https://github.com/goller/daggerverse/blob/main/useflags/main.go

[wintry prism]

I can 't wait for someone to build an ngrok module!
Plenty of good examples like https://github.com/marketplace/actions/ngrok-tunnel-action
and nice Go library 🙂 https://github.com/ngrok/ngrok-go

[sharp zealot]

CLI verbs still feel clunky when demoing. I think we’re close but not quite there yet

@muted plank after seeing a demo asked: “what’s the difference between call, shell, and up?” I had an answer but it was not a very convincing one..

[muted plank]

The confusion also came from the fact that the called function was returning a container image.

[sharp zealot]

Wouldn’t it be cool if you could compose dagger CLI commands in a unix pipeline? Might make it more understandable what’s going on

ID=$(dagger call -m foo bar baz)
echo $ID | dagger print
echo $ID | dagger exec
echo $ID | dagger save
echo $ID | dagger up
echo $ID | dagger push

Of course this would require stable IDs… cc @thorn moat @chrome pilot @rocky harbor @kind carbon

[quick wind]

Python modules - chaining commands

[tame wolf]

👋 so I'm just learning about zenith/modules, and I immediately have a couple high-level questions.

• how are modules glued together under the hood? Since they're language agnostic, I assume each one runs in a container and all the function calls serialize to something like json or protobuf?
• It seems like the CLI becomes the star of the show. How do you envision bootstrapping a new developer? I liked being able to tell people git clone foo ; cd foo ; go run ./build because I was confident they already had docker and golang installed.

[rocky harbor]

Thanks for pointing out the outdated docs @ruby night 🙂 This will fix it https://github.com/dagger/dagger/pull/6120

[short prism]

Hi all. I've been using Dagger for a bit but haven't really looked into Zenith until now. I listened to a couple podcast episodes about it, but I'm still confused about a couple things:

• Is Zenith essentially going to be "next-gen" Dagger or Dagger 2.0?
  • If not, is it eventually going to "replace" Dagger as we know it? Or merge back into mainstream Dagger?
• Is there a "homagepage" for the project?
  • I found https://docs.dagger.io/zenith/ but it's just a new set of Dagger docs?

[tame wolf]

I'm also just learning about it. It seems like the dagger engine is fundamentally unchanged. Modules and the user interactions are pulling in dependency containers and composing them in novel ways.
To me, this suggests that the existing SDK and my custom go CLI around it will continue to work. But I don't know what kind of support guarantees the dagger team is making around that.

[ruby night]

thank you Erik!

the slim dagger module demo'ed during the community call (for others to use as a reference, a slightly enhanced version of what Solomon/Kyle built during KubeCon): https://github.com/slimtoolkit/daggerverse/tree/main/slim-with-docker

[wintry prism]

Thanks @short prism! Plan is to keep both styles.

Zenith is still experimental and evolving, but we think that calling fuctions from the CLI may become the dominant way to run Dagger because of the many benefits:

• reduced host env requirements/setup: you only need the dagger cli and the ability to run containers (no local dependencies like golang, python, node, etc
• cross-language modules ecosystem: Any module written in any SDK is usable in your module written in your SDK of choice (e.g. use a Python mod from Go)
• expressive verbs: dagger exec, dagger save, daggger print, dagger call, etc (names in flux a bit)
• easier/shorter/quicker to write for most folks

@tame wolf You're right that the same SDKs and core API are at work here, so the current style of running from your host is not going away, but will likely be used more for particular advanced use cases. In fact there is some thinking about how modules might be made consumable by the current non-Zenith Dagger: https://github.com/dagger/dagger/issues/5993

[ruby night]

daggerverse/slim-with-docker at main · s...

[sharp zealot]

Nailing the CLI experience

[ruby night]

what's the best practice for calling module functions with optional parameters in the Go SDK? right now I have something like this, which looks a bit messy: s.Minify(ctx, container, OptEmpty[string](), OptEmpty[bool](), OptEmpty[string](), OptEmpty[bool](), OptEmpty[bool]())

[wintry prism]

Discussion about that topic here: #1174466237024182352 message

[sharp zealot]

I think @ruby night has a slightly different issue. The problem is that calling your own functions from within the module, is a different DX than calling another module’s functions, when there are optional args involved.

@ruby night there is a helper function Opt() which is what you’re looking for. Not perfect but definitely will make it less painful.

[sharp zealot]

Actually nevermind, OptEmpty I believe is the correct way,. and I agree it is messy

[ruby night]

It's supposed to be messy when you are getting started 🙂 Just wanted to make sure I wasn't missing anything

[ruby night]

Adjacent problems... I'm interested in that problem too because I was thinking about using WithX() to configure optional params

[latent trellis]

OptEmpty is None in Rust. I like that better.

[signal perch]

Howdy! I'd like to be able to push images to a registry. Is it possible to pass and or mount docker creds into a module? I see WithRegistryAuth as well.

[spiral whale]

I published only gale to daggerverse and was planing to keep exclude dependency sub modules from daggerverse but it seems it's automatically publishes sub modules as well

[thorn moat]

Yep, it does. Daggerverse is more akin to pkg.go.dev than rubygems or other package registries; it'll crawl dependencies, and in the future modules may be auto-published there when they're used by anyone in the CLI. It's definitely a bit surprising at the moment since otherwise publishing is manual, and everything shows up pretty prominently. 🤔

[spiral whale]

Hmm, I understand. I designed those modules as private so they don't make sense for users to get them. I need to change my perspective about publishing modules. Thanks for the info 🙏

[wintry prism]

Howdy! I'd like to be able to push

[sonic vessel]

How do I start to make a module support for new SDK? In this case I want to kickstart the Elixir SDK and see what need to be fix in the SDK before adding module support.

[rocky harbor]

How do I start to make a module support

[warped canyon]

What modules are people working on? My next one will be for quill (https://github.com/anchore/quill)

[void widget]

I was about to work on this one this weekend: https://github.com/indygreg/apple-platform-rs/tree/main/apple-codesign (solving the same use-case)

[warped canyon]

It's a race! 🏃

[void widget]

Would be glad to sync with you, next week, to see how to open a PR against bun, on whoever's best module ? 👼😏

[subtle halo]

Does creating a /ci directory still make sense in the new zenith paradigm? I'm not expecting to publish any modules at this point

[sharp zealot]

Yes, your ci directory will just become a dagger module. It should be a pretty easy gradual change

At the end of the day, a module is just a directory with code that the Dagger Engine knows how to load

There is no obligation to publish

[subtle halo]

Definitely not opposed, just very much on the exploration/does this make sense or solve issues phase of things

[sharp zealot]

@thorn moat another facet of the CLI UX I wanted to discuss with you: what we print to the terminal and how 🙂

[shy ledge]

I just found daggerverse. One thing I feel is missing is a filter on the language used to create the module. 🙂

[sharp zealot]

Agreed!

[wintry prism]

I can make an issue to track that. But I wanted to understand why 🙂
Since you can call module functions written in any language, from any language, would this be your use case, or something else?

As a module developer who intends to write a module in a particular language (e.g. Python), I want to be able to quickly find all of the modules in my language so I can use them as code examples/inspiration for my module.

[shy ledge]

Yeah, that sounds good. @wintry prism I didn't know that any language module can be used with any other language though. Are there docs for how that works?

[quick wind]

Same as it works for modules of the same language 😉 - the module will be available at dagger.<Module> once you've added it to your dagger.json

The bit I haven't figured out yet is this ctx argument - anyone point me at docs/note/source for this?

In Go that just seems to be a context lib - there's language-specific libraries for that?

[wintry prism]

Ah, yes, this is a killer feature of Zenith 🙂
We definitely need to add examples to the docs that are forming up here: https://docs.dagger.io/zenith/developer/, https://docs.dagger.io/zenith/

[wintry prism]

But in essence, just run dagger mod install in your module's directory as suggested in the https://daggerverse.dev listings and you can use that module's functions. Here, I've made a module and installed a Python module (Infisical) and a Go module (Trivy).
After I installed both in my module, here's the dagger.json

mymod ➤ cat dagger.json
{
  "name": "mymod",
  "sdk": "go",
  "dependencies": [
    "github.com/jpadams/daggerverse/infisical@21f9211e5727a0b6acce8a0ab6c35d01896dc642",
    "github.com/jpadams/daggerverse/trivy@21f9211e5727a0b6acce8a0ab6c35d01896dc642"
  ]
}

You can see the VS code autocomplete off the dagvariable in Go or the dagger variable in Python. Still DX work to do 🙂 to unify that.
cc @kind carbon

[thorn moat]

Re: showing the language for a module - the data is collected by Daggerverse already, but I decided not to show it because it might give the wrong idea: "I need to find a module written in my language" which might lead to "I found one in another language, guess I need to rewrite it" and then there are 2 unnecessarily competing modules

[restive shore]

I think this is a valid concern. And I can see developers interpreting it that way.

[sharp zealot]

Agree. Didn’t think of that.

[shy ledge]

Well, the Daggerverse should also be teaching devs what modules are all about, before they start using them. Right? A well placed, "read/ watch this first" to an article or a video about modules and why they exist and how to use them should be the first stop for anyone visiting Daggerverse IMHO. Once that is front and center, then guessing about what devs might think or not think is taken pretty much out of the equation. In other words, if they do something silly after that, it's on them. 😛 🙂

[shy ledge]

And just FYI, there are a lot of open questions in my mind on how modules work. I'm still not certain about the mechanics. I'm wondering for instance, if different languages can use functions of other languages, could this be expanded to more than only CI pipelines?

[ocean escarp]

I think it would also be interesting to have the publication date of the module in the daggerverse

[lime comet]

Hey folks! I've been building a few modules lately and I constantly found myself re-writing functions like WithSource(src *Directory) and With<Variable>(value type) every time. I think I understand the reason behind having dagger calls run in completely sandboxed environments and I do like it, but maybe there is a way for dagger to automatically provide functions to set values defined in the struct? Not sure how it would look like. One idea I had is analogous to setters. We could potentially look at the fields the struct/class has and automatically provide setters that respect their type and follow a naming convention. So if I have a struct like so:

type Foo struct{
    Source *Directory
    Credential *Secret
}

Dagger would automatically generate setters so that I could do: dagger call <set/with/?>-source --source "." <set/with/?>-credential --credential "supersecret" and then chain it together with my own functions that use this values

[dense canyon]

ref: #1174466237024182352 message

[sharp zealot]

Yes 🙂 Dagger is a general-purpose pipeline engine, that happens to be great for CI. I believe over time, the community will find broader and broader applications.

[thorn moat]

Noted! We're actually tracking that data now (publish date = commit date), just need to figure out where to show it

[spiral whale]

Hey, daggerverse.dev is failing when search used. https://daggerverse.dev/search?q=dind

Will creating a bug for it. Just write here first

☝️ https://github.com/dagger/dagger/issues/6131

[thorn moat]

looking into it, thanks!

[thorn moat]

deployed a fix

also now search will only show the latest version of each module, since it was super confusing to show all of them

[sharp zealot]

@thorn moat @rocky harbor do you know why dagger up doesn't forward any ports unless I explicitly add -p flags?

[rocky harbor]

I don't know the reasoning, but there's the --native option to automatically forward all ports w/ matching frontend/backend. I feel like that would be a reasonable default behavior too if no explicit -p flags are provided. But would want to double check w/ @thorn moat if there's a reason that wasn't done intentionally

[thorn moat]

No good reason, it defaults to random ports right now I think, making -n a default makes sense, just need to decide how it interacts with -p

[sharp zealot]

@thorn moat does -p support -p foo:bar ?

what does -p 80 actually do?

[thorn moat]

Yep

[thorn moat]

same as -p 80:80

[sharp zealot]

so -n is like docker run -P ? (if my memory serves me correctly, it's been a while)

ie. --publish-all

[thorn moat]

apparently the current default behavior is -P:

  -P, --publish-all                    Publish all exposed ports to random ports

not sure if docker run has an equivalent for --native

[sharp zealot]

oh I see - the random part is the difference

In our case, I think it makes sense to have "native" be the default

with maybe some UX bikeshedding

[thorn moat]

sounds good to me, I've been using --native all the time too

and if you pass any ports (-p), that implies --native=false?

[sharp zealot]

that's the UX bikeshedding part 🙂

It's tied at least in part, to the broader #1174585230649217055

(Since that conversation might change what command the flags are attached to, and with it, the context for the flags)

[sonic vessel]

Just have the time to publish compose here https://daggerverse.dev/mod/github.com/wingyplus/daggerverse/compose. Currently, support only ports and env. 🙂

[ocean escarp]

how to execute dagger call if the function parameter is a list?

[kind carbon]

In Cobra, that's parsed as a CSV or using the same flag multiple times. A pkgs []string arg becomes: dagger call foo --pkgs git,curl or dagger call foo --pkgs git --pkgs curl.

[sharp zealot]

I did not know about the csv syntax! Nice

[ocean escarp]

Thanks! now I can run my pkgx module dagger shell -m github.com/tsirysndr/daggerverse/pkgx install --pkgs jq,gh,gum,glow . And this module is written in typescript using a custom Dagger module SDK for Deno

[ruby night]

am i reading it right... it also supports only one service/container from the compose file

[spiral whale]

┃ Error: generate code: template: module:59:3: executing "module" at <Modul
┃ ainSrc>: error calling ModuleMainSrc: failed to convert field type: unsup
┃ rted type *types.Map
✘ generating go module: gale ERROR [0.15s]

Do we have an open issue for tracking not supported types at zenith? I was checking current main for new fixes and just saw we still don't support map types in zenith

[kind carbon]

How would a map type translate to GraphQL? List of pairs?

[spiral whale]

probably as an object

[rocky harbor]

No don't have an issue yet, feel free to file one

[kind carbon]

GraphQL objects can't have dynamic fields.

[rocky harbor]

Yes that's how it's represented typically, that would be the workaround in the meantime for any users that need this; can define a custom object with fields for Key and Value

That's essentially what we'd do underneath the hood to support it

[kind carbon]

Yep

[spiral whale]

bug for the issue: https://github.com/dagger/dagger/issues/6138

[spiral whale]

ohh, then as you said adding a workaround under the hood would be awesome.

[kind carbon]

Constructors

[thorn moat]

Continuing the conversation re: Optional args in Go, ran into a pretty nasty extreme case: https://github.com/dagger/dagger/pull/6079#issuecomment-1824957847

[rocky harbor]

ETOOMANYARGS

[dense canyon]

👋 just realying a #help request regarding variables being shadowed with the Go SDK here: https://discord.com/channels/707636530424053791/1178719192585879643. Particuarly json

[upbeat herald]

For the curious who wants to follow how the Node SDK integrates Zenith, you can check that PR: https://github.com/dagger/dagger/pull/6159.

It's currently in development, not usable at all since from dagger CLI because the runtime isn't integrated for now, but it will come as soon as possible 😄

[uneven rock]

Is there a list of all subcommands for Dagger Zenith CLI? I tried looking over docs for zenith - the CLI reference to be more exact - but the docs only contain the "old" CLI reference.

[sharp zealot]

if it exists it will be at https://docs.dagger.io/zenith

But note that the commands will probably still move, that part of the design is still WIP

[warped canyon]

Vikram just created this issue to properly generate the CLI reference: https://github.com/dagger/dagger/issues/6174
the docs site isn't up to date with the zenith subcommands from what I've seen but dagger help <subcommand> with the hidden subcommands provides the usage. Those subcommands are call, up, functions, shell, download, maybe I'm forgetting one 😂

[wintry prism]

The subcommands are mentioned in docs

But hidden in

I was just looking in code to see if you could set a flag/env var to show them

[void widget]

I agree, I was surprised yesterday by that

[sharp zealot]

I’m leaning towards a single subcommand, see #1174585230649217055

[ocean escarp]

very interesting, here's how I did it for my typescript modules :

• runtime entrypoint: https://github.com/fluentci-io/daggerverse/blob/main/deno-sdk/sdk/src/mod/cli.ts
• my custom deno module sdk runtime: https://github.com/fluentci-io/daggerverse/blob/main/deno-sdk/main.go
  I haven't used decorators yet, for the moment I'm parsing the comments docs

[bleak nest]

There is an open PR to update the CLI reference page, just pending merge approval. Feel free to use it/comment on it @uneven rock : https://github.com/dagger/dagger/pull/6069

... and for examples of how to use the new subcommands, I recommend our zenith quickstart https://docs.dagger.io/zenith/user/quickstart and would be very interested in feedback on it 🙂

[uneven rock]

Thanks!
Also, maybe a bit unrelated - is there a list, what types can be passed as arguments? I know that you can use context.Context, but it gets "hidden" from required arguments in the CLI. I also know, that a bool arg is true if passed or false otherwise. Also found out, that a *Secret can be passed - a string from the command line gets automatically converted... Alot of unknowns for a beginner like me but I get it that Zenith is still experimental. 🙂

[uneven rock]

Well I followed the https://docs.dagger.io/zenith/developer/go/457202/test-build-publish by example, so I guess I missed the other commands... My bad! But yeah, the quickstart is great!

[deft rain]

Would we ever be interested in an (opt-in!) module vendor directory that you'd commit into git?
I quite like using them in go for a couple reasons:

• less need to fetch the world after a clone (could be useful for us as part of reducing network requests)
• makes it easy to review changes to dependencies (i use this to make it easier to see if there are obvious breakages/security implications/etc)

Pretty much it would just contain the pre-cloned modules embedded in the local filesystem, ready to be uploaded and processed in dagger.

[rocky harbor]

Would we ever be interested in an (opt-

[lapis willow]

Hey guys - just came back to zenith and walked through the zenith getting started docs. (fantastic btw)

Just wondering two things.

1. Can I init the "playground" somehow of my module? How about multiple modules? Looking to explore the GQL
2. Somewhat related to the above, can I persist a GQL server with access to all my modules? Like I have a dagger daemon (pre zenith) - running right now for our builds.. can I "load" modules onto it? Would be super neat if I could just call that server's GQL directly to run some pipelines

[lapis willow]

Hey guys - just came back to zenith and

[spiral whale]

Hey everyone, I'm considering separating services from 'gale run'. However, I'm facing an issue. How can we automatically connect a running service to a Dagger container, without having to manually do it? Any ideas?

[grizzled basin]

Trying to rework an old dagger pipeline (super duper basic as thing) to zenith - ```
❯ dagger -m ci call --help
✘ load call ERROR [6.13s]
├ [5.40s] loading module
├ [0.73s] loading objects
┃ Error: main object not found

[grizzled basin]

oh god.

type FindDupes struct{}
``` vs

type Finddupes struct{}

Mixed case didn't work

[sharp zealot]

https://tenor.com/view/101dalmatians-roger-trombone-roger-trombone-cruella-devil-gif-21397793

[sharp zealot]

Those are the worst to find, glad you didn’t waste too much time on it

[grizzled basin]

lol

is that a bug, or a convention I've missed somewhere along the way? I wonder if https://docs.dagger.io/zenith/developer/go/525021/quickstart#step-2-add-a-function should make that clear?

[wintry prism]

I love this tweet and this gif so much
https://twitter.com/kcqon/status/1731812594577547615

[warped canyon]

from one of my buddies

Oh my god this is so memable im using this

[sharp zealot]

Great, my first viral meme and I look like an Iguana

[upbeat herald]

Hello 😄

Remember last week I said we're close to have a Typescript integration to Zenith? Here we are: https://github.com/dagger/dagger/pull/6220

Indeed, it's unstable, they are probably things not supported but it works with the basic example so feel free to try and give me feedbacks, I added explanations and a demo in the PR to show how you can get started

[sharp zealot]

amazing Tom!

[wintry prism]

@kind carbon When developing a Python module, is there ever any action needed on the developer's part to record the required dependencies for the module other than a from/import statement in module code?

[kind carbon]

Do you mean other modules or other python packages?

[wintry prism]

For other modules: dagger mod install ...

For other Python packages, yes

[kind carbon]

For other python packages they can be added to the module's pyproject.toml.

It would look something like this:

[project]
...
dependencies = [
  "cryptography",
  "click>=7, <9, != 8.0.0",
  "python-dateutil==2.8.*",
  "numpy~=1.21.4",
]

If using poetry it's a bit different.

[wintry prism]

I realized this when an import (not from x import y, I think) didn't quite work for me at some point. Need to repro.

I was surprised because I had just done this when developing the Infisical module:
https://github.com/jpadams/daggerverse/blob/main/infisical/src/main.py#L4
from infisical import InfisicalClient

I didn't update pyproject.toml myself, I don't think...or maybe I did 😆

I suppose the dagger mod sync wrote the dep to my pyproject.toml?

infisical ➤ grep -ri infisical                                                   git:main
./pyproject.toml:  'infisical>=1.5.0'
./dagger.json:  "name": "infisical",
./src/main.py:from infisical import InfisicalClient
./src/main.py:    """Get a secret from an Infisical project using secret name, project token, env, and path"""
./src/main.py:    inf_client = InfisicalClient(token=await token.plaintext())
./src/main.py:    """Insecure test using default Infisical project and plaintext token"""

Then I looked in our docs and didn't see this covered.

So I want to create an issue to document, but realized that I didn't understand the mechanism.

[kind carbon]

No, mod init is the one that writes a first pyproject.toml but after that it isn't touched.

You must have added that yourself.

Otherwise your module wouldn't work.

[wintry prism]

Ah, okay, I must have been in the zone 😂

[warped canyon]

#1169448382494945310 message 😄

I didn't understand the mechanism

It is I, the mechanism

[spiral whale]

Has anyone tried accessing the directory of a service? I'm trying to replace my CacheVolume hack for artifact exports due to #1180660724452818985 error. However, I couldn't find any other way to access my files and directories other than cache volumes

[wintry prism]

I can see other folks struggling with that. We should show adding a python dep in our quickstart. I'll make an issue.
https://www.jumpingrivers.com/blog/python-package-managers-pip-conda-poetry/

[warped canyon]

Yeah, worth mentioning in the quickstart. I think in general it's a python thing, not a dagger thing, so we don't have to get too deep into it

[wintry prism]

Agreed. Not trying to get deep. It seems we've already made a decision about pyproject.toml which is more Poetry-aligned, maybe? According to my 1 article worth of research above.
Folks who casually use a requirements.txt or an environment.yml might not instantly know what to do in a Dagger Python module. Think a devops or data science person who is less of a Python tooling master and more of an infra or math person. There's also the second pyproject.toml under sdk/ which will confuse someone. So might be there are some helpers in the CLI. Docs to start, of course.

[kind carbon]

That’s not quite right. I’ll reply when I can. I’ve ansmwered most of this on Linear but related to updating the docs on the site, not zenith. With modules I’ve explained this more ad-hoc I think.

[wintry prism]

Got it. I'll take a look in Linear 🙏

[kind carbon]

See DOCS-41

Hyperlinks in text are relevant for more context.

[obtuse lion]

Hey yall. Long time 🙂 Is there going to be a binary form of a module? I know that might be heresy but is it even possible to point to a prebuilt module, not the source?

[sharp zealot]

Any chance we could daggerize ./hack/dev? It seems like it could be made a proper module now, right?

And as a follow-up: any chance we could add support for a new source field in dagger.json, to support moving the module's source code to a subdirectory? This would allow, for example, moving our own dagger.json to the root of the dagger repo (with source: "./ci"). which in turn would allow discovering Dagger functions by simply typing dagger functions anywhere in the repo

[rocky harbor]

Yeah there's an issue for it here: https://github.com/dagger/dagger/issues/6137 I am sad every time I modify the current CI setup without converting it over more but it's just hard to find bandwidth atm

[sharp zealot]

If there were a list of reasonably-sized sub-tasks, I think some of us could chip in in a way that we can't today, because we don't have the necessary knowledge of our entire CI

(I would definitely volunteer to help with those sub-tasks, it would be a good dogfooding opportunity)

[rocky harbor]

I listed the two possible high level approaches to the overall effort in that issue. Basically, either

1. break down all functionality into reusable utilities called in both mage and the module, then convert each github workflow file one by one from mage to the module (subtasks in that case are each workflow file)
2. Run our current mage setup itself as a module, convert everything over to that, and then start internally refactoring to give it nicer interfaces and ux/dx

So the subtasks depend on the approach we go with. Both felt viable so i was leaving it up to whoever picked it up to make a call (or propose a different approach).

If i was to do it, I’d probably try out 2 and see how it goes in practice, falling back to 1 if it ends up being awkward or messy for whatever reason

[rocky harbor]

@here Project Zenith call starting in dev-audio if anyone wants to chat about zenith and/or modules you're working on 🙂

[thorn moat]

dagger shell seems to be broken in v0.9.4:

❯ dagger shell wolfi --packages go
✘ dagger shell wolfi ERROR [6.27s]
┃ Error: failed to get shell endpoint: input:1: container.withExec.shellEndpoint assignment to entry in nil map                                                                                                                                                                              
• Engine: 889c1dd5e15a (version v0.9.4)
⧗ 7.10s ✔ 54 ∅ 8 ✘ 1

[sharp zealot]

@signal perch we're happy to go over your module if you want

Tying the two topics of 1) interfaces and 2) simplified CLI:

It would be neat if each "action flag" in dagger call (ie. --exec, --publish, --print, --shell or whatever) each mapped to a specific interface.

eg. "--exec will call the function exec in the resulting object."
"--print will query all scalar fields, and print them"
etc

--publish will call publish

[latent trellis]

I've been rewriting some of my older, Mage-based Dagger pipelines to Zenith recently (eg. https://github.com/portward/portward/pull/45)

Previously I was paying attention to creating pipelines (overloaded term; here it means calling Pipeline so it appears nicely on Dagger Cloud), but not this time.

I have to say that both the migration and code organization were way easier. Not having to think about calling Pipeline at the right place and time removed so much of the mental overhead of creating new Dagger pipelines.

cc @ember walrus

[ember walrus]

I've been rewriting some of my older,

[plucky ermine]

I finally published a module I am proud of that does something useful 🤓

https://daggerverse.dev/mod/github.com/levlaz/daggerverse/launchdarkly@57cbc7fcd528163cb7b8dd5420eaae910cf2ba19

Allows you to run the code references utility to find all references to feature flags in your code and report back to LaunchDarkly.

[plucky ermine]

Do we need to do something special to run dagger call in a CI environment? Right now it looks like its causing a panic - I assume its because we are not in a TTY or something.

Caught panic:

runtime error: slice bounds out of range [3:1]

Restoring terminal...

goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x13
github.com/charmbracelet/bubbletea.(*Program).Run.func1()
        /go/pkg/mod/github.com/charmbracelet/bubbletea@v0.24.2/tea.go:440 +0x91
panic({0x1194640?, 0xc00035c618?})
        /usr/local/go/src/runtime/panic.go:920 +0x270
github.com/charmbracelet/bubbles/viewport.Model.visibleLines(...)
        /go/pkg/mod/github.com/charmbracelet/bubbles@v0.16.1/viewport/viewport.go:123

[thorn moat]

--progress=plain for a workaround, they're probably running with a TTY attached but not setting the window size, I think @warped canyon ran into this with CircleCI. Last time I looked into this it was a panic in an upstream dependency 😭

[warped canyon]

Yeah I've only had to set that with CircleCI so far

[plucky ermine]

Thanks so much! I am also confused because this panic did not return a non zero exit code - feels like an edge case we would want to address somehow.

You can see here: https://app.circleci.com/pipelines/github/levlaz/dagger-demo/1/workflows/b0382325-0450-429b-8480-a6ecbb926ca4/jobs/2

[plucky ermine]

Hm, I am seeing something strange now. I think I upgraded to 0.9.4 and now when I try to run dagger mod init I see this error:

levlaz@Levs-MacBook-Pro ci % dagger mod init --name=ci --sdk=go
✘ generatedCode ERROR [0.73s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [0.30s]
┃ Error: load package ".": err: exit status 1: stderr: go: go.mod requires go >=
┃ .21.5 (running go 1.21.3; GOTOOLCHAIN=local)                                  
✘ generating go module: ci ERROR [0.04s]
├ [0.00s] go mod tidy
┃ writing dagger.gen.go                                                         
┃ writing go.mod                                                                
┃ writing go.sum                                                                
┃ creating directory querybuilder                                               
┃ writing querybuilder/marshal.go                                               
┃ writing querybuilder/querybuilder.go                                          
┃ go: go.mod requires go >= 1.21.5 (running go 1.21.3; GOTOOLCHAIN=local)       
┃ needs another pass...                                                         
• Cloud URL: https://dagger.cloud/runs/97549a63-01e9-49fe-b4f4-51b7a0feb2d2
• Engine: fcb863982a1f (version v0.9.4)
⧗ 1.79s ✔ 16 ∅ 4 ✘ 3
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule.generatedCode failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

[sharp zealot]

@obtuse lion I've been thinking about your comment last friday, that you think of Dagger/Zenith as a platform for functions, more than as a platform for CI specifically. I'm curious to learn more about that, since "function" is such a common, open-ended term, what about Dagger's version of functions caught your interest? I would love to be able to convey that. The focus on CI is useful but in some cases (including yours) it can be too limiting, and obsfucate the full potential of the platform.

[sharp zealot]

A LaunchDarkly module! Cool @plucky ermine 🙂 https://daggerverse.dev/mod/github.com/levlaz/daggerverse/launchdarkly

[plucky ermine]

nginx module just dropped - will come in handy as an easy way to serve an arbitrary directory in the future 😄

https://daggerverse.dev/mod/github.com/levlaz/daggerverse/nginx@526f1a99c0ff9bae1dbe9276f226acc38c666b0e

[wintry prism]

module machine!

[sharp zealot]

Would be cool to have a discord notification somewhere when a new module drops. Perhaps in #github-feed ?

[sharp zealot]

this looks like it could be a killer module 😁 https://wiremock.org

[plucky ermine]

Today its pretty clear how to use dagger modules from other modules, but its not obvious to me how to use dagger modules from "original dagger" code, I was experimenting with things yesterday and ran into some strange behavior that I cant fully describe.

One question that came out of this for me was what happens to dagger run in a world of dagger call

Ultimately, do we imagine there being a clear integration or onramp from current dagger to modules, or does this assume people rewrite everything as a module.

[sharp zealot]

There are still question marks on that. See https://github.com/dagger/dagger/issues/5993

Actually would you mind asking your question on that issue @plucky ermine ? It might help get the conversation going

[plucky ermine]

Happy to, thanks!

[rocky harbor]

Mentioned a completely tangential idea at the end of this comment https://github.com/dagger/dagger/issues/6229#issuecomment-1853273864

That I think we should do either way. Basically, create a GetStarted module that opens an interactive shell and teaches you how to use Dagger. Should work because we can put the dagger CLI inside the interactive container and it will have access back to the dagger API, so should be able to work seamlessly.

Just throwing it out there in case anyone wants to bite 🙂 I'll create a separate issue for it either way

[warped canyon]

That's a really cool idea!

[rocky harbor]

Here's an issue with a few more thoughts: https://github.com/dagger/dagger/issues/6267

[sharp zealot]

@rocky harbor I have fallen behind on review time, but plan on reading & commenting later today

Do we have an issue for the topic of a module stdlib?

[rocky harbor]

No I don't believe so

[honest hearth]

Just random thought on stdlib, i like how deno does it as an end user, I can see what they officially support and then a list of third party modules: https://deno.com/

[warped canyon]

@rocky harbor I lost track of this one - did the constructors stuff make it in for 0.9.4?

[rocky harbor]

Yep! We also just merged docs yesterday: https://github.com/dagger/dagger/pull/6221 (not sure if deployed yet)

[warped canyon]

Awesome! My golang module broke with 0.9.4, and I'm assuming it's from this. Checking the docs

[sharp zealot]

Wait, I can register a function as the top-level of my module, with the arguments I want?????!?!?!?!

[rocky harbor]

Yes 🙂

[sharp zealot]

Nice!

I searched the zenith docs but couldn't find the part about that, FYI

dagger mod publish says it succeeded, but the resulting link fails with 404 in my browser..

https://daggerverse.dev/mod/github.com/shykes/daggerverse/hello@2d123e61569

[sharp zealot]

https://tenor.com/view/our-lives-are-about-to-change-eric-cartman-south-park-s12e8-the-china-probrem-gif-21171257

It's like a whole major release inside of a hidden feature inside of a patch release

https://tenor.com/view/omg-trippy-hand-endless-gif-15914653

[sharp zealot]

@rocky harbor does that mean I could potentially dagger call -m MYMODULE without arguments?

Also, do I pass my module's arguments as flags to dagger call? How do we avoid namespace clash with global flags?

I guess left or right of call will matter

[sharp zealot]

Still no luck. dagger mod publish siliently fails for me

[rocky harbor]

Right now the constructor still has to return the "main module object" (i.e. if your mod is named foo then in Go it has to return that Foo struct we autogenerate for you). But in theory it's now possible to support what you're saying. So you could just return whatever you want from the constructor

[rocky harbor]

We have an issue open for the general problem of conflicts in flags (it already existed due to --help), but yeah the current workaround in the case of a conflict would be to provide the flag to the left of call

[sharp zealot]

FYI I'm reading (and re-reading) your proposal @rocky harbor, trying to form a strong opinion, but it's hard...

[upbeat herald]

@rocky harbor I'm confused about something, if a function is returning void, what should be the argument sent in the call of returnValue?
I tried undefinied, {}, "", nothing do

And I have to send something as return value, so what should it be hmm?

[rocky harbor]

@Erik Sipsma I'm confused about

[upbeat herald]

@kind carbon A PR with basic tests for Typescript Zenith and a lot of fixes included
https://github.com/dagger/dagger/pull/6270

Some fixes are from the tomorrow's demo, other from issues I had while doing tests 🚀

[winter walrus]

Is daggerverse still on v0.9.3?
I ran into a 404 pushing a Python module earlier today. dagger mod publish completed just fine, but clicking the "+ publish" button shows that its failing due to https://github.com/dagger/dagger/pull/6183

My module: https://daggerverse.dev/mod/github.com/KGB33/daggerverse/rust@3b8e9d04c63e1ff56e03eeb6b21c1b993d41bd37

57: [0.66s] │ /opt/venv/lib/python3.11/site-packages/main.py:2 in                  │
57: [0.66s] │                                                                              │
57: [0.66s] │    1 import dagger                                                           │
57: [0.66s] │ ❱  2 from dagger import dag, function                                        │
57: [0.66s] │    3                                                                         │
57: [0.66s] │    4                                                                         │
57: [0.66s] │    5 @function                                                               │
57: [0.66s] ╰──────────────────────────────────────────────────────────────────────────────╯
57: [0.66s] ImportError: cannot import name 'dag' from 'dagger' 
57: [0.66s] (/sdk/src/dagger/__init__.py)
57: exec /runtime ERROR: process "/runtime" did not complete successfully: exit code: 1

[wintry prism]

Thanks for reporting @winter walrus ! I'll take a look.

cc @thorn moat

[ocean escarp]

a Dagger Module for uploading files to Cloudflare R2 or any S3 compatible storage : https://daggerverse.dev/mod/github.com/tsirysndr/daggerverse/r2-sync

[quick wind]

I'm getting a bunch of module 'dagger' has no attribute 'host', module 'dagger' has no attribute 'container' errors today after updating to 0.9.4. Skimmed the release notes and didn't see anything that should impact this? Don't think these are related other than perhaps being 0.9.4 Yeah ignore, global client change

[wintry prism]

Hi @winter walrus please try to publish your module again now 🙏

Should be good to go!

[winter walrus]

It works! Thanks.
https://daggerverse.dev/mod/github.com/KGB33/daggerverse/rust@3b8e9d04c63e1ff56e03eeb6b21c1b993d41bd37

[wintry prism]

Thanks again for bringing it up! Excited to check out your module too!

Oh and by the way @winter walrus.
Welcome to the ranks of the Module Builders!

[sharp zealot]

Hi all, just FYI, I'm still unable to publish my module...

@wintry prism

$ dagger mod publish -f
✔ dagger mod publish -f [0.55s]
┃ publishing github.com/shykes/daggerverse/hello@2d123e61569c7d80fbd1657a89754aab3aaee0bd 
┃ to https://daggerverse.dev                                                              
┃ published to https://daggerverse.dev/mod/github.com/shykes/daggerverse/hello@2d123e61569
┃ c7d80fbd1657a89754aab3aaee0bd                                                           
• Cloud URL: https://dagger.cloud/runs/2ec60308-56f8-483f-a145-21e78cd445da
• Engine: 05a6c1161a61 (version v0.9.4)
⧗ 2.30s ✔ 3

$ curl -q -I https://daggerverse.dev/mod/github.com/shykes/daggerverse/hello@2d123e61569c7d80fbd1657a89754aab3aaee0bd
HTTP/2 404 
content-type: text/plain; charset=utf-8
vary: Accept-Encoding
x-content-type-options: nosniff
date: Thu, 14 Dec 2023 16:25:37 GMT
content-length: 19
server: Fly/5f77bacf (2023-12-11)
via: 2 fly.io
fly-request-id: 01HHMJBPAVRW10EDT3EN6059XY-cdg

[thorn moat]

Ah, the CLI hasn't been updated for the new async publishing flow. It also still needs error-handling (obviously). Issue tracking it is here: https://github.com/dagger/dagger/issues/6066 - I can just pick it up now

[sharp zealot]

Ah, so it's a known issue that dagger mod publish always silently fails for everyone? Or is it something specific to me?

[thorn moat]

Well, it's a known issue now 😅

[sharp zealot]

Ah, I just tried manually publishing in the web UI, and indeed it worked on the first try

[thorn moat]

Previously it would silently fail if anything went wrong, but things just didn't usually go wrong. Now it fails because GET /mod/... doesn't trigger a publish, there's an async PUT /crawl flow now

adds dagger mod publish integration test to Daggerverse TODO list

[thorn moat]

https://github.com/dagger/dagger/pull/6274

[plucky ermine]

@warped canyon I was trying to use https://daggerverse.dev/mod/github.com/kpenfound/dagger-modules/dockerd@ec3133891c9cea4ec964c6fb597e62567dfecc7e yesterday but the container I was attaching kept complaining that docker was not running, do you happen to have any example of this working somewhere that I could look at?

[thorn moat]

I've got a Docker module too if you wanna give it a try! https://daggerverse.dev/mod/github.com/vito/daggerverse/docker@5a53a93aaa08e2caf02c249fe045ee0ae613c85e

[wintry prism]

He's off for a couple of days. Might lose a customer to @thorn moat 's module! 😉

There can be only...5

[thorn moat]

https://tenor.com/bcsA0.gif

[sharp zealot]

Damn I had a whole docker module, but got pulled into something else and never published it

missed opportunity

https://tenor.com/view/minions-me-raise-hands-pick-me-gif-13999775

[rocky harbor]

@here Zenith community call is starting in dev audio for anyone who wants to chat about Modules their working on or what we're cooking up 🙂

[sharp zealot]

will join from train station in a few minutes!

[sharp zealot]

Mmmmmm

@rocky harbor what if the module constructor was the answer to "I don't want to explicitly pass everything to each function of the module every time"?

You could conceivably pre-configure some arguments to some modules, and have the in a config file somewhere

The arguments to a module constructor are kind of like a module configuration

[rocky harbor]

Oh right, yeah sorry I was having trouble decoding all the streams of information during the meeting and missed the part of about the problem being around each function specifically. Constructors definitely solve that specific part of the problem since yeah what you return from the constructor is available to all the functions you define on the top level of your module.

CLI callers still have to explicitly pass the flags as of now, but it does at least make it more obvious how a config file like what you're suggesting would work

Since there's only one place to map config file args to, not N functions (that could all have different but overlapping signatures)

[plucky ermine]

Hey @sharp zealot I was trying to do the thing that you did during the podcast with Bret a few weeks ago using the dagger functions command against your supergit module, but it does not seem to work. Is this a known issue?

levlaz@Levs-MacBook-Pro public % dagger functions -m github.com/shykes/daggerverse/supergit
✘ load functions ERROR [2.11s]
├ [1.35s] loading module
├ [0.76s] loading objects
┃ Error: query module objects: json: error calling MarshalJSON for type *dagger.Module: retur
┃ d error 400 Bad Request: failed to merge schemas of [daggercore supergit]: schema validatio
┃ failed: input:2168: Field SupergitRemoteBranch.commit can only be defined once.            
┃                                                                                            
┃       | ...                                                                                
┃  2165 |                                                                                    
┃  2166 |         Return the commit referenced by the remote tag                             
┃  2167 |                                                                                    
┃  2168 |         """                                                                        
┃  2169 |         commit: SupergitCommit!                                                    
┃  2170 | }                                                                                  
┃  2171 | """                                                                                
┃       | ...                                                                                
• Cloud URL: https://dagger.cloud/runs/168c275e-3f05-4cb0-afa3-8ad041232ba4
• Engine: a94d9042e856 (version v0.9.4)
⧗ 3.45s ✔ 29 ∅ 6 ✘ 1

This is also a bit strange behavior because I would only expect functions to expose "public" go functions but I see a bunch of extra stuff here in my module that only has a single public function called "Find"

levlaz@Levs-MacBook-Pro public % dagger functions -m github.com/levlaz/daggerverse/launchdarkly
✔ dagger functions [0.00s]
┃ object name     function name         description                                          
┃                                                                 return type                
┃ *Launchdarkly   launchdarklyVersion   LaunchDarkly Container version (default: latest)     
┃                                                                 String!                    
┃ *Launchdarkly   token                 LaunchDarkly token                                   
┃                                                                   Secret!                  
┃ *Launchdarkly   directory             LaunchDarkly directory to scan (dir)                 
┃                                                                   Directory!               
┃ *Launchdarkly   project               LaunchDarkly project id (projKey)                    
┃                                                                   String!                  
┃ *Launchdarkly   repo                  LaunchDarkly repo name (repoName)                    
┃                                                                   String!                  
┃ *Launchdarkly   find                  example usage: "dagger call find --token $LD_ACCESS_T
┃ OKEN --project $LD_PROJ_KEY --repo $LD_REPO_NAME --directory ."   String!                  
• Cloud URL: https://dagger.cloud/runs/42acb8f1-fa43-49ad-bcbe-9be01dd878ee
• Engine: a94d9042e856 (version v0.9.4)
⧗ 6.53s ✔ 73 ∅ 3
levlaz@Levs-MacBook-Pro public % 

[sharp zealot]

@plucky ermine I'm getting the same error. I suspect 0.9.4 broke my module. Is that a known possibility @rocky harbor ?

[rocky harbor]

@Lev Lazinskiy I'm getting the same

[rocky harbor]

This is also a bit strange behavior

[subtle plinth]

I'm wondering if there's a trick to using Dagger as a dev environment beyond an infinite select {}?

func (m *RawkodeAcademy) Dev(ctx context.Context) {
    supabase := dag.Supabase().DevStack("rawkode-academy", "http://localhost:4321")

    tunnel, err := dag.Host().Tunnel(supabase.Studio()).Start(ctx)
    if err != nil {
        panic(err)
    }

    defer tunnel.Stop(ctx)
    select {}
}

I've tried this, but my Supabase studio isn't available locally

Rubber ducked, I think I found the problem 😂

[subtle plinth]

Does anyone have an example of running a service and exposing to localhost indefinitely?

Oh, there's a dagger up I should be using instead of dagger call

projects/rawkode.academy/dagger on  main  [$⇕]
❯ dagger up dev -n
✘ dagger up dev ERROR [0.09s]
┃ Error: unexpected type <nil>
• Engine: 1f7c70872fef (version v0.9.4)

I've added --debug and --progress=plain, but I'm not getting extra information about this nil.

Any debug tricks I could use?

[winter walrus]

I think its supposed to be dagger up -n dev .
I'm pretty sure flags before your function (dev) are for dagger, flags after are passed as args to it.

[subtle plinth]

It's the same result either way 😦

[spiral whale]

did you return a correct return type from dev function?

[subtle plinth]

Yeah

[sharp zealot]

I'm finding myself writing code snippets using Dagger, and being annoyed that they either make sense pre-Zenith, or post-Zenith, but not both. Isn't there a change to the pre-Zenith DX that we could ship quickly, that makes the diff less painful?

I remember @chrome pilot mentioned the idea of removing the need for a dagger.Client that you pass around everywhere. Instead there could be a global client variable already available. Maybe we land on our feet with that, and make it so pre-Zenith and post-Zenith snippets are the same from day one? It would be one less moving part to worry about at launch.

[subtle plinth]

Oh

So when we use dagger up, we need a func that returns a *Service

[sharp zealot]

Right.

Also, hello 🙂

[subtle plinth]

It's working 😄

[sharp zealot]

dagger up is just a slight variation of dagger call that does further processing on the result of the pipeline after calling it (specifically: running it and forwarding its ports)

[subtle plinth]

I started by using dagger call and adding select {} to force the endless loop after creating a tunnel

but dagger up is much nicer

I'm making an assumption now that if I return a single service and it has dependencies, that those dependencies will stay up?

Which I think isn't the case, drats

[spiral whale]

Are you trying to start multiple services together or use started service on container?

[thorn moat]

dependencies of a service should stay up, but tunnels won't be established for those, only the "main" one

[subtle plinth]

Cool. There’s a lot going on in the stack, so maybe there’s an error in a dependant service which I’m not able to see

[thorn moat]

could try running with --focus=false, e.g. dagger --focus=false up -n

[sharp zealot]

I'm curious what you think @rocky harbor @thorn moat @kind carbon @deft rain 👆

[subtle plinth]

Got it working, thanks @thorn moat

I had to tweak my service bindings to force some services to stay up, but it seems happy now 😄

It's quite a chunky Supabase module

https://github.com/RawkodeAcademy/RawkodeAcademy/blob/main/projects/dagger/supabase/main.go

[spiral whale]

supabase module

[rocky harbor]

I think it makes sense and is not likely too hard to do, opened an issue here https://github.com/dagger/dagger/issues/6289

[subtle plinth]

Any plans to support OCI for modules?

[sharp zealot]

All modules run inside OCI containers 🙂

But I'm guessing you mean using OCI registries to store and distribute the modules at rest. Are you thinking the binary artifacts, or the source code?

[subtle plinth]

I meant registries, yeah 😅

Binaries or source. Not fussed. Git clones are slow, especially on larger repositories.

[sharp zealot]

The idea is to keep everything git-based, and aggressively leverage caching, both locally and in a global CDN run by us. Same model as Go modules

[subtle plinth]

Not an ideal situation for mono repositories

[plucky ermine]

This isn't really zenith specific, but feels like its slightly harder to debug in modules, but I just ran into another case where I had a weird bug and the fix was to use "sh" "-c" -- do we have any documentation on when you must use this?

[wintry prism]

Could you expand on that?

This isn't really zenith specific, but

[kind carbon]

Yeah, that's the case for Python and NodeJS today since they both did this before starting support for modules. I showed it with Python in my last community call demo. I'm going to submit an issue tomorrow to update the docs (PTO today). Been wondering if it was worth it to invest in doing the same for Go.

[subtle plinth]

OCI Registry Module Distribution

[latent trellis]

Let's say I have a Go module that creates a container and sets some cache volumes:

        WithMountedCache("/root/.cache/go-build", dag.CacheVolume("go-build")).
        WithMountedCache("/go/pkg/mod", dag.CacheVolume("go-mod"))

If I initialize a container like that multiple times in the same session (eg. build two separate go binaries), are those going to be the same cache volumes or different ones?

If they are the same and I want to make sure that different ones are used, does that mean I have to add a name parameter to my Go build container?

[latent trellis]

It seems to me that my builds are stepping on each others toes and bust the go build cache for each other.

[wintry prism]

That's what I'd expect too Márk.

[latent trellis]

That's what I guessed. Some sort of a build or pipeline identity would be useful to differentiate between cache volumes, but I think it would be really uncomfortable if it had to percolate through the chain of modules from the top through function calls.

[latent trellis]

In other news: here is my fully daggerized Helm release workflow: https://github.com/openmeterio/openmeter/pull/481

Build helm chart package (useful for local testing and running linters):

func (m *Build) HelmChart(version Optional[string]) *File {
    chart := m.helmChartDir()

    opts := HelmBasePackageOpts{
        DependencyUpdate: true,
    }

    if v, ok := version.Get(); ok {
        opts.Version = strings.TrimPrefix(v, "v")
        opts.AppVersion = v
    }

    return dag.Helm().FromVersion(helmVersion).Package(chart, opts)
}

func (m *Build) helmChartDir() *Directory {
    chart := dag.Host().Directory(filepath.Join(root(), "deploy/charts/openmeter"), HostDirectoryOpts{
        Exclude: []string{"charts"}, // exclude dependencies
    })

    readme := dag.HelmDocs().FromVersion(helmDocsVersion).Generate(chart, HelmDocsBaseGenerateOpts{
        SortValuesOrder: "file",
    })

    return chart.WithFile("README.md", readme)
}

Here is how you release and publish to GHCR:

func (m *Ci) Release(ctx context.Context, version string, githubActor string, githubToken *Secret) error {
    var group errgroup.Group

    group.Go(func() error {
        chart := m.Build().HelmChart(Opt(version))

        _, err := dag.Helm().FromVersion(helmVersion).
            Login("ghcr.io", githubActor, githubToken).
            Push(chart, "oci://ghcr.io/openmeterio/helm-charts").
            Sync(ctx)
        if err != nil {
            return err
        }

        return nil
    })

    return group.Wait()
}

Here is the GHA part:

      - name: Release
        run: nix develop --impure .#ci -c dagger call release --version ${{ github.ref_name }} --github-actor ${{ github.actor }} --github-token ${{ github.token }}
        env:
          DAGGER_CLOUD_TOKEN: ${{ secrets.DAGGER_CLOUD_TOKEN }}

      - name: Stop Engine
        run: docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)
        if: always()

Relevant modules:

https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/helm
https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/helm-docs

[sharp zealot]

💯 it’s a known issue: https://github.com/dagger/dagger/issues/3345

[latent trellis]

Thanks Solomon! I commented on the issue.

[sharp zealot]

@upbeat herald hey do you still have a Dagger Module that can test other Dagger Modules? 🙂 I want to setup a CI for my modules...

[upbeat herald]

Hey, it's a custom module I made to test mine, it's specialized for my use case so I'm not sure you can use it, however I made a module to call the dagger CLI, idk if that's useful?

My CI module: https://github.com/quartz-technology/daggerverse/tree/main/ci
My Dagger CLI module: https://github.com/quartz-technology/daggerverse/tree/main/dagger

[latent trellis]

Is there a workaround for this until the fix gets released?

https://github.com/dagger/dagger/issues/6272

[plucky ermine]

I am having a tough time translating how to work with and export directories and files in zenith vs traditional dagger.

I am trying to mount the cwd using WithMountedDirectory(path, src). and it seems to mount the directory that has the module in it instread.

    path := "/src"
    src := dag.Host().Directory(".")

In contrast, this worked before I refactored as a module

src := client.Host().Directory(".")

[sharp zealot]

Yeah this is a combination of three things:

1. In a module you can't access the host filesystem of your client in that way (because you're running inside a container)
2. However you can still run dag.Host(). It's just that it's relative to your own module - because your module code is now the client, making requests to the Dagger API. The module's container is the host (note: this might make more sense if instead of "host" it was called "client").
3. Separately, there was the need to have a call to access the module's own source code (to access assets embedded with the source code, like data files, config files, etc). Instead of creating a new call, the reasoning was: let's just mount the module's source in the module's container's workdir, as a convenience. It will be like an implicit API - a convention.

Each of these steps make sense individually, but the end result might still be confusing to developers without all this context. Which is what you're experiencing now.

[rocky harbor]

🐞 runtime error: index out of range [0]...

[rocky harbor]

What Solomon said, I'll also add that one way to use filesystems from the caller is to have your function accept an argument of type *Directory, which then allows the caller to pass it explicitly:

func (m *MyMod) MyFunc(dir *Directory) error {
  //...
}

That can then be called via dagger call my-func --dir ./some/dir

This is more explicit for the caller, but when it comes to their filesystems on their host, being explicit about what they are passing is not necessarily the worst thing.

[sharp zealot]

Oops sorry, I addressed the confusion but didn't talk about how to actually solve the problem 🙂

[rocky harbor]

No worries, my queue of responses to get to was piling up so I was glad to have help 😄

[plucky ermine]

Thanks all for your help - I think this speaks to a slightly bigger theme of "I wish dagger was more like make sometimes (or even hugo as a more modern example)" where I could be explicit when needed, but otherwise just run a single command and it does the right thing.

Funny enough I got all this working properly in my LD module since I have directory as an argument.

Also its worth noting that in this particular case I am not making a module to be used as a library, but instead writing a module that is going to be the "main" CI module that uses other modules and runs my entire CI pipeline.

It feels that in this particular case "run CI in the current working directory" is a very common pattern that I would love to figure out a better DX around that does not require folks to be explicit.

In traditional CI there would be a checkout step that happens and then subsequent commands run in the root of the project. I guess this would make sense too using supergit, but locally it makes less sense since I already have all the files I need.

[plucky ermine]

I got this working for getting data in but still stuck on how to get data out - if I wanted to export a directory at the end back to my local computer (without calling dagger export) is it possible?

FWIW I am not opposed to dagger export it works great! But in this case I have a pipleine with a bunch of things happening, exporting the directory is like step 3 of 10 so it feels like calling dagger export in this case does not make a ton of sense.

[rocky harbor]

Yeah right now you have to return the Directory from your function (e.g. func (m *MyMod) MyFunc() (*Directory, error), you can leave the error out if it's not needed), then callers need to run dagger download my-func.

But in this case I have a pipleine with a bunch of things happening, exporting the directory is like step 3 of 10 so it feels like calling dagger export in this case does not make a ton of sense.
Yeah depending on the use case a pattern like this can also make sense:

func (m *MyMod) MyFunc() (*MyResult, error) {
  // do stuff ...
  return &MyResult{
    TheDir: someDirYouCreated,
    SomeOtherStuffYouWantToReturn: foobar,
  }, nil
}

type MyResult struct {
  TheDir *Directory
  SomeOtherStuffYouWantToReturn string // or whatever type it may be
}

In which case callers can get the dir via dagger download my-func the-dir or access the other results via something like dagger call my-func some-other-stuff-you-want-to-return (may not be dagger call depending on what the actual type of SomeOtherStuffYouWantToReturn is)

[sharp zealot]

@rocky harbor doesn’t the “simplify CLI” plan add a complication here?

[rocky harbor]

Not really, it at least simplifies everything to just dagger run. So dagger run my-func the-dir export --path=./some/where and dagger run my-func some-other-stuff-you-want-to-return.

Other than that it's orthogonal fwict

[sharp zealot]

If you call “directory.export” from your module code it exports to the container fs. If you call it from the CLI it exports to the host fs. Wondering if it will confuse people? Like watching Inception and wondering “wait which dream are we in right now?”

[rocky harbor]

I agree that can be confusing, but it's the exact same situation+considerations as the inverse direction of import (i.e. same considerations here #daggernauts message)

[plucky ermine]

Thank you very much!

[sharp zealot]

It feels little different I think, because the import is presented to the user as passing an argument (via CLI flag) , but export is not presented as a “return” which would be symmetrical. Not questioning the CLI UX decision, I still like it, just wondering how it affects the DX of the host API

[plucky ermine]

This all works for me, but I think my whole mental model is still catching up to how all of this works.

I thought I wanted to put the directory on my local machine to then keep going with the next part of the pipeline, but now I realize this whole step is unnecessary. The only reason I wanted to do that is so I could see things working.

There is no actual reason to export the directory in step 3, I can just pass it to other parts of the pipeline within the module "host". This took a bit of trial and error to unlock in my brain. But now that I am here, its actually fine and everything works pretty logically.

Just need to remember that I am programming some other container and not my local laptop when I am writing this sort of "main" module.

[rocky harbor]

Yeah that makes total sense, there is for sure a shift in mental models required for all this. The goal is that A) it's relatively minor and doesn't require e.g. taking a course to understand and B) the end result is satisfying+useful, even if the road to it has some unavoidable bumps.

I'd also add that there's some interesting ideas floating around that could potentially serve as a bridge of sorts. @obtuse lion had an idea around a generic module that allows you to create pipelines that consist of a series of "steps" that each operate "something that returns a Directory". So you are just passing a Directory through steps that each modify it.

The end result of which would potentially be a lot closer to how a lot of users currently think about this whole problem space. But while also not changing how dagger works in any fundamental ways.

The pre-req for that sort of model is interface support, which I have working in a PR that I'm slowly chipping away at in the background (CLI stuff is higher priority, so only spending a little time each day on interfaces right now, but should be merged in the fairly near future): https://github.com/dagger/dagger/pull/6254

[plucky ermine]

@sharp zealot its not clear to me if your supergit module was made for doing push type things with git (which is kind of an anti pattern I guess during CI time) but in case it was, how would you approach auth or keys?

Perhaps a better question, is what is the zenith way to do this: https://docs.dagger.io/723462/use-secrets/#use-secrets-from-the-host-filesystem

[sharp zealot]

supergit auth

[sharp zealot]

I'm trying to develop on the CLI (nothing fancy, mostly UI), but getting this error on dagger call:

$ ./dagger call -m github.com/shykes/daggerverse/hello message
✘ load call ERROR [17.36s]
├ [17.21s] loading module
├ [0.15s] loading objects
┃ Error: query module objects: input:8: Cannot query field "constructor" on type "ObjectTypeDef".      
✘ checkVersionCompatibility(version: "0.9.3") ERROR [0.00s]
• Cloud URL: https://dagger.cloud/runs/08d9582f-045d-4696-a683-22b34cf40ece
• Engine: a722567dc2db (version main)
⧗ 27.65s ✔ 71 ✘ 2

[sharp zealot]

@thorn moat I'm hitting this again: #1175211498675634266

[cold dawn]

To add to a use case that I'm looking at, having "old school" dagger code being able to use modules would be super useful.

We're looking at integrating dagger into our mono repo and having independent project creating and running a dagger client from within that projects test suite. It would be great to have a module for say sql / redis ... that each of these projects could share

Code that I currently have is:

mysqlService := cli.Container().
  From("mariadb:10.11.2").
  WithEnvVariable("MARIADB_USER", "user").
  WithEnvVariable("MARIADB_DATABASE", "dns_test").
  WithExposedPort(3306).
  AsService()

and I was about to copy pasta that into other projects, but then moved it over to a module:

func (m *Mysql) WithDatabaseName(name string) *Mysql {
    m.databaseName = name
    return m
}

func (m *Mysql) AsService() *Service {
    var (
        databaseName = m.databaseName
    )
    if databaseName == "" {
        databaseName = "dns_test"
    }
    return dag.Container().
        From("mariadb:10.11.2").
        WithEnvVariable("MARIADB_USER", "user").
        WithEnvVariable("MARIADB_DATABASE", databaseName).
        WithExposedPort(3306).
        AsService()
}

With the aim of using this in each project as

mysql.AsService()

or

mysql.WithDatabaseName("testing").AsService()

as of just now, I have the following error when importing the module, which I guess is expected

... is a program, not an importable package ...

@plucky ermine , I think we're due to book in a call, at least my unread emails tell me I need to reply to you! I'm not really free until the new year, but happy to talk async here

[subtle plinth]

Does using a Dagger module break the ability to call os.Getenv() ?

[warped canyon]

at the moment, yes, since modules don't have host access at all. They have to be passed in as parameters

[subtle plinth]

Drats

Can we get dagger run --env-passthrough or --env-passthrough-prefix?

[warped canyon]

We've talked about something like that a couple of times, I'm not sure if there's an issue where that discussion is taking place

[restive shore]

related: https://github.com/dagger/dagger/issues/6077, https://github.com/dagger/dagger/issues/6112

[subtle plinth]

Subscribed

Thanks

Looks like @sharp zealot already suggested the cli flag idea, so that's good.

I hope this is an easy fix, because the DX of dagger up dev is phenomenal, but without being able to pass in some secrets - it's a bit of a bust

"easy" he says 😂

[sharp zealot]

You can pass all the secrets you want right now already, as CLI flags. That works today

[subtle plinth]

Yeah. I just don’t like that approach

I’m happy to wait

I don’t like it because I’ve got 8 environment variables I need access to

And I could forgo that and instead pass my infisical token instead and fetch with the SDK in the module; but I don’t want to pass a super secret via a cli parameter either

[warped canyon]

This is what I've done for some demos 🙂 fwiw you don't have to pass it as plain text, you can say --infisical-token $INFISICAL_TOKEN

https://github.com/kpenfound/greetings-api/blob/main/ci/main.go#L111

[subtle plinth]

True, but at some point I’m gonna mess that up and probably on a live stream 😂

[subtle plinth]

I notice that from a module we can do dag.Host().File() and .Directory(), so in-theory we could eventually do .Env("SOMETHING")?

[rocky harbor]

I notice that from a module we can do

[subtle plinth]

I’ve just realised, that if I want to build my entire monorepo I need to mount the entire codebase into a container now?

[rocky harbor]

You don't need to mount it in necessarily, you can write a function like:

func (m *MyMod) MyFunction(monorepoDir *Directory) {
   // do stuff with monorepoDir
}

Which then allows you to pass the dir from the CLI like:

dagger call my-function --monorepo-dir ./a/local/path/to/it

Or (using the currently undocumented git ref support), something like:

dagger call my-function --monorepo-dir https://github.com/someorg/monorepo#v0.0.1

[subtle plinth]

Doesn’t dagger call send the entire current directory into a container to evaluate function availability?

[rocky harbor]

It sends in the module source code yes, but by default that's just the dir rooted at where the module's dagger.json is located (so may or may not be the whole monorepo depending on layout). There's also undocumented support for include/exclude settings in dagger.json right now that let you filter if needed.

We're planning on refactoring all this to be more clear as part of https://github.com/dagger/dagger/issues/6291, but happy to help more in the meantime if I can

[subtle plinth]

I don’t need a main module, I think I’ll go back to slinging clients around and working on the host.

That’s not going away, right?

[sharp zealot]

@subtle plinth we had missed you 😁

[subtle plinth]

Ha. And here I thought you were wishing I’d disappear again 😂

[rocky harbor]

We're not planning on removing support for running SDKs directly on the host no, but you currently can't get bindings to module APIs for SDKs running on the host (there's an issue for that too, it's planned)

[sharp zealot]

Rawkode: bringing out the “stress” in “stress testing” 😛

[subtle plinth]

The DX on dagger up is fantastic but needing to run it in a container does introduce a lot more complexities

[rocky harbor]

As a consolation to everyone, these are at least so far all things we're aware of and on our radar to fix soon 😅 So at least no new wrenches

[sharp zealot]

@subtle plinth our goal is to convince you that it solves way more complexity than it creates

[rocky harbor]

Is the main problem that you want to avoid loading the whole monorepo?

[sharp zealot]

most of the UX issues with invoking a module are focused at the entry point. Whereas the issues with a non-modularized pipeline are spread out across your code, so they get worse as your code grows.

[subtle plinth]

Yeah, that’s one of them. I want to be able to run a build at the macro / repo level and run everything that’s changed

But I also need to be able to work on individual services

I thought that modules were the answer but in order for a module to have a dependency “up” the tree, we need to load everything

You’re gonna end up with a Dagger equivalent of a BUILD file it feels like?

[rocky harbor]

There's ways to avoid that today, it's confusing and undocumented (which is intended to be fixed by that issue I linked above). But in the meantime I can help unblock you if you want. I just need to know the basic structure of what you're trying to achieve (i.e. the basic layout of the directories with modules and the directories that you don't want to load)

[subtle plinth]

If I have 5 directories, each with its own project and dagger module; can I list all the functions without mounting everything?

My initial approach was to keep the dagger modules in a subdirectory of each project, but then it has no access to the code to build

I don’t need to fill this channel, sorry. We can thread or discuss in forum channel

[sharp zealot]

My initial approach was to keep the dagger modules in a subdirectory of each project, but then it has no access to the code to build

For that part specifically: https://github.com/dagger/dagger/issues/6237

[rocky harbor]

PR for the improved secret passing support discussed above here https://github.com/dagger/dagger/pull/6314

[sharp zealot]

By any chance, is anyone still around who is familiar with the internals of the zenith CLI?

[rocky harbor]

By any chance, is anyone still around

[sharp zealot]

I am using zenith on a slow internet connection, and it is quite slow - looking for low-hanging fruits that might not make a difference on high-speed links, but will be noticeable on slower ones

For example, the SDKs reference a base image. Maybe they could pin a digest? Would save a round-trip as buildkit looks up the latest digest each time

[sharp zealot]

If I wanted to contribute a performance optimization to a Zenith SDK, where would I go? Is it under sdk/ in the top-level Dagger repo? Or somewhere else?

[rocky harbor]

@here chatting about Zenith in dev-audio for the next hour, happy to discuss anything you've been working on or any questions you have 🙂

[rocky harbor]

If I wanted to contribute a performance

[lapis willow]

Is there some clever way to parse src files loaded into a zenith module with native golang code?

I've got some code that walks a file directory for go generate commands, and turns it into an array of []string so that I can pass it to dagger WithExec .

Would love to just call this directly, rather than compiling and doing some stdout parsing, but I think that's what I might have to do

[plucky ermine]

I am running into an issue with trying to follow the docs for python SDK, has anyone gotten this to work?

https://docs.dagger.io/zenith/developer/python/419481/quickstart

levlaz@Levs-MacBook-Pro module % dagger call container-echo --string-arg "Hello"
✘ load call ERROR [56.08s]
├ [0.20s] loading module
├ [55.88s] loading objects
┃ Error: query module objects: json: error calling MarshalJSON for type *dagger.Modul
┃  returned error 400 Bad Request: failed to get schema for module "module": cannot a
┃ ach new fields or functions to object "Module" from outside module     

[void widget]

Given this: https://github.com/levlaz/daggerverse/blob/f9ed02a6c9bf60a787218b73d29b4cd9eef2fc98/openring/main.go#L27

How do you pass a file as an arg using the CLI:
dagger call -m github.com/levlaz/daggerverse/openring@f9ed02a6c9bf60a787218b73d29b4cd9eef2fc98 openring --sources <FILE>

it seems that i just pass the file as an arg, just wanted to double check

Same for a container, if a func accepts a ctr, is it relevant to show the cli equivalent, because I dont know how to add it as a param

[rocky harbor]

Yep that's how it works from the CLI, you can just pass the path to the file as the argument value

[rocky harbor]

For containers, you can pass a registry reference, so if the function accepts an arg foo of type Container, then in the CLI you can do e.g. --foo docker.io/library/alpine:latest

[void widget]

Thanks, do we have documentation on the cli behavior regarding types ? Did not find it 🙏

[rocky harbor]

Not yet! The docs are still very barebones. We're finalizing+stabilizing the CLI over the next few weeks, once that's done it there will be lots of doc writing 🙂

[void widget]

I am exploring the generation of CLI examples for the daggerverse from the modules schema. Is it still too early ? Having a first implementation might be helpful (on our side), or everything will change drastically, not worth it yet

[latent trellis]

I have a dockerfile that starts like this:

FROM --platform=$BUILDPLATFORM tonistiigi/xx:1.3.0@sha256:904fe94f236d36d65aeb5a2462f88f2c537b8360475f6342e7599194f291fb7e AS xx

FROM --platform=$BUILDPLATFORM golang:1.21.5-alpine3.18@sha256:d8b99943fb0587b79658af03d4d4e8b57769b21dcf08a8401352a9f2a7228754 AS builder

COPY --from=xx / /

Is there an easy way I can achieve the same with Dagger (notably, the COPY part)?

[rare nimbus]

hi, after a few months of using "classic" dagger, I've finally spent some time with Zenith, converting some Go code to Dagger modules.
And I have to say that the experience is really good, mainly thanks to the dagger CLI. Being able to easily open a shell on the resulting container is pretty cool. It really solves a lot of pain points I had with the classic SDK.

My main concern now is to bridge the gap between the classic SDK and the new modules. From what I understand, these are 2 different things - at least for the moment - and it's not possible to import/call a module from the classic SDK - and just relying on a module to build a project is too limited - at least for our use-case.

So if I continue migrating my code to modules, I'll have to use the CLI to call them - or wait until "missing" features such as https://github.com/dagger/dagger/issues/6237 are implemented?

[void widget]

I am getting this issue from Solomon's supergit module (https://daggerverse.dev/mod/github.com/shykes/daggerverse/supergit@4113b803fcf4ba83b39cd464856af94656197cbf):

dagger functions -m github.com/shykes/daggerverse/supergit@4113b803fcf4ba83b39cd464856af94656197cb 
✘ load functions ERROR [0.61s]
├ [0.61s] loading module
┃ Error: failed to get module config: resolve git ref: input:1: git.commit.commit failed to load cache key: repository does not contain ref 4113b803fcf4ba83b39cd464856af94656197cb, output: ""               
✘ commit ERROR [0.60s]
✘ git://github.com/shykes/daggerverse#4113b803fcf4ba83b39cd464856af94656197cb ERROR [0.60s]
• Engine: a8ed07519657 (version v0.9.5)
⧗ 1.98s ✔ 7 ✘ 3

And the repo seems to be on the proper hash: https://github.com/shykes/daggerverse/tree/4113b803fcf4ba83b39cd464856af94656197cbf/supergit

Any idea on how to circumvent that ? When cloning it locally, I'm able to find the commit hash on main

[rare nimbus]

you're missing a f at the end of the commit hash 😉

[void widget]

Ahahah, so obvious that i didn't see it. Thanks 🙏

[ember walrus]

During today's community call, we'd like to talk about the user experience for publishing Dagger modules. If you'd like to give feedback but cannot make the call today, please reply.

@wintry prism will lead the discussion

[hexed flume]

@ember walrus I don't see the today's event. The next one is the 12th... 🤔

[ember walrus]

@sleek nest

[ember walrus]

Thank you for pointing that out. I don't think it was intentional to not have a call for today.

[ember walrus]

To clarify - I didn't mean the official community call that happens every two weeks. This is the weekly Zenith call.

Which happens every Friday

[void widget]

Hey Erik,

Following the discussion during the community call, atm a module can only reference types that have been declared in the module (from what I understood) ? For example, for https://daggerverse.dev/mod/github.com/shykes/daggerverse/supergit@4113b803fcf4ba83b39cd464856af94656197cbf, the repository.Commit func returns a Commit type which has been declared in the supergit module:

commit(digest: String): Commit

But, at the moment, for example, is it impossible for any of the functions of supergit to return another module referenced on the daggerverse, as my Cargo type declared in my other cargo module ?

I've read a bit your interface PR, and am trying to understand the impact on the index presentation this will have on the Daggerverse (to anticipate, and if necessary). I am also a bit confused on how a dagger call module-with-interface CLI exec looks like. Does that work atm, like can I already hack some stuff and execute those interfaced modules on the CLI, to see the behavior / patterns ?

From what I grasped, this won't have much impact: users will continue to not be able to return another module type. And, on the contrary, the interfaces available for interoperability will be available in the module schema: instead of having the schema as of now, we will have, for a given schema, a few other objects representing all the interfaces available / or all the interfaces used ?

[latent trellis]

I wonder what the best way would be access a Directory as a git repository. From what I can tell, neither dag.Git nor supergit works with "local" git repositories. My particular use case is detecting the commit hash of HEAD and getting the commit creation time. I can certainly just run git commands in a container, but I wonder if this use case fits into any of the existing tooling/modules available for Git repos in Dagger.

[rocky harbor]

Hey Erik,

[latent trellis]

I started to confuse the dagger and go CLI tools. 😆

go mod sync vs dagger mod tidy

I've done it at least a dozen times

[latent trellis]

I quite like the new constructor feature for modules.

I managed to simplify my go module quite a bit by removing some of the from/with functions, and I actually managed to remove an entire layer (struct/submodule).

Here is the change: https://github.com/sagikazarmark/daggerverse/pull/19/commits/aacd66e0c00534d982d06c4da2f66147986f2317

One slight concern I had (though I'm not actually sure this is an issue): technically the module is backwards compatible with older dagger versions which means I either call New (or equivalent) in functions that rely on the initial state when it's missing or risk getting weird errors/panics. It's probably not an issue at this time, but as new features are getting added, new versions are being released it may become one.

I was thinking adding a minimum version constraint in dagger.json would help with that.

For now, I just added a badge to the readme specifying the minimum version.

[latent trellis]

Is there a way to get the exit code of a command in zenith?

There is a ExecError in the SDK, but it's not in the generated code.

Can I still use it from the SDK? (My gut tells me no, but I was lazy to test it so far)

[latent trellis]

While rewriting a GHA workflow to pure Dagger (ie. no initial checkout step) I noticed that the dagger does not seem to work with PRs. More specifically, the temporary commit created for the pull request is not being pulled (which is somewhat understandable).

This is true for both dagger call and dag.Git.

For dag.Git there is actually a workaround. Instead of the following code:

        source = dag.Git("https://github.com/openmeterio/benthos-openmeter.git", GitOpts{
            KeepGitDir: true,
        }).Commit(c).Tree()

You can use supergit:

        source = dag.Supergit().
            Repository().
            WithRemote("origin", "https://github.com/openmeterio/benthos-openmeter.git").
            WithGitCommand([]string{"pull", "origin", ref}).Worktree()

For the module itself it's actually not that important to use the merged version (although it would certainly be better), so for now I just went with github.event.pull_request.head.sha in the GHA workflow to use the head commit.

Has anyone had this issue before? Any other methods to workaround the problem?

Given this is a common use case, I think it would make sense to improve the user experience in Dagger.

[plucky ermine]

While rewriting a GHA workflow to pure

[plucky ermine]

[zenith] New source field in dagger.js...

[restive shore]

Getting back into some Zenith dev after holidays. I see that the packaged go SDK version is 1.21.3, Locally I'm on 1.21.5. So even dagger mod sync fails with

Error: load package ".": err: exit status 1: stderr: go: go.mod requires go >= 1.21.5 (running go 1.21.3; GOTOOLCHAIN=local)

I see this is as a pain point for developers having to juggle SDK versions. Is there a workaround other than changing my local Go version?

[rocky harbor]

Yeah there's fortunately an easy fix to just change go.mod to say go 1.21 instead of 1.21.5. Issue for fixing this here: https://github.com/dagger/dagger/issues/6329#issuecomment-1874220377

Basically, there's not actually an incompatibility. The go toolchain did something in 1.21 that results in the minor versions being specified which we just need to update to handle better.

[restive shore]

That worked. Also note, I had to change the version to go 1.21 in my go.work file too.

[rocky harbor]

Good to know, thanks! Added a note to the issue: https://github.com/dagger/dagger/issues/6329#issuecomment-1885786593

[rare nimbus]

I think it's because you have GOTOOLCHAIN=local. if you use GOTOOLCHAIN=auto (which should be the default I believe), then go will be smart enough to download and use the right version - see https://go.dev/doc/toolchain for more details

btw I'm using 1.21.5 locally too, and my go.mod for my module requires go 1.21.3, and I have no issue

[restive shore]

Interesting because I don't have it set anywhere. When I do a go env I see GOTOOLCHAIN='auto'

[dense canyon]

@rocky harbor @deft rain before the next release: did anyone raise an issue about function directory arguments not invalidating the cache when changed? We were doing some tests with @plucky ermine today and we believe we came across a weird scenario with that. If you give me 10m I can try to get a repro

[dense canyon]

i.e dagger call my-func --dir . and then using the passed *dagger.Directory directly in WithMountedDirectory within a function

[rocky harbor]

That's a new one, can you open an issue w/ the steps to repro (including what change is made to the local . dir)?

[dense canyon]

on it 🙏

[rocky harbor]

I thought that too but I can repro the same issue and go env shows GOTOOLCHAIN='auto' for me, so needs some more investigation to pin down exactly what scenario this is happening in. Good to know it works for you though. What platform are you on btw (linux, macos, windows, etc.)? Just in case that ends up being relevant somehow

[deft rain]

just a quick check - there's not an export to somewhere in the current directory, which we then include? i've been tripped up by this quite a few times.

[plucky ermine]

@deft rain you may be familiar with this because its a hugo project (not using hugo module though, mostly for my own learning purposes)

This is the function that builds hugo

// build hugo site and return directory
func (m *Ci) hugo(ctx context.Context, ci *Ci) *Directory {
    // get ref to local project
    path := "/src"
    src := ci.Dir
    outPath := "/src/public"

    // get openring snippet
    // snippet := m.openring(ctx, ci)

    return m.Ctr.
        WithMountedDirectory(path, src).
        WithWorkdir(path).
        // WithMountedFile("layouts/partials/webring.html", snippet).
        WithExec([]string{"hugo"}).
        Directory(outPath)
}

After adding a new markdown file to the hugo content directory the call to run hugo was still cached, which is not expected.

[dense canyon]

can't seem to get a repro now. don't consider this as a blocker for the release. I'll sync with Lev to double check what's happening in his example

[void widget]

Quick question to all Module developers

As I am hacking around the automatic example generation of modules, for shell and compatible SDKs (tests / POC), I wonder what you expect in below cases.

Given the module structure of this package: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/bats@329116361218e383992277d7a3ca5d1acb1fc163

type Bats
    container: Container
    withSource(src: Directory!): WithSource
    run(args: [String]!, source: Directory): Container
type WithSource
    container: Container
    run(args: [String]!): Container

With the dagger CLI, it is expected to have a core type as return type for the command to be considered as "complete", and for dagger call to run the command.

In this case, we have 2 defined types:

• Bats (the module)
• WithSource

The commands available at the cli levels are the functions available at the module type.

If I want to show an example of one the withSource type functions, it makes sense to show one of bats functions leading to a WithSource type:

dagger call -m "github.com/sagikazarmark/daggerverse/bats@329116361218e383992277d7a3ca5d1acb1fc163" with-source --src string container

But, if we were to show its example in code, would people prefer this construct or directly:

dag.
    WithSource(). // withSource type
    Run(args: []string{...})

or

dag.Bats().
    withSource(src: "string"). // bats type, but function leading to withSource type
    Run(args: []string{...})

I have a feeling that the second one is "safer" in a sense that a well designed module should lead to that progression. But it also seems to impose a style ?

Cc @thorn moat or @rocky harbor if you have an opinion

[ember walrus]

👋 Problems using the TypeScript Quickstart/SDK

I'm following the TypeScript quickstart and am stuck on a fundamental step 😅
Engine version: 0.9.5

https://docs.dagger.io/zenith/developer/typescript/5063016/quickstart#step-3-use-input-parameters-and-return-types

My potatoes are always mashed 🥔

I run:
dagger call hello-world --count 10
Note that I did not specify the optional --mashed parameter, which should be false by default

The output:
Hello world, I have mashed 10 potatoes

Has the means to specify optional parameters changed?

cc @upbeat herald for tomorrow

[rocky harbor]

__Quick question to all Module

[wintry prism]

👋

[rare nimbus]

okay I found out what's happening:

• first, you only have the issue if you run "go mod init" before "dagger mod init", and if you have go > 1.21.3 locally. because "go mod init" will create a go.mod with your local version of go (say 1.21.5), but when running "dagger mod init", it will run in a container with go 1.21.3 in local mode, so it won't be able to do anything

• fix 1: don't run "go mod init", because "dagger mod init" will take care of creating the go.mod file anyway if it doesn't exist, and it will have the "right" go version (the one used by the dagger container)

• real fix: ensure the container is properly configured, with GOTOOLCHAIN=auto. by default, the official go images have GOTOOLCHAIN=local, which is ok because you want a controlled env. but in dagger context, I believe it makes sense to be able to run the code if even it requires a newer version of go. just adding the GOTOOLCHAIN=auto env to the container will do it:

• echo $(docker run --rm -it golang:1.21.3 go env GOTOOLCHAIN) => local

• echo $(docker run --rm -it -e GOTOOLCHAIN=auto golang:1.21.3 go env GOTOOLCHAIN) => auto

(I also wrote the same thing in the github issue https://github.com/dagger/dagger/issues/6329)

[restive shore]

Nice find! On thing to note is if you are using go workspaces, the go.work file's go version also matters. So you have to manage that when you change module versions. Right now I have to set it to go 1.21 along with all modules that are referred in the go.work file.

[rare nimbus]

you shouldn't change it back to 1.21 - because go will keep updating it to the latest version declared by your deps.

[restive shore]

But the way modules work today I don't have an option right? I can set everything to 1.21.3 but go work will still fail saying I'm running 1.21.5

[rare nimbus]

if you put 1.21.3 everywhere you should be good - unless you have a dep that requires >= 1.21.4

[restive shore]

Hmm, last I tried it didn't work. I'll give it another shot. Everything is working fine with 1.21 though (until i dagger mod init a new module)

[rocky harbor]

@here chatting about modules in the dev audio channel for the next hour for those interested 🙂

[wintry prism]

@fallow tusk Love your GitHub CLI module idea. Let us know here if you hit any snags. Lots of examples of modules on https://daggerverse.dev

[plucky ermine]

I am seeing some very strange behavior in latest dagger, I am trying to pass a secret as an env var (which was working fine before). Notice how its printing the value of the env var in the console.

You can reproduce with the following function

func (m *GetSecret) Echo(token *Secret) *Container {
    return dag.Container().From("alpine:latest").WithExec([]string{"echo", *token.plaintext})
}

export TOKEN="super-secret"
dagger call echo --token=$TOKEN

...
✘ load call ERROR [2.97s]
├ [0.07s] loading module
├ [2.90s] loading objects
├ [0.00s] traversing arguments
┃ Error: failed to get value for argument "token": secret env var not found "sup
┃ -secret"                                                                      
┃ Run 'dagger call echo --help' for usage.                                      
• Cloud URL: https://dagger.cloud/runs/e3f1ca71-7271-42f1-b4d7-96a6f38ae12a
• Engine: f85835699a1d (version v0.9.6)
⧗ 4.29s ✔ 71 ∅ 7 ✘ 1

[sharp zealot]

@plucky ermine there was a breaking change, secrets are now referenced by name instead of value.

Try dagger call echo --token=TOKEN (note the lack of $)

[sharp zealot]

Or, more explicit: dagger call echo --token=env:TOKEN

[sharp zealot]

That makes me realize, this is a risk of credential leak

Quick PR to fix it: https://github.com/dagger/dagger/pull/6414

[plucky ermine]

Thanks @sharp zealot I must have missed this. Thanks for the quick fix and the info.

[wintry prism]

Trying to understand this one...with my infisical module with Dagger 0.9.6, I continue to run

dagger call -m github.com/jpadams/daggerverse/infisical test --token `cat ~/infisical_service_token`

and it works fine.

When I put the token value into an env var, I can't get it to work

INF=`cat ~/infisical_service_token`
echo -n $INF
<my token value>

dagger call -m github.com/jpadams/daggerverse/infisical test --token env:INF
>>>>error from infisical client, I guess<<<<<
Function execution error: The token is not in correct format!


dagger call -m github.com/jpadams/daggerverse/infisical test --token env:$INF
>>>>error and token value printed<<<<<
Function execution error: The token is not in correct format!

[plucky ermine]

@wintry prism one thing I see is your input type is "string" (I know its python) but I wonder if this only applies when the type is a dagger.Secret?

async def test(token: str) -> str:

[wintry prism]

Lol. Forgot I was using an insecure string there :

Good catch.

Seeing this in my module, though, which makes me wonder what behavior will be. Going to try properly this time.

grep -r 0.9.5 *
src/dagger/_engine/_version.py:CLI_VERSION = "0.9.5"

even though I'm on 0.9.6 and have run dagger mod sync

might just be the sdk dir and I can delete and get a new one

hmmm removed sdk/ and did dagger mod sync and go back same thing 🤔

dagger call -m github.com/jpadams/daggerverse/infisical get-secret --name="DATABASE_URL" --env="dev" --path="/" --token=env:INF

✘ load call ERROR [1.92s]
├ [1.15s] loading module
├ [0.77s] loading objects
├ [0.00s] traversing arguments
┃ Error: failed to get value for argument "token": secret env var not found "INF"
┃ Run 'dagger call get-secret --help' for usage.
• Engine: 0a79be6d7d6c (version v0.9.6)
⧗ 3.18s ✔ 91 ∅ 22 ✘ 1

INF exists with my token inside

Contents of sdk/src/dagger/_engine/_version.py

# Code generated by dagger. DO NOT EDIT.

CLI_VERSION = "0.9.5"

[plucky ermine]

This might be a python issue? My thing works now in go

[wintry prism]

Yeah, I see this
https://github.com/dagger/dagger/blob/main/sdk/python/src/dagger/_engine/_version.py

but when create a new module with python and dagger 0.9.6, I get
CLI_VERSION = "0.9.5"

in that file

[plucky ermine]

Seems like a bug, I can reproduce it on my end to when I make a new module using 0.9.6

[wintry prism]

yeah, same in a linux container

with Typescript sdk too

• Engine: d6d9d44b129f (version v0.9.6)
⧗ 29.75s ✔ 138 ∅ 5
/usr/local/inf # grep -r 0.9 *
sdk/provisioning/default.ts:export const CLI_VERSION = "0.9.5"

opening issue

https://github.com/dagger/dagger/issues/6415

[sharp zealot]

Unrelated to your issue but: you should also be able to do this:

dagger call test --token file:$HOME/infisical_service_token

[latent trellis]

Is there a way to preserve the file name passed to an argument in dagger call?

Sometimes tools rely on the file type. Currently I'm passing in the original file name as a separate argument. :/

I also considered trying to detect the format from the content, but that's equally meh.

[rocky harbor]

Not possible atm, but the engine internally knows the name of the original file, it's just not exposed in our API. We could very easily add a File.name field to the API that would support this. If you don't mind opening an issue, would appreciate it; we can add that to the short-term backlog

[latent trellis]

Thanks!

Opened an issue here: https://github.com/dagger/dagger/issues/6416

[latent trellis]

It pains me to say, but separate GHA workflows with GitHub's own caching currently outperform a single Dagger pipeline: https://github.com/openmeterio/openmeter/pull/525

Individual steps are around 1-2 minutes, whereas the single Dagger pipeline takes 5 minutes. (I know it doesn't seem like a fair comparison, but the Dagger job runs on a larger node, so it should have the resources to run everything in parallel)

Taking a closer look at the relevant run (https://dagger.cloud/openmeter/runs/99fb075a-62e3-406d-a4a7-4260b8ea6fa2), it looks like the cache volume download is about 1,5 minutes. Stopping the engine also takes another 1,5 minutes.

Not sure if I'm doing something wrong (some of the modules are fairly complicated now), but the cloud UI is certainly not making my job easier in figuring out what's going on. :/

[dense canyon]

ci: update CI modules by sagikazarmark ·...

[normal blaze]

Hey folks, has anyone built a module for creating debian and rpm packages?

I checked the daggerverse and did not see anything, if not I will attempt this in the morning

[sharp zealot]

What a great idea. I don’t know of anyone having tried that.

[normal blaze]

Sweet, I started to look at this, so my approach will probably be: prepare the files. Start a debian container, mount the files. Execute dpkg. Copy artifacts.

It looks pretty straightforward, I will support multi arch too

[latent trellis]

I used nfpm in the past: https://github.com/goreleaser/nfpm

[normal blaze]

oh interesting, that might make it easier as I want to create rpm's too

I am currently using goreleaser to create the packages, I want to migrate my whole pipeline to dagger. Removing goreleaser

[latent trellis]

I'm working on the same. I don't build debs though. I'm working on the github release part right now.

[normal blaze]

That is awesome, I need that too

[latent trellis]

It would be cool to create a drop-in replacement module for goreleaser 😄

[normal blaze]

A module can wrap modules right?

[latent trellis]

yep

[normal blaze]

That would be nice then

[latent trellis]

it's possible, but it might be more work than it's worth.

I just reuse parts of my build pipeline at the moment

[normal blaze]

I need to notorize my binaries too for osx I see there is a unix based tool that does not depend on the osx toolchain

This is also in my eventual goals

[latent trellis]

Cool!

[sharp zealot]

@normal blaze is this demo relevant to the "notarize my binaries for osx" part? https://discord.com/channels/707636530424053791/1187279114265182249

[dense canyon]

👋 I've noticed that the dagger functions output it v0.9.6 is way more verbose compared to the previous version. Just checking if this is intentional? Since the new output contains a lot of unnecessary information and it's quite hard to follow. cc @kind carbon @rocky harbor

[normal blaze]

It looks like it yes, very useful

[kind carbon]

That was because of the PR to support chaining core types but https://github.com/dagger/dagger/pull/6317 fixes that. There's a new PR coming to clean --help as well.

[sharp zealot]

Speaking of which, it’s ready to review 😇

[upbeat herald]

I'm currently trying to undertand how interface works so I can add its support in the TypeScript SDK
However, I tried a simple use case and it seems it's not working with Go: https://github.com/TomChv/playground/tree/main/play-with-dagger/dagger-interfaces

I put every information I have in the README so anyone can reproduce my issue.
I think the errors happens with the object is converted into the interface

Would be super nice if someone could help me unblock that, once I get something working, I can try to have a something with TypeScript
/cc @rocky harbor @thorn moat

[thorn moat]

playground/play-with-dagger/dagger-inter...

[wintry prism]

Strange, at some point flippng between recent dagger versions and running dagger mod sync, I got this in my dagger.json:

[thorn moat]

that field has been gone for quite a while

removed Oct 17

[sharp zealot]

@lime comet @dense canyon I LOVE that webhook project and was just planning to hack on something similar. How do I use it? 😁

do you have an example somewhere?

[wintry prism]

Related to above (#daggernauts message) can I call a function from the dagger CLI that takes a dagger.Secret as an argument?

Do I need to create a secret with id first and pass that?

I tried to pass a string and have it autoconvert to a dagger.Secret, but get errors from the library I'm using saying the secret is in the wrong format.

105: [1.22s] ╭─ Error ──────────────────────────────────────────────────────────────────────╮
105: [1.22s] │ Function execution error: The token is not in correct format!                │
105: [1.22s] ╰──────────────────────────────────────────────────────────────────────────────╯
105: exec /runtime ERROR: process "/runtime" did not complete successfully: exit code: 1

100: dagger call get-secret-plaintext
100: [1.36s] Error: response from query: input:1: infisical.getSecretPlaintext failed to get function output directory: process "/runtime" did not complete successfully: exit code: 1
100: [1.36s]
100: dagger call get-secret-plaintext ERROR: response from query: input:1: infisical.getSecretPlaintext failed to get function output directory: process "/runtime" did not complete successfully: exit code: 1

infisical ➤ dagger --progress=plain call get-secret-plaintext --name="DATABASE_URL" --env="dev" --path='/' --token=file:./infisical_service_token

[sharp zealot]

yes, from env or file

your use looks correct

[wintry prism]

when I run like this

dagger --progress=plain call get-secret-plaintext --name="DATABASE_URL" --env="dev" --path='/' --token=env:INF

with my token in INF then I get

3: [1.10s] Error: failed to get value for argument "token": secret env var not found "INF"
3: [1.10s] Run 'dagger call get-secret-plaintext --help' for usage.
3: load call ERROR: failed to get value for argument "token": secret env var not found "INF"

this is Python SDK

dagger 0.9.6

[sharp zealot]

is INF actually, definitely set?

[wintry prism]

infisical ➤ echo $INF                    
st.653b00b.............

yep. correct, same contents as the file infisical_service_token

[sharp zealot]

double checking, is it exported in your shell?

your use seems correct so I’m going down the list of possible mistakes

[wintry prism]

ooh, good point, might have lost that. Resetting.

[sharp zealot]

you can try prepending your dagger command with INF=foo

[wintry prism]

yep.

I did an export set this time. Looks like I failed to originally.

New error!!

105: exec /runtime
105: [2.98s] ╭─ Error ──────────────────────────────────────────────────────────────────────╮
105: [2.98s] │ Failed to serialize result: Object of type SecretBundle is not JSON          │
105: [2.98s] │ serializable                                                                 │
105: [2.98s] ╰──────────────────────────────────────────────────────────────────────────────╯
105: [2.98s] st.653........
105: exec /runtime ERROR: process "/runtime" did not complete successfully: exit code: 1

100: dagger call get-secret-plaintext
100: [3.15s] Error: response from query: input:1: infisical.getSecretPlaintext failed to get function output directory: process "/runtime" did not complete successfully: exit code: 1
100: [3.15s]
100: dagger call get-secret-plaintext ERROR: response from query: input:1: infisical.getSecretPlaintext failed to get function output directory: process "/runtime" did not complete successfully: exit code: 1

and.....I exposed my token!! (op 105) used to be the full token

will rotate that, even though it's only sample data.

[lime comet]

We broke it a few minutes ago . I'm gonna work on it tonight hopefully have it running for tomorrow with a short & sweet README that has some examples! What we want is for devs to just run dagger up -m github.com/franela/webhook and connect the webhooks of their repositories to it. The contents of the "pipeline" should be contained within the repo, in a hooks.json file (similar to what you typically have with github workflows)

[wintry prism]

New error!!

[sharp zealot]

webhooks example

What values can I pass to a Service argument from the CLI?

[rocky harbor]

tcp://<endpoint>, so e.g. tcp://localhost:1234

udp is technically handled but I'm not sure services v2 tunneling works with udp yet so I think just tcp atm

[thorn moat]

yeah udp isn't supported for tunneling at the moment

[sharp zealot]

Ok thanks

Would be awesome to add support for oci://index.docker.io/nginx or something like that

[rocky harbor]

this is where the args for services are handled, oci would be slightly more complicated, but seems doable https://github.com/sipsma/dagger/blob/ce44608a87b72152b1d4c1b2d042417299d2188d/cmd/dagger/flags.go#L347-L347

[sharp zealot]

I'm having trouble getting the tcp://localhost:1234 form to work

With nc -l 4242 running in another terminal:

$ dagger call -m github.com/shykes/daggerverse/tailscale diagnostics --backend tcp://localhost:4242
✘ dagger call diagnostics ERROR [0.14s]
┃ Error: response from query: input: resolve: tailscale: diagnostics: call function "Diagnostics"
┃ : process "/runtime" did not complete successfully: exit code: 2                               
┃ input: resolve: tailscale: diagnostics: call function "Diagnostics": process "/runtime" did not
┃  complete successfully: exit code: 2                                                           
┃                                                                                                
┃ Stdout:                                                                                        
┃ input: resolve: loadServiceFromID: ports: service qpt2dim1bto12 is not running                 
┃ input: resolve: loadServiceFromID: ports: service qpt2dim1bto12 is not running                 
┃ Stderr:                                                                                        
✘ diagnostics ERROR [0.13s]
✘ ports ERROR [0.00s]
▶ diagnostics
  ✘ exec /runtime ERROR [0.12s]
  ┃ input: resolve: loadServiceFromID: ports: service qpt2dim1bto12 is not running               
  ┃ input: resolve: loadServiceFromID: ports: service qpt2dim1bto12 is not running               
• Cloud URL: https://dagger.cloud/runs/66e722ae-d3ab-4f4e-baef-fbe77871038b
• Engine: f2394d01a587 (version devel ())
⧗ 4.97s ✔ 93 ∅ 4 ✘ 4

[sharp zealot]

I'm running into a best practice question while developing my tailscale module: specifically how to cleanly run multiple daemons in the same container

[sharp zealot]

My module has a function that returns []PortForward (a core type), but it's failing with this error:

$ dagger functions
✘ load functions ERROR [1.36s]
├ [1.35s] loading module
├ [0.00s] loading objects
┃ Error: query module objects: json: error calling MarshalJSON for type *dagger.Module: retur
┃ d error 400 Bad Request: failed to get schema for module "tailscale": failed to create func
┃ on "portForwards": failed to find mod type for function "portForwards" return type: "[Tails
┃ lePortForward!]!"                                                                          
• Engine: f2394d01a587 (version devel ())
⧗ 1.81s ✔ 102 ∅ 3 ✘ 1

[deft rain]

This is caused because some core types aren't IDable in v0.9.6 - in v0.9.7 later this week with @thorn moat's work to integrate our custom graphql server (https://github.com/dagger/dagger/pull/6297), then we shouldn't have this anymore!

[lime comet]

We're writing a module that starts a service inside one of the functions and has to make requests to it and we stumbled into something that works differently with modules than with the SDK. Quick example module:

func (m *Stop) RunSvc(ctx context.Context, server *File) error {
  svc := dag.Container().
    From("golang:1.21").
    WithFile("/server/main.go", server).
    WithExposedPort(8080).
    WithEntrypoint([]string{"go", "run", "/server/main.go"}).
    AsService()
  tunnel, err := dag.Host().Tunnel(svc).Start(ctx)
  if err != nil {
    return err
  }
  defer tunnel.Stop(ctx)

  endpoint, err := tunnel.Endpoint(ctx)
  if err != nil {
    return err
  }

  res, err := http.Get("http://" + endpoint + "/hello")
  if err != nil {
    return err
  }
  defer res.Body.Close()
  io.Copy(os.Stdout, res.Body)
  return nil
}

The problem we're facing has to do with tunnel.Stop(ctx). When I run this code outside of a module using Dagger's SDK it works fine, it can stop the tunnel and move on. However, when I run it inside a module function call it hangs in tunnel.Stop . The workaround I found (although maybe this is the expected behavior, but it's different to using the native sdk) is to first stop the Service that is being tunneled and then stop the tunnel, that works as expected:

func (m *Stop) RunSvc(ctx context.Context, server *File) error {
  svc := dag.Container().
    From("golang:1.21").
    WithFile("/server/main.go", server).
    WithExposedPort(8080).
    WithEntrypoint([]string{"go", "run", "/server/main.go"}).
    AsService()
  tunnel, err := dag.Host().Tunnel(svc).Start(ctx)
  if err != nil {
    return err
  }
  defer func() {
    svc.Stop(ctx) // <-------- adding this makes it so that `tunnel.Stop` can run successfuly
    tunnel.Stop(ctx)
  }()
  // ...
}

Can somebody else reproduce it? If so, is it the expected behavior or should calling tunnel.Stop stop the service it's tunneling without the additional svc.Stop (like how Start works)

[thorn moat]

Service lifecycle with tunnels

[dense canyon]

on a similar context around this issue we've also found that when performing dagger up and then eventually pressing ctrl+c, the underlying service container will never be stopped (even after the pipeline finishes) which causes future dagger up calls to stop working altogether. cc @thorn moat is that something that has been reported?

[normal blaze]

Hi, just trying to build out my first module, I am getting a bit of an error...

I apologise but I don't know where to begin for debugging right now

https://github.com/nicholasjackson/daggerverse/blob/main/main.go

Just lookiung through the docs, I had not called dagger mod sync after updating the interface, but I am getting a different error

let me try something

Tried creating a new module and added my code, when I run sync I get

✘ generating go module: deb ERROR [0.46s]
├ [0.10s] go mod tidy
┃ writing dagger.gen.go                                                                                                                                                                                                                                                                                                                                                     
┃ writing go.mod                                                                                                                                                                                                                                                                                                                                                            
┃ writing go.sum                                                                                                                                                                                                                                                                                                                                                            
┃ needs another pass...                                                                                                                                                                                                                                                                                                                                                     
• Engine: 863ba20810d5 (version v0.9.6)
⧗ 5.10s ✔ 18 ∅ 2 ✘ 3
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule.generatedCode failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

I have found the problem, this is a bit of an odd one, but, if you have an argument Package you get an error, guessing that internally you convert this to package which is go reserved.

I was brain dead copy pasteing with my function args, no idea why I was using upper case, if I had used lower case the compiler would have caught the package thing. But interesting edge for you

[deft rain]

ahhh, yeah, this looks a lot like https://github.com/dagger/dagger/issues/6186

[normal blaze]

Another question, inside my modules, if I create a file using a go construct ...

    tmpDir, err := os.CreateTemp("", "dagger")
    if err != nil {
        return "", err
    }

    defer os.RemoveAll(tmpDir.Name())

    controlFile := path.Join(tmpDir.Name(), "control")
    os.WriteFile(controlFile, []byte(controlContents), 0644)

How would I add that file to a container, this does not seem to work...

        WithMountedFile("/etc/DEBIAN/control", dag.Host().File(controlFile)).

Guessing Host() is my local computers disk, not the temp directory defined in the go code as this would be executed in a docker container by dagger

I am guessing for modules when working with temporary files you need to use something like wd := dag.Directory().WithNewFile("control", controlContents)

[normal blaze]

Ok so I am mostly working now, but

My module returns a File ) (*File, error) { when I use dagger call I get the contents back of the file but I am not sure that it is binary

if I try to pipe this into a file from the command I do not get a valid file

➜ dagger call build --binaries $PWD/example_app/bin/example_app --architecture amd64 --package-name "example" --version "0.0.1" --maintainer "Nic Jackson" --description "blah" > example.deb  

I am guessing this is because Contents on File returns string not []byte

[rocky harbor]

This is caused because some core types

[normal blaze]

TY

[warped canyon]

Try dagger download build --binaries $PWD/example_app/bin/example_app --architecture amd64 --package-name "example" --version "0.0.1" --maintainer "Nic Jackson" --description "blah"

related: https://github.com/dagger/dagger/issues/6229

[normal blaze]

TY, I will give this a go after dinner, so far the experience has been rather plesant

[normal blaze]

This works great my next question tho is how to pass a function a list of binaries, I am guessing the correct pattern would not be to use a single function and use a builder pattern. Soemething like ...

func (m *Deb) WithFile(path string, file *File) *Deb
func (m *Deb) Build(ctx context.Context, package string) *File

So the call would then be
d := &Deb{}
d = d.WithFile("/bin/example", example).
 WithFile("/etc/blah",blah).
 Build(ctx, "example",...)

To test something like that how would you use call or download?

[warped canyon]

That would be call, although the Deb.Build would probably return a *File, right? I would also potentially pass in a *Directory instead of a series of individual files, since building that directory looks pretty similar and it's already core. But whatever feels like a more natural DX for how you're picturing the usage

[normal blaze]

Maybe yes, I have updated the example, you could just construct the directory with the files to put into the deb archive when calling the module and then just pass the directory. I could then actualyl just add the DEBIAN/control file to that directory and build. Much easier

TY bud

Job for tomorrow

[warped canyon]

Looking forward to seeing more! Is this basically generating the control and then using ar to archive?

[normal blaze]

Yeah, dpkg-deb, I think by setting the environment for the container I can specify say an arm64 archive or an amd64 and just pass the right binaries

Very easy to construct

I am loving this

[warped canyon]

Glad to hear it 😄

[normal blaze]

There are a few conventions to get over but once you figure that out it is an incredibly fast process. This is step 1 to migrating Jumppad to Dagger.

[warped canyon]

I think the issue I linked with subcommands will be partially (or completely?) addressed with 0.9.7. Does that sound right @rocky harbor ?

[rocky harbor]

Yeah the basic support is actually already in v0.9.6, if you have a function my-file that returns a File you can do e.g. dagger call my-file contents to get the file contents written to stdout

[restive shore]

In modules how do I ensure the From image is always pulled? Equivalent of the docker --pull? Right now my publish module seems to be using a cached version

[sharp zealot]

Unless you specify a digest, the engine will always check for the latest version, and pull it. If you're getting a cache hit, it's because that's still the latest version at the repo+tag you specified

[restive shore]

That is what I thought. Thanks for confirming. Not what I'm seeing. And looking further, it doesn't look like it's a dagger issue. renovate/renovate:latest is the image I am downloading. The SHA I'm getting is not the same as listed here - https://hub.docker.com/layers/renovate/renovate/latest/images/sha256-ef0eb13bedffb8ca881b48b2807d81c7818e6357d7f79d3fc2458e7fbcd3d0c1?context=explore.

I tried pulling it outside of dagger. Same result. Weird. Some kind of issue with docker.io maybe?

[sharp zealot]

Mmm... How do you check the digest, with Dagger, and without?

[restive shore]

When docker pulls it, it shows.

in dagger if I do --focus=false it shows the digest of the resolved image.

[upbeat herald]

Hey 😄
another PR that fixes a lot of things on the TS SDK: https://github.com/dagger/dagger/pull/6433
I would love a review on it so I can merge it tomorrow

PS: https://github.com/dagger/dagger/pull/6423 is still waiting for a review too 😄
I'll add more PRs tomorrow, the faster we got these merge, the easier will be the 0.9.7 Thursday release ^^

[wintry prism]

reviewing : https://github.com/dagger/dagger/pull/6423 now

[wintry prism]

@upbeat herald do you need a rebase? 👆

[upbeat herald]

I can rebase tomorrow, I’ll check on that issue, I have never seen that one during my test

Did you correctly cleaned your engine and stuff?

Because it might come from that

Try remove: images, volumes, container et retry

Because it seems it’s an error that comes before the execution of the TS module inside the engine

[wintry prism]

Yeah, the ./hack/dev removed my old engines, but must not clean up a lot.

That worked!

[upbeat herald]

Yeah, you keep volumes and source image, I usually clean everything before trying

Amazing!

Feel free to update your review 😉

[sharp zealot]

I’m looking for a linux process/init manager that can be easily instrumented by code 😇 Anybody know one?

[wintry prism]

My cleanup

docker stop $(docker ps -a -q -f 'name=dagger-engine')
docker rm $(docker ps -a -q -f 'name=dagger-engine')
docker system prune -f -a --volumes

[upbeat herald]

I usually delete the images too, just in case

(Not ideal if you have a slow network)

[wintry prism]

Good to add to the contrib / dev guide if not already there

I’m looking for a linux process/init

[wintry prism]

Why does the Go SDK mention dagger mod sync steps explicitly in the Quickstart, Python mentions it once at the top, and Typescript doesn't mention it at all?
https://docs.dagger.io/zenith/developer/go/525021/quickstart

Also, seems it's not needed with Typescript SDK perhaps?

[wintry prism]

Yep git shows no changes to my repo if I dump new code in my src/index.ts, commit it, and then run dagger mod sync with the Typescript SDK.

[kind carbon]

None of them need it. It’s only for the IDE.

[kind carbon]

It’s in the FAQ.

[latent trellis]

What do people generally return in modules?

A *Container or the result of the Stdout call?

I usually just return the container, but I noticed it also generates a huge help text for dagger call --help.

Returning the container may be useful for debugging, but it's not really useful otherwise when I'm simply curious about the output of a command.

[kind carbon]

I prefer returning container but there’s a couple of changes coming here, both in help cleanup but also on what happens when you stop chaining at container. Rather than rely on the implicit stdout you’ll need to explicitly append stdout to your command.

[latent trellis]

Where do I report potential security issues? 🙂

I'd rather not do it on the issue tracker

[latent trellis]

@normal blaze GH release module: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/gh@ef6b5bf932277ffc4b7e1ea9fd9a1c26a2780da3#Release.create

Example: https://github.com/openmeterio/benthos-openmeter/pull/31/commits/befb2cb50bbc051d52980c55cfd8620d58317a9d

[kind carbon]

You can DM someone in the team and we'll report internally 🙏

[latent trellis]

Discussed it in private. There is already a fix in a PR.

[upbeat herald]

@deft rain Is there any issue with the CI? I keep having this error:
failed to compute cache key: failed to get state for index 0 on copy /runc /usr/local/bin/runc

/cc @strong ingot (I forget to ping you too)

Seems like its fixed after a retry

[deft rain]

enjoy 🙂 https://github.com/dagger/dagger/issues/6111

(this is on the list of things i'm interested in investigating at some point)

[normal blaze]

Awesome ty, I am just testing my deb module in my build pipeline

https://daggerverse.dev/mod/github.com/nicholasjackson/daggerverse/deb@b2243239e1418ebb14cae682708cc7d7f507e3f7

I need your module for my pipeline too, thanks for creating this

[latent trellis]

I'm thinking about creating a higher level GitHub release module that could update release assets on reruns, push container images and helm charts, etc with a friendly API, but this should do for now.

[normal blaze]

Nice, do you use artifacts at all?

[latent trellis]

Do you mean GHA build artifacts?

[normal blaze]

yes

[latent trellis]

Not at the moment, but I push container image snapshots, so it would be nice to upload snapshots of binaries and helm charts as artifacts as well.

I have CI function that builds those on each PR and main push anyway

[normal blaze]

My pipeline uses them quite a bit, I was thinking about building a module. I could not decide if this is a separate module or if I should PR you

[latent trellis]

Well, my module right now is just a wrapper around the gh CLI tool. I looked at its code and it was easier to just wrap it than rewriting all that logic in a Dagger module.

I don't know if the gh CLI tool can upload assets in a GHA workflow.

If it can, sure, send a PR!

Otherwise it might make more sense to build a github-actions specific module. Or github actions + release + anything that might be useful in CI.

[normal blaze]

I will take a look in the morning, just hada quick look at the existing gh action. Seems it uses the typescript module for gh

[normal blaze]

Apologies, I am being a bit thick, how do I reference my deb module from my dagger go code?

[latent trellis]

1. dagger mod install
2. restart LSP
3. dag.Deb()

[normal blaze]

ah, thanks, I was not sure if dagger mod was just for modules, I created my go code just by adding the dagger package

[latent trellis]

Yeah, zenith takes some time to get used to. You don't actually import your own code. Dagger generates a stub that calls the engine that calls your code 😄

[normal blaze]

Things have moved fast

[latent trellis]

That's how you can use modules across languages

[normal blaze]

using dagger to run the build also answers my question around logging 🙂

[wintry prism]

It’s in the FAQ.

[normal blaze]

Ok so when I am using modules my main module, everything has to be a module and I need to follow the same conventions. I can't mix and match the old way and zenith

[warped canyon]

not yet 🙂

[bleak nest]

Thanks, I'll make this point clearer in the quickstart.

[normal blaze]

All good, I was confusing myself by looking at two different docs

[normal blaze]

Folks, this experience is really very good, once you get your head round a few things, I liked dagger before, I love it now

One more question, how do I get my logs to output in the dagger ... cli output

[warped canyon]

if you mean you don't want all the steps to be collapsed in the tui output, the flag --focus false is what you're looking for

[normal blaze]

No I would like to define my own custom steps in the log output e.g.

Building arm64
         amd64
Packaging arm64
Running tests
  TestABC Passed
  TestCDE Passed

[warped canyon]

Ahh like the Pipeline() heirarchy from pre-zenith. I'm actually not sure what the current thinking is there @rocky harbor

[normal blaze]

Yes, it is essential to see that output really

Especially when I get a failed build

But minor problems, really looking forward to replacing this build

[warped canyon]

Yeah all the stdout/stderr should be visible for all steps with --focus false, but I don't think there's a way to customize labels/names if that matters

[normal blaze]

I think having a standard Dagger way to log would be really helpful

[rocky harbor]

Yeah right now you get one logger+log-level: write to stdout+stderr 😅 The idea of a logging api is where we landed last time we discussed this too. I'll open an issue to track/discuss, it makes perfect sense I think.

[normal blaze]

TY, I could invent something but I think it is really nice if everyone follows the same convention. That way the output looks the same as the output from your modules

Having clear sections will be really helpful too, when you use something like GitHub actions for a large build this breaks the output across multiple log outputs. If I run my pipeline with dagger I potentially have one. This is a huge difference to groking log output

[rocky harbor]

https://github.com/dagger/dagger/issues/6442

[normal blaze]

TY

[sharp zealot]

I have strong opinions on this that have been brewing for a while, I will write them down in that issue. FYI @warped canyon @rocky harbor @normal blaze

[sharp zealot]

Done: https://github.com/dagger/dagger/issues/6442#issuecomment-1896811240

[warped canyon]

dagger functions with v0.9.6 is.. something 😂 has this been noted already?

[dense canyon]

Yep, has been raised already

[rocky harbor]

Tracked here specifically https://github.com/dagger/dagger/issues/6358

[warped canyon]

related to namespacing, haven't run into this one before: https://github.com/dagger/dagger/issues/6438
you can't mod install a module with the same name as your current module

[plucky ermine]

Is it possible to dagger shell into a container that is already running as a service with dagger up?

[sharp zealot]

I don't think so

[plucky ermine]

one day

I used dagger up to bring up mariadb for super easy and consistent local and ci dev and I really love the workflow ❤️

I first used @wintry prism module but it was too opinionated :p

So I whipped up a local module inside my ci/mariadb directory that looks like this

// example usage: dagger up --port 3306:3306 serve
func (m *Mariadb) Serve() *Service {
    return dag.Container().
        From("mariadb:latest").
        WithEnvVariable("MARIADB_ALLOW_EMPTY_ROOT_PASSWORD", "1").
        WithExposedPort(3306).
        AsService()
}

Thank you @wintry prism for the inspiration 🙂

[wintry prism]

Daggerverse, you're one-stop shop for:

• Ready to use, generic modules
• Specific implementation examples/experiments
• Inspiration

[sharp zealot]

While playing with my tailscale module, I find myself really really wanting one-liner scripting

[plucky ermine]

@sharp zealot tell me more about what you mean?

[sharp zealot]

that module doesn’t lend itself to calling from the CLI

[wintry prism]

Like the python ones we demoed?

[sharp zealot]

so I have to create a full blown test module just to tinker with it

yeah python go or ts, I’m not picky 😛

would be a game changer for the module discovery experience

[plucky ermine]

I am not sure I follow, how would you imagine the DX working?

I have been using justfile for all my stuff lately and I would love to see a DX that looks basically just like that with dagger, but not sure if you have somethign else in mind.

For example:

# serve site on localhost:4000
serve:
    dagger up --focus=false -m ci --port 4000:4000 serve --dir=.

# run ci with empty commit 
empty: 
    git commit --allow-empty -m 'trigger ci' && git push

# push to gitlab
push: 
    git push -u gitlab main && git push -u origin main

# start dev database 
db:
    dagger up -m ci/mariadb --port 3306:3306 serve

Are you talking about something else when you say one-line scripting? -- this is what I had in mind

Daggerverse, you're one-stop shop for:

[sharp zealot]

dagger script --go ‘var backend = dag.Container().From(“nginx”); ts := dag.Tailscale().Gateway(backend, “www”, token)’ -m github.com/shykes/daggerverse/tailscale

[plucky ermine]

that motivates me

[sharp zealot]

note: I don't know how to pass arguments to that yet

[plucky ermine]

This immediately led me to want dagger repl -m $MODULE - that might open some sort of environment that is purpose built for tinkering with modules?

We are about 3 more ideas away from emacs I think

[wintry prism]

https://github.com/dagger/dagger/issues/5774
https://github.com/dagger/dagger/issues/6229

[sharp zealot]

Yes agree, I think dagger script without argument should just give you a repl. Exact command name is open to bike shedding of course

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/6444

[sharp zealot]

Mmm, it seems that sync to github didn't work 🤔 Re-created the issue in github

[plucky ermine]

Does Host to Container work in Zenith? Specifically trying to do this: https://docs.dagger.io/757394/use-services/#expose-host-services-to-containers

[rocky harbor]

Yes if your function accepts an arg of type Service you can provide a host service to it like --argname tcp://localhost:1234.

v0.9.6 had a bug the meant you needed to explicitly start the service in your module code (see example here: https://github.com/shykes/daggerverse/blob/3d43167457b0de8c8929a9de411480daddddab7f/tailscale/main.go#L22-L26). Fixed in next release though

[plucky ermine]

I am running into this issue that I feel like I have seen before but I cant figure out how to fix it.

I have this module https://daggerverse.dev/mod/github.com/levlaz/daggerverse/mariadb@85f4f9942ba7f694b04caf6dc690fe04144e6d72

I can call it with dagger up -m github.com/levlaz/daggerverse/mariadb --port 3306:3306 serve without any issues

However, when I try to install it for use in another project I get this error:

levlaz@Levs-MacBook-Pro ci % dagger mod install github.com/levlaz/daggerverse/mariadb
✘ generatedCode ERROR [3.04s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [1.65s]
┃ Error: load package ".": package name is empty                                
✘ generating go module: mariadb ERROR [1.42s]
├ [1.38s] go mod tidy
┃ writing dagger.gen.go                                                         
┃ writing go.mod                                                                
┃ writing go.sum                                                                
┃ creating directory querybuilder                                               
┃ writing querybuilder/marshal.go                                               
┃ writing querybuilder/querybuilder.go                                          
┃ needs another pass...                                                         
• Cloud URL: https://dagger.cloud/runs/83ec286b-4301-4c86-acf9-83ed07d31c39
• Engine: f85835699a1d (version v0.9.6)
⧗ 6.50s ✔ 29 ∅ 3 ✘ 3
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule.generatedCode failed to get schema introspection json during ci module sdk codegen: failed to get schema for module "mariadb": failed to call module "mariadb" to get functions: failed to get function output directory: process "/usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

[sharp zealot]

@plucky ermine the issue may be with your current directory when running that command... is there a dagger.json in the current directory?

[plucky ermine]

Yeah there is

levlaz@Levs-MacBook-Pro ci % tree 
.
├── dagger.gen.go
├── dagger.json
├── go.mod
├── go.sum
├── main.go
└── querybuilder
    ├── marshal.go
    └── querybuilder.go

With this

{
  "name": "ci",
  "sdk": "go"
}

This is a "unpublished dagger module as main ci pipeline file" situation

Oh I think I know the issue - this is a name collision, my module has a function called Serve() and so does the main ci function

[sharp zealot]

I don't think that conflict should be possible, everything is namespaced by module name

[plucky ermine]

Hm, yeah nevermind I am wrong about that

[rocky harbor]

Can repro, looking into this since it's quite odd that dagger up works but install doesn't, will let you know what I find...

[rocky harbor]

Not that it helps you at this moment, but on a whim I tried repro'ing on my PR that refactors all the module config/loading and it is fixed there... So this will if nothing else be fixed in the release next week 😅 I would suspect there's something going wrong around resolving the git ref during dagger mod install specifically since before that PR there were multiple codepaths for resolving git refs.

Let me see if there's any workarounds in the mean time

[sharp zealot]

love an accidental fix

https://tenor.com/view/shitts-creek-journey-alexis-rose-love-that-journey-for-me-gif-15621263

[rocky harbor]

🎉 🤷‍♂️

[plucky ermine]

Haha, thanks for looking into this - since this is a very trivial module I worked around it by just re-implementing the code snippet inline for now.

[rocky harbor]

Okay cool, yeah I even tried hand editing dagger.json and it still happens if you try to call anything, so probably better to invest time in getting that PR merged than keep looking 😄

[plucky ermine]

Now that I got this working its really awesome to see things coming together.

We are living in a world where its possible to encapsulate your dev environment into a dagger module.

Now when a new engineer joins your project they can spend 0 time fiddling with dependencies and instead

1. install dagger cli
2. clone repo that has a beautiful ci dagger module
3. run dagger up --focus=false -m ci --port 4000:4000 serve --dir=.

[sharp zealot]

technically we can get rid of step 2 😇

The time will come when even installing git on your host machine will be redundant

[sharp zealot]

@rocky harbor sorry I really wanted to join that ongoing conversation about dagger.json, but had to go to another call..

[sharp zealot]

@thorn moat what were the files you mentioned in the real-world monorepo example on the call? Linter rules? Can't remember

[thorn moat]

Translations, like for frontends

iirc there was a translation team that would maintain a bunch of .json files containing translations, and those would be somehow imported into downstream projects from other teams