#daggernauts

1 messages · Page 6 of 1

sharp zealot
#

Over several months of testing these features with the community, we noticed something interesting: Dagger Functions are great for CI, but they can be applied to other problems as well. From test data management, to local AI development, to dev environments, to just-in-time artidfacts, to SaaS integration, to OpenAI development… The world is full of pipelines, just waiting to be Daggerized. We look forward to seeing what you build next!

latent trellis
#

It's so weird it's called Functions and not Zenith. That name kinda grew on me. 😄

sharp zealot
#

Now it will be a cool codename only the old timers remember

prime ruin
#

From someone who didn't look in the guts yet, excited about announcement but curious:
Do I need to install all the runtimes/compilers of all the tools my dagger functions use?

The promise of Functions is to have smooth cross-language functions, but if my python function calls a golang one, do I now need golang compiler on my machine?
That feels wrong, like eww now I'd have to care about every one of my function dependencies' language? So I'd think maybe dagger itself runs each of those runtimes in containers?

#

I initially thought that each module's call chain is turned into the language-independent GraphQL query, hence no compiler/interpreter needed, but then realized that if I use any native function (say in python concatenating an image name + a tag, as string operations), then GraphQL doesn't understand that, so you still need to run the real code.

sharp zealot
prime ruin
#

fantastic =)
Can't wait to start making "CI Libraries but not Jenkins"

plucky ermine
#

Hello, my main module sources are in the

restive shore
sharp zealot
#

We'll make a special edition sweat shirt 😁

restive shore
#

Count me in for one of those 🙂

dense canyon
#

👋 just noticed that dagger init for Go doesn't add the internal or the dagger.gen.go files to .gitignore anymore?

restive shore
#

didn't dagger stop adding any of the codegen to .gitignore? That's been the case in the last few releases afaik

dense canyon
wintry prism
kind carbon
#

That's Go SDK specific. The motivation was that linting module snippets for docs failed in CI without dagger develop first. There's lots of them. @rocky harbor reasoned that there's precedent for committing generated files, like with protobuff, for example. I'm in favor of not committing them though.

dense canyon
wintry prism
kind carbon
wintry prism
honest hearth
#

was there a takeaway on above? Can I safely .gitignore the internal directory in my new go modules?

sharp zealot
restive shore
quick wind
#

Getting Service.up(ports: [{front-end: null, backend: 8080, protocol: TCP}] during dagger call execution despite the frontend port being set (to 8080) in the module. Is this something to do with ports being equal, or unexpected behaviour? The service isn't available at localhost:8080 when the logs say it is, so seems like something isn't right?

dense canyon
dense canyon
#

👋 is there a way to define a default value for a Dagger type? i.e I find myself typing dagger call $my_func --dir . every time to send the current dir to the function. I was wondering if there was a way to tell dagger to use the default CLI working directory as the default dir.

sharp zealot
#

So, now that Project Zenith is complete, we need to decide what to do with this channel. Options are:

  1. Archive this channel
  2. Rename it to #modules-dev : a place to discuss the development of modules
  3. Other?

What do you think?

sharp zealot
rocky harbor
#

I think mounting a directory is

mossy hazel
rocky harbor
#

Chatting in dev-audio about Zenith for the next hour for anyone who wants to join!

honest hearth
#

@kind carbon do you have any WIP on your rye/pip stuff pushed up? Was curious to look at it for some inspration

plucky ermine
subtle plinth
#

How does Dagger establish if a pipeline is cached or needs to run? Does it check this on the host or tar up and send to BuildKit first?

potent bramble
#

this is going to be fun 🙂

sharp zealot
#

@rocky harbor I just remembered why ID being a reserved word was actually a problem for my core wrapper: I need to expose loadFooFromID(id: String!) and that id: String! is forbidden

#

I'm getting this error initializing a new module, using dev dagger:

$ dagger init --sdk go
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.7s
✘ ModuleSource.asModule: Module! 0.8s
  ✘ Module.withSource(
      source: ✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
    ): Module! 0.8s
    ✘ Container.directory(path: "/src"): Directory! 0.8s
      ✘ exec /usr/local/bin/codegen --module-context /src --module-name core --propagate-logs=true --introspection-json-path /schema.json 0.7s
      ┃ Error: exit status 1
      ✘ generating go module: core 0.3s
      ┃ creating directory internal
      ┃ creating directory internal/dagger
      ┃ writing internal/dagger/dagger.gen.go
      ┃ creating directory internal/querybuilder
      ┃ writing internal/querybuilder/marshal.go
      ┃ writing internal/querybuilder/querybuilder.go
      ┃ writing main.go
      ┃ go: open ../go.mod: no such file or directory

Error: failed to generate code: input: moduleSource.withContextDirectory.withName.withSDK.withSourceSubpath.asModule failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name core --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1
sharp zealot
#

Update: it works outside of my daggerverse repo 😭

sharp zealot
#

It seems related to having a go.work. Removing it made the problem go away. Impossible to init any Go module in the repo otherwise

#

Somehow my developing one module interferes with developing others in the same daggerverse repo

#

The go.work seemed fine. No idea what was wrong with it

dense canyon
#

👋 not sure if this is considered an issue and/or it has been reported but is it expected for a constructor argument with a default value to resolve into a required argument for the Go SDK?

i.e:

func New(
    // +default="foobar"
    image string,
) *MyType {
    return &MyType{}
}

resovles to dag.MyType(image) where image is a required argument even though it has a default value.

edit: testing in v0.10.0
if I add the +optional pragma it works as expected, but I was expecting that not to be necessary given that it has a default value

sharp zealot
#

Oh, I know. I had just renamed a module

dense canyon
# dense canyon 👋 not sure if this is considered an issue and/or it has been reported but is i...

just double checked with the typescript SDK and seems to be working as intended as the generated Go code seems accurate even though I didn't have to specify stringArg as an optional parameter for containerEcho

  @func()
  containerEcho(stringArg: string = "foo"): Container {
    return dag.container().from("alpine:latest").withExec(["echo", stringArg]);
  }

resolves to:

 func (*dagger.Testmodule).ContainerEcho(opts ...dagger.TestmoduleContainerEchoOpts)
rocky harbor
#

@Erik Sipsma I just remembered why ID

rocky harbor
# dense canyon 👋 not sure if this is considered an issue and/or it has been reported but is i...

Yeah there's some subtleties around optional/default still, @kind carbon wrote an issue describing them all recently https://github.com/dagger/dagger/issues/6749

GitHub
Zenith: Confusion over what "optional" means · Issue #6749 · dagger...

Summary While documenting optional and default values in Python functions I came upon some inconsistencies in implementations due to different assumptions. The term “Optional” that’s used in module...

#

How does Dagger establish if a pipeline

#

dagger/core/docs/d7yxc-operator_manual.m...

wheat river
# sharp zealot It seems related to having a `go.work`. Removing it made the problem go away. Im...

It occurs the same to me. By removing the go.work file, it was able to create the module. Isn't expected though. For a set of modules/functions inside a mono repository (I guess it's going to be the most common use case for shared modules) isn't ideal to remove the go.work and put it back after the module was created. 😛

Error: load package ".": err: exit status 1: stderr: go: module main appears multiple times i
      ┃ ace

thinkies

honest hearth
#

It seems related to having a go.work.

honest hearth
#

Is there a way to handle when a library you might call in your function container needs an arbitrary binary dependency?

The example I ran into was trying to use psycopg2 as a function which has a dependency on the psql client binary (usually brew install postgres or similar puts it on).

I guess one somewhat non-ergonomic approach is to point the binary at the module and stage it in to the container manually via a function argument.... not really a super great UX though

subtle plinth
#

You can use.file("") on an image that has the binary and WithFile in your container that needs it

thorn moat
sharp zealot
#

I think I am hitting a namespace conflict between 2 different dependencies...

  • github.com/shykes/daggerverse/docker-compose(): DockerCompose
  • github.com/shykes/daggerverse/docker().Compose(): DockerCompose

When I load both in my module, one DockerCompose type seems to overwrite the other.

thorn moat
#

That's at least what we've said in the past, think we just never got around to it

latent trellis
#

Has anyone experimented with SLSA and Dagger?

deft rain
#

(for context, i worked on the attestations support in buildkit for quite a while)

subtle plinth
#

I can't get this to work:

.withMountedCache(
    `${workingDirectory}/node_modules`,
    dag.cacheVolume(packageName)
)

Keep getting GraphQLRequestError: /code/node_modules: cannot retrieve path from cache

Also, why aren't "used modules" available via dagger call or dagger functions ?

honest hearth
#

Are Services supported with functions? I noticed all the services docs kind of disappeared with the latest release. I'm trying to get h2c working right now where I get a host service and expose it to my Function and it's not quite clear how to achieve that

deft rain
warped canyon
honest hearth
#

It has to be passed as an argument. For

sharp zealot
#

I am stuck on either the same problem, or a very related one... This time there is no namespace conflict, and I have no clue what is causing it 😭

#

I'm going to hang out in #911305510882513037 for a while, in case someone is feeling generous and wants to help me troubleshoot. I'm blocked from shipping the solution for #1213207059974197308 (fyi @dense canyon )

#

Could anyone try to reproduce this please?

dagger functions -m github.com/shykes/daggerverse/scratch/bitgantter

I see this error:

Stderr:
# bitgantter/internal/dagger
internal/dagger/dagger.gen.go:2549:4: cannot assign to val.Client (neither addressable nor a map index expression)
internal/dagger/dagger.gen.go:2731:23: field and method with the same name Client
    internal/dagger/dagger.gen.go:2721:2: other declaration of Client
internal/dagger/dagger.gen.go:2736:11: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in struct literal: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2746:11: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in struct literal: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2760:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2809:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2822:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2835:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2847:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2859:34: cannot use r.Client (value of type func() *DockerCli) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *DockerCli does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
internal/dagger/dagger.gen.go:2859:34: too many errors

@deft rain I was told by @mossy hazel that this error looks suspiciously like the name mangling issue he encountered with his SSH module. In my case, note that there is DockerCli, which looks like it could suffer from a similar problem.

thorn moat
deft rain
sharp zealot
thorn moat
sharp zealot
#

I tried with a python downstream module, and there is no error

#

Currently filing an issue

deft rain
#

this is because of Client as a method

#

I screwed up, and accidentally made it a reserved name

sharp zealot
#

@deft rain thank you! So if I remove the field Client from my type Image, the module should work with 0.10.0?

#

Ah maybe I can even just make it private

#

Nice! Just making that one field private fixed the problem.

#

Note, I would never had guessed which field was causing the issue... Another one for the "confusing error messages" epic

sharp zealot
#

@thorn moat when I persist a Service in an object's field, then chain several calls to that object, is it possible that the service gets re-instanciated instead of persisted?

#

I'm investigating a bug in my module, and it seems to point to a situation where the state of my service gets wiped in between calls to it

thorn moat
sharp zealot
thorn moat
#

the issue does, the relevant context is in the comments

sharp zealot
#

Oh I see, the solution to that problem is the cause of my problem

thorn moat
#

yep yep

sharp zealot
#

Got it

subtle plinth
deft rain
subtle plinth
#

1: GraphQLRequestError: /code/node_modules: cannot retrieve path from cache

#
  1. Boooo
deft rain
# subtle plinth 2. Boooo

any suggestions for how we could get round the inevitable name conflicts? what if different modules define the same names?

subtle plinth
#

dagger use ../rust --alias rust or omit alias to have none

dagger call rust:build ?

#

Moonrepo does a really good job of providing composable builds/tasks for monorepos, definitely some inspiration could come from there

latent trellis
#

I started to see No space left on device errors on GHA. Could be anything, I just wonder if it's something new that others started to experience as well?

kind carbon
#

dagger use ../rust --alias rust or

wintry prism
#

I started to see `No space left on

sharp zealot
#

Calling for suggestions! We're going to announce Daggerverse soon. As part of that, we want to highlight specific modules with specific examples of using them. The goal is to get people excited about the possibilities.

I would love to see your suggestions for the most exciting snippets 🙂

Also: what medium should we use in the blog post to communicate the examples? CLI snippet only? Code snippets also? An animated gif? Something else?

Thanks, and I look forward to sharing the joys of the Daggerverse with the world!

sharp zealot
#

I got hit by the arbitrary laziness rules while developing a module.

  • One function returns a custom type and an error (I force synchronous resolution, things can fail). But Go SDK exposes it as lazy. I cannot get the error right away. Unclear how to control that (add a Sync() method to my custom type? But how do I implement it?)

  • Another function returns a scalar, with no error. It's just an accessor. But Go SDK forces clients to pass a client and check for error.

TLDR: I got the perfect combo of too much laziness, and not enough

sharp zealot
#

Dumping a few requests, I will file issues (or maybe they already exist).

  1. I really would love dagger install to always work. This means defaulting to installing in the user directory if not currently inside a dagger module. This would help simplify docs and daggerverse, because dagger install could always be step 1, then all the dagger call examples could reference the shortname.

  2. Some commands, like dagger develop, don't work in subdirectories of the module. But they should. Same as git commands. I think someone mentioned that one already.

  3. Query multiple fields from the CLI. GraphQL is super powerful for querying structured data, but dagger call is not good a querying modules that take advantage of this. A simple improvement that would go a long way let me query multiple scalar fields at once, comma-separated. Eg. dagger call lint checks filename,error,rule-name,rule-description.

  4. Allow querying objects. When dagger call returns an object, don't fail with "requires a sub-command". Instead, query all the scalar types, and render them. For core types (container etc) query a safe and useful subset (which is almost all of it). Is it perfect? No. But it's better than the current situation.

  5. Error messages... Lots of low hanging fruits there. But that's already on the radar.

wintry prism
#

@sharp zealot would 6. be ability to call core API from the CLI?

sharp zealot
#

Yes that would be cool too. I have a workaround, so not as urgent. But yes.

wintry prism
#

I remember you mentioning that and found myself wanting it the other day.

warped canyon
#

Big fan of all of these!

#

When I was working on a typescript module yesterday the runtime was taking about 30 seconds to build every time I ran anything. Afaik that's expected right now since the runtimes aren't cached between sessions, right?

sharp zealot
#

I was going to show off a cool module, but I just hit a major performance edge case, so instead I'm going to show off that 🙂

#

The less-cool but less-slow example:

Run the official Dagger markdown linter against @warped canyon 's modules repo, and show the raw json report 😛

dagger call -m github.com/shykes/dagger/functions/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 json
#

You can also query the individual fields of each check:

dagger call -m github.com/shykes/dagger/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 checks error-detail

--> Unfortunately you can't query more than one scalar field from the CLI... yet 🙂 See checks --help for a list of fields

#

Now here's the part that gets slow: since I can't query multiple scalar fields in the checks, I thought - hey, what if I added a format function?

dagger call -m github.com/shykes/dagger/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 checks \
 format --text 'Dude! You messed up {{.FileName}} at line {{.LineNumber}}, by violating this rule: "{{.RuleDescription}}".'

--> This works, but on large results it's VERY slow. It looks like it runs the function sequentially against each check in the array. With the current overhead of function calls, it appears that adds up a lot.

#

Ah, actually it's pretty fast on that repo. I tried running it against dagger/dagger, and gave up before it completed...

#

For you @warped canyon 😛

Dude! You messed up dockerd/README.md at line 9, by violating this rule: "Multiple consecutive blank lines".
Dude! You messed up fly/README.md at line 4, by violating this rule: "Fenced code blocks should be surrounded by blank lines".
Dude! You messed up golang/README.md at line 5, by violating this rule: "Hard tabs".
Dude! You messed up golang/README.md at line 6, by violating this rule: "Hard tabs".
Dude! You messed up golang/README.md at line 7, by violating this rule: "Hard tabs".
Dude! You messed up netlify/README.md at line 4, by violating this rule: "Fenced code blocks should be surrounded by blank lines".
Dude! You messed up proxy/README.md at line 1, by violating this rule: "Trailing punctuation in heading".
Dude! You messed up proxy/README.md at line 5, by violating this rule: "Hard tabs".
Dude! You messed up proxy/README.md at line 6, by violating this rule: "Hard tabs".
Dude! You messed up proxy/README.md at line 7, by violating this rule: "Hard tabs".
Dude! You messed up proxy/README.md at line 10, by violating this rule: "Multiple consecutive blank lines".
#

And finally... You can also fix the simple errors (in theory)

# 
dagger call -m github.com/shykes/dagger/linters/markdown@daggerize \
 fix --source https://github.com/kpenfound/dagger-modules \
 export --path ./fixed-it-for-you/
warped canyon
sharp zealot
#

Lately we've been talking a lot about the downsides of having functions be completely separated from the git context of their module (no ambient access to the monorepo etc).

But there are advantages too... This is a good example.

Is that embedded markdown linter a "main" function or a "utility" function? I say: yes 🙂

rose hinge
#

Hi, I begin to play with dagger functions since some days but I'm facing a bug.
I have somewhere in my module an optional input with a list of lists when I'm calling the code directly from my golang code everything work without using this input
When I try to do the equivalent with the cli with same args I have the following error: Error: unsupported list of lists for flag: system-setup-cmds
(So it's not blocking me but I was finalizing my module documentation)

#

Should I avoid to have list of lists and doing chaining functions ?

#

what is strange is dagger installing the module or calling my test code for testing this function is working well

sharp zealot
#

@rose hinge it's perfectly valid to take list of lists as argument in your function. There's just a limitation in the CLI, which prevents passing that particular argument. It's strictly a limitation of the CLI, as a client to your module's API. Your module is perfectly fine.

#

(sorry about that limitation 🙂

rose hinge
#

ok no problem so it means when I have this type of args the function is no more callable from the cli even if this one is optional

sharp zealot
#

Since it's optional, it should still be possible to call the function without passing that particular argument

sharp zealot
rose hinge
#

@sharp zealot do you want I open an issue or there is yet one ?

sharp zealot
#

@warped canyon export only the linted files:

dagger call -m github.com/shykes/dagger/functions/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 files \
 export --path ./linted/
sharp zealot
#

Query the contents of the file for an individual check (limited to one check here, to avoid too much output)

dagger call -m github.com/shykes/dagger/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 checks --limit=1 \
 file
#

Query the fixed file for an individual check (if fixable):

dagger call -m github.com/shykes/dagger/linters/markdown@daggerize \
 lint --source https://github.com/kpenfound/dagger-modules \
 checks --limit=1 \
 fix
#

And finally... 🙂

#

Show the diff for each (fixable) check:

 lint --source https://github.com/kpenfound/dagger-modules \
 checks  \
 diff
#

I kind of want to shorten that path, and make it github.com/dagger/dagger/linters

sharp zealot
wheat river
#

A quick and sort of a more broad question rather than a specific Zenith question (can I still use Zenith? it has a special space in my heart I'd say 😅 ) — what's the most recommended approach to pass content into our modules? I usually apply WithMountedDirectory but sometimes when the directory that's being mounted has a lot of files (e.g.: a .terraform folder, or a .git directory) makes things slow — and, as far as I can tell there isn't a way to exclude files or directories as we have when we're using WithDirectory where we're copying into the container's FS.
🤔

latent trellis
#

dagger init doesn't generate a .gitignore file anymore?

latent trellis
#

Also, the module name in go.mod is weird.

dagger init --name slsa-verifier --sdk go --source .

Results in: module dagger/slsa-verifier

Is that intentional?

deft rain
#

yes, it's been changed from main now

#

we need something unique per module, otherwise the daggerverse structure doesn't work (since go.work is a thing, blehh)

#

and we can't just have the dagger module name, since some go module names are reserved (like go)

deft rain
#

there's a prior conversation somewhere

latent trellis
#

Understood, thanks

latent trellis
#

How do I achieve something like this using WithMountedFile?

c.Directory("/work/artifacts").WithFile("", artifact)

Basically I want to mount the file using it's original name, without having to resolve it first. Is that possible?

#

This is what I'm trying right now:

    artifactsDir := dag.Directory().With(func(d *Directory) *Directory {
        for _, artifact := range artifacts {
            d = d.WithFile("", artifact)
        }

        return d
    })

ctr = ctr.WithMountedDirectory("/work/artifacts", artifactsDir).

But I don't know if this is the best option from a performance perspective.

ashen ember
#

👋🏼

I am not sure if something has changed but I am getting 403 errors while pulling a public container image from GitHub

func (m *TroubleshootDagger) TestPull() *Container {
    return dag.Container().From("ghcr.io/chainloop-dev/chainloop/cli:v0.75.2")
}

and this is the error

dagger call test-pull                      
✔ initialize 0.5s
  ✔ ModuleSource.asModule: Module! 0.1s
✘ TroubleshootDagger.testPull: Container! 0.6s
  ✘ exec /runtime 0.6s
  ┃ json: error calling MarshalJSON for type *dagger.Container: input: container.from failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Achainloop-dev%2Fchainloop%2Fcli%3Apull&service=ghcr.io: 403 Forbidd
  ┃                                                                                                                                                                                                                                                                                       
  ✘ Container.from(address: "ghcr.io/chainloop-dev/chainloop/cli:v0.75.2"): Container! 0.2s
dense canyon
#

👋 question about service binding and the ability for the DAG to start resolving things ahead of time which don't require the service to be ready. i.e I have the following code

    ctr, err := m.goBase(ctx, c)
    if err != nil {
        return "", err
    }

    return ctr.
        WithServiceBinding("postgres", postgres.AsService()).
        WithExec([]string{"go", "run", "main.go")
        Stdout(ctx)
}

thing is that goBase has some WithExecs like go mod download before the postgress service gets bounded but given how the DAG gets resolved today, the go container won't execute any anything until the postgress service is healthy. I'm aware I can improve this by manually calling sync before binding the postgres service, but it still doesn't quite solve the problem as the postgres service won't initialize until the sync is done 😬 .
I'd assume it's not a trivial thing to do. cc @thorn moat ?

sharp zealot
# wheat river A quick and sort of a more broad question rather than a specific Zenith question...

WithMountedDirectory is cheaper because there is no copy required. WithDirectory always copies, so even though it has include/exclude, that will only optimize that last copy. If you mount, there is no copy.

Either way, your choice will not affect performance of how the input directory is produced upstream from you. In particular, if it's loaded from a local filesystem (dag.Host.Directory, dag.CurrentModule.Workdir...) and that directory is very large, any includes/excludes need to be applied upstream at that point. At the moment the CLI cannot specify include/exclude, that's the issue @latent trellis is encountering.

deft rain
#

still got a lot of work to do, but happy with the skeleton skeleton

thorn moat
sharp zealot
#

Should I expect a merge conflict for my little markdown linter PR?

deft rain
#

thankfully not ❤️ i'm still working out exactly how to integrate this in, i'd like to aim to get it in asap, even if it results in a slightly worse experience - at least that motivates us to iron out the issues

sharp zealot
thorn moat
#

really nice seeing all that dogfooding feedback

deft rain
#

new tui is so nice

#

it does such a good job of "what's happening right this second" - which is actually really helpful to indicate where there may be performance issues

sharp zealot
#

agreed!

deft rain
#

if i could have a strange feature request @thorn moat 😆
so i made this little helper dev function to drop you into a terminal with a connected dagger instance, and i really like it, and it's neat.
but. i kind of lose visibility of the engine logs while that's happening, and there's not really a way to grab them, but it almost feels like now i'm asking for a full on terminal emulator

#

(i might also not always want service logs... like it's almost like i want a tmux integration or something like that)

thorn moat
# deft rain if i could have a strange feature request <@108011715077091328> 😆 so i made th...

lol, that's literally what it's doing (it's actually running in a virtual terminal emulator and painstakingly making sure to write things that go out-of-view into the outer terminal's scrollback), but I never got around to implementing side-by-side visualization in one TUI. implementing a whole terminal emulator is a lot of work so now I'm playing with @sharp zealot 's idea of running a separate dagger watch/wiretap/trace/logs command to view activity from your dev shell instead

deft rain
#

for context - today i tend to develop dagger with 3 terminals:

  • run builds, run tests, dev commands, etc
  • view logs of the running dagger-engine.dev
  • one client terminal with _EXPERIMENTAL_DAGGER_RUNNER_HOST set, so i can run round my local filesystem and play with examples
    collapsing everything together with one module command is nice, but it feels like i lose a little bit of visibility into what's going on
#

dagger watch would be pretty close! something that could let me attach to a specific running session and view different things that are going on 😄

thorn moat
#

yup yup. I actually had to implement all the necessary infra already as part of the otel work. there's a little otel pub/sub stack, but it's not exposed yet. we could probably expose it as a GraphQL subscription

#

could possibly even let you interactively select which session you want to watch, based on the trace's root span name (which is the full command + args, so should be easy to identify)

deft rain
#

this also plays a bit into another weird ask, which is that sometimes with lots of services, the screen can get a bit full - it would be really nice if i could cycle between the combined view and each service / the main pipeline as i want to

#

but then the tui becomes interactive, so maybe that's a no-go

thorn moat
#

keyboard interactivity is fine, mouse interactivity is fine too as long as we don't grab the scroll wheel. I've been meaning to add arrow keys so you can move back to things that go out of view at least

sharp zealot
#

@deft rain what I want with watch is a continuous TUI feed of everything I'm running on a particular machine, or perhaps terminal. Just leave it there running all day, like an activity monitor

thorn moat
deft rain
#

is there an internal/external writeup of the context of the otel work? sorry, i'm a bit out of the loop so far on it

thorn moat
# deft rain is there an internal/external writeup of the context of the otel work? sorry, i'...

not really, tl;dr we're at a point where we can break compatibility with the old Cloud event stream, and we figured it'd be better to take that opportunity to adopt a standard telemetry pipeline rather than double-down on Progrock for another iteration. the extra context is otel now has support for logging (kind of... we've had to write our own Go SDK boilerplate), which was the main value-add Progrock had over it

latent trellis
#

Question: which one do you think makes more sense?

type Python struct {
    // +private
    Ctr *Container
}

func (m *Python) Container() *Container {
    return m.Ctr
}

vs

type Python struct {
    Container *Container
}

Or to put it differently: does it make sense to hide struct members? They aren't "writable", are they?

I remember I started using this pattern for some reason, but I can't really justify it anymore.

sharp zealot
#

I hide them to keep my api clean

deft rain
sharp zealot
#

unless I specifically want to expose a getter function

latent trellis
#

Yeah, that's my default strategy as well. But it proved useful on more than one occasion to gain access to the underlying container as sort of an "unsafe" API.

deft rain
#

maybe // +hidden could be a thing? hide it, but still allow some sort of access?

latent trellis
#

The question here is whether hiding struct members and exposing them through functions make any difference. Again, I can't recall why I started doing this.

deft rain
thorn moat
latent trellis
thorn moat
#

a configurable scope(?) could make sense

latent trellis
#

yep

#

but for now, I'm good with not mounting cache volumes by default

sharp zealot
thorn moat
#

yea I think at the time we just didn't have bandwidth to shake out the implementation details

rose hinge
#

I ask that because actually we have a pipeline which consist to update and store in a cache volume a maxmind db then we are different services which pull this cache volume for testing / build purpose

#

like that we don' t have to maintain the maxmind db in several repository

sharp zealot
#

@rose hinge you could still do that as long as all the functions using that cache volume are all in the same module

#

But, if they are not in the same module, then you should avoid using cache volumes as an "implicit API" between your modules

latent trellis
#

Here is another question: most of my modules are "low-level", meaning they are centered around a single tool (eg. Go or Helm). In some cases, there are tools around the same ecosystem that rely on each other.

For example:

  • Golangci-Lint depends on Go
  • Helm docs depends on Helm

Currently these (golangci-lint, helm-docs) are separate modules. Although it isn't visible in the API, cross module dependencies actually make these modules more complex.

I'm working on a python module now and I'm trying an alternative approach: instead of making a separate poetry module, it's gonna be a "submodule" of Python.

For example:

dag.Python().WithPipCache(...).Poetry()

// here comes the poetry specific API

A part of me is against it, because it's hard to define a fine line what should be part of a python module and what shouldn't. At the same time, maintenance is definitely easier (for example: due to that WithPipCache call, part of the module initialization would have to be delayed in case of a separate poetry module)

Any thoughts about that?

warped canyon
#

Here is another question: most of my

sharp zealot
# sharp zealot 6. call core functions from the CLI 7. `-m FOO` should load the latest tagged v...
  1. Let me reference submodules of my dependencies 🙂 That way I could do:
# Install my daggerverse
dagger install github.com/shykes/daggerverse --as=shykes
# Call supergit from my daggerverse
dagger -m shykes/supergit call --help
# Call docker from my daggerverse
dagger -m shykes/docker call --help

etc. Right now this fails because it interprets it strictly as a local directory. 🙏 🙏

sharp zealot
#

@kind carbon would you consider running a "How to write a Dagger SDK" mini-workshop? 🙂 🙏 I would attend eagerly

#

Trying to play with your codegen module at the moment

#

Is there a way to introspect my dependencies?

kind carbon
kind carbon
sharp zealot
#

Are you sure? That would be useful for my use case, but seems like a pretty serious scope leak no?

#

How does a dependency (in this case the module codegen) have access to its siblings in the dependency tree?

kind carbon
#

The codegen module simply makes a introspection query to the API. It's what any SDK needs to do and is native to GraphQL. The difference is just marshaling the resulting JSON into a data structure and attaching functions to it. But the data is available to anyone with any access to the API.

sharp zealot
#

I would need a way to pass a target module as argument to codegen, so that it can somehow introspect that

kind carbon
#

No, the main introspection feature has no knowledge of modules and functions. It's just accessing the whole GraphQL schema. Try putting this in the playground or a dagger -m <mod> listen session:

dagger -m github.com/helderco/daggerverse/codegen@v0.1.0 call introspection-query | pbcopy
sharp zealot
#

Yeah, I understand

kind carbon
#

If you want module specific introspection, and navigating through the intermediary TypeDef representations, you can see how the following's implemented:

dagger -m github.com/helderco/daggerverse/codegen@v0.1.0 call experimental --help
sharp zealot
#

But whether I use raw graphql introspection, or typedef introspection, I'm still only introspecting the current API available to the codegen module, which happens to not include any of the caller's own dependencies, right? That's the problem I'm trying to solve: I want to write a tool that can interact with the caller's current dependencies. It seems there isn't a way to do that from within a 3d-party module, so I might have to hardcode it into a custom client instead (or fork the dagger CLI itself )

#

ie. I want to be able to do:

$ dagger install foo
$ dagger instal bar
$ <magic-command> list
foo
bar
#

I guess I could pass the dagger.json as an argument

kind carbon
#

Is it limited? I'd expect it would be able to see the whole API, but haven't actually tried it like that.

sharp zealot
#

To my knowledge dependencies are scoped: each module only sees its own dependencies. This is a valuable feature, for example it eliminates a whole class of "dependency hell" problems. Different modules in the dependency tree can import different versions of the same module, without conflicts

kind carbon
#

Ah, that's right. The codegen module also works from an introspection result if you can get one by other means. That's what I planned on using in the runtime module since an introspectionJson argument is provided by the server. But where can you get access to the API with dependencies included?

#

Right, you can loop the dependencies in dagger.json and load them individually from your module.

#

There's an API for that, not sure atm if it's public or not.

plucky ermine
sharp zealot
#

@kind carbon yeah exactly, I was wondering if codegen did any of that, or if not - maybe I can contribute it? 🙂

#

Also I was wondering if the CLI had the ability to pass a Module type as argument, maybe that can be a contribution down the line also

thorn moat
sharp zealot
#

wat

thorn moat
#

he's also doing that ID flattening thing i've been bringing up. truly a real-life 🎅

sharp zealot
kind carbon
sharp zealot
kind carbon
#

Oh look at that:

$ dagger call experiment --mod-source=.
codegen
docker

dagger.json:

{
  "name": "python-codegen",
  "sdk": "python",
  "dependencies": [
    {
      "name": "codegen",
      "source": "../codegen"
    },
    {
      "name": "docker",
      "source": "github.com/shykes/daggerverse/docker@c9a80c9eac0675a53a7e052da3594207f4235988"
    }
  ],
  "source": ".",
  "engineVersion": "v0.10.0"
}
#

Function:

    @function
    async def experiment(self, modSource: dagger.ModuleSource) -> list[str]:
        deps = await modSource.dependencies()
        return [await mod.name() for mod in deps]
sharp zealot
#

Ha ha amazing 🙂

#

I thought I'd share a few neat tricks I discovered today:

  1. You can run a function directly from a pull request 🙂 For example: dagger call -m github.com/dagger/dagger/linters/markdown@pr/6816/head --help

  2. Sometimes your function runs a command in a container, and you also want to expose the container for interactive debugging. But then you don't have anywhere to store the same command. The trick: you can write that command to the container's shell history, so from the terminal you can just type ⬆️ and it's there 🙂

kind carbon
#

Oh 2) is genius!

sharp zealot
sharp zealot
#

how cool would it be if a future version of daggerverse included auto-generated gifs of the coolest way to use a module 😁

sharp zealot
#

🐞 I'm hitting another codegen error triggered by installing a dependency... 😦

dagger init --sdk=go --source=.
dagger functions # success
dagger install github.com/quartz-technology/daggerverse/dagger@7c59dc94edaf73d9324abe2c36f946ace0d832e6
dagger functions # error

My output:

✘ initialize 0.6s
  ✔ ModuleSource.asModule: Module! 0.3s
  ✘ Module.initialize: Module! 0.2s
    ✘ exec go build -o /runtime . 0.2s
    ┃ internal/dagger/dagger.gen.go:1949:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
    ┃ internal/dagger/dagger.gen.go:2008:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
    ┃ internal/dagger/dagger.gen.go:2023:21: field and method with the same name Query
    ┃         internal/dagger/dagger.gen.go:1903:2: other declaration of Query
    ┃ internal/dagger/dagger.gen.go:2027:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
    ┃ internal/dagger/dagger.gen.go:2041:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
Error: input: module.withSource.initialize failed to initialize module: failed to call module "dsh" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1

Stderr:
# main/internal/dagger
internal/dagger/dagger.gen.go:1917:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:1929:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:1937:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:1949:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:2008:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:2023:21: field and method with the same name Query
    internal/dagger/dagger.gen.go:1903:2: other declaration of Query
internal/dagger/dagger.gen.go:2027:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
internal/dagger/dagger.gen.go:2041:15: r.Query.Select undefined (type func(ctx context.Context, query string) (string, error) has no field or method Select)
$

Calling the dependency itself works fine:

$ dagger functions -m github.com/quartz-technology/daggerverse/dagger@7c59dc94edaf73d9324abe2c36f946ace0d832e6
Name      Description
cli       CLI returns a ready to use Dagger container with CLI installed.
install   Install returns the same Container with the Dagger CLI installed in it.

Looks like maybe a name conflict on Query? https://github.com/quartz-technology/daggerverse/blob/7c59dc94edaf73d9324abe2c36f946ace0d832e6/dagger/commands.go#L66

GitHub
daggerverse/dagger/commands.go at 7c59dc94edaf73d9324abe2c36f946ace...

Daggerverse modules made by Quartz. Contribute to quartz-technology/daggerverse development by creating an account on GitHub.

#

Assuming the problem is indeed a name conflict: I don't know if there are new name conflicts popping up over time, or if I'm just gradually discovering them, but it's been quite disruptive, at least to me.

#

sorry @upbeat herald I had to uninstall your module

upbeat herald
#

However that’s strange, I used the module in other modules and it worked just fine

glass zephyr
#

dumb question, how do I run a command that creates a file inside a container and then fetch that file?

ctr.WithExec(args).Stdout(ctx)

for some command that writes to the filesystem, and then

ctr.File("/path-to-file-written-by-command-above").Contents(ctx)

shows the file not found

┃ 2024/03/07 10:31:17 error reading key for sa: input: resolve: container: from: withEnvVariable: withEnvVariable: withFile: file: lstat /root/sa-key.json: no such file or directory
#

I'm guessing File() operates on the container image rather than the container filesystem that results from the first command?

deft rain
#

hm, that's really odd, File() should be operating on the filesystem in ctr

#

what i'd do, is re-assign ctr = ctr.WithExec(args)

#

then get Stdout and File on it, and there should be no issue hopefully!

glass zephyr
#

oh cool, yeah maybe I'm expecting ctr to be mutated, but I actually need the reference to the new one. I'll test this - thanks @deft rain!

deft rain
#

generally, i don't think any dagger value ends up getting mutated - it's a bit of a more functional approach to an api (which means you can easily get to any point in history of a build, which is quite cool)

deft rain
#

potential idea for daggerverse here: could we run codegen for submitted modules for go/ts/python and check that the codegen succeeds? i'm guessing this isn't something we're currently doing? @thorn moat

thorn moat
dense canyon
deft rain
#

yeah the engine loading works fine, it's when the module gets codegen-ed for go that it has the issue

thorn moat
#

oh i see what you mean now

agile saddle
#

Is there a way to do a higher order function? Eg a module with a parameter which is a function that takes a container and produces another container?

Use case: implement codemod s using dagger, and provide a higher order function that produces a diff a before/after and runs the tests after running the codemod

sharp zealot
agile saddle
#

I take yet to mean there’s at least some interest in this. Is there an issue tracking this request?

sharp zealot
thorn moat
agile saddle
thorn moat
agile saddle
#

I’d like to create an implementation of the interface in another module (eg run clang-tidy, or comby replace class A with class B). Then call a function in the module that defined the interface to run a diff and test it

#

So the thing to pass would be either a factory function name or a constructor to specify the implementation of the codemod to use and probably a container as the argument to the codemod

sharp zealot
#

Oh sorry, maybe I misunderstood. Was focusing on the idea of passing functions by value in an argument. But it looks like maybe that's not what you were asking for.

agile saddle
#

Ideally I can write these in different languages, but what I care about is Python on both sides because that is what I can expect colleagues to write

agile saddle
thorn moat
#

I've wanted the same thing, so you can define a module for your "Home" (i.e. personal dev environment, editor config etc.) that can install itself into per-project dev entrypoints (assuming a common OS, in my case Wolfi)

agile saddle
#

That would be another use case Ive considered @thorn moat

#

I just figured tooling around portable code mods was an easier sell 😅

thorn moat
#

Yeah I think the thing in common is taking an interface arg, which is tricky because the problem to "solve" in the CLI then is how to pass an arbitrary object in. Theoretically it's possible by passing any object's id string fetched from the GraphQL API, but that's not the best DX, and isn't exposed atm. If you have a module that can be constructed without params, maybe we could support something like <module ref>#<function> (strawman), so something like github.com/vito/daggerverse/home#installer for my case, or github.com/foo/codemod#modder for yours (not sure, just spitballing)

agile saddle
#

Yeah, I’m fine with default constructable being a requirement

thorn moat
#

Or we go full IDs + pipes, dagger call -m usermod use-iface --fooer $(dagger call -m foo fooer id) - a bit verbose, just putting it out there

agile saddle
#

For non default constructable things, pipe syntax is an acceptable compromise, but default constructors give me 90% of my use cases

sharp zealot
# thorn moat Or we go full IDs + pipes, `dagger call -m usermod use-iface --fooer $(dagger ca...

I needed that exact pattern for my daggy module (LLM does the stitching via CLI, so need the IDs). I got it to work, but needed a wrapper module (github.com/shykes/daggerverse/supercore).

I needed the wrapper because:

  1. You can't call ID from core types (special case in the CLI to block it)
  2. You can't call core functions from the CLI
  3. Even if you could, the CLI interprets ID arguments in a custom way, you can't pass the litteral ID

Note: I'm talking about the calling pattern, not the part where you pass functions or interfaces around. I'm doing it for regular objects.

thorn moat
#

(this is the "ID flattening" thing I mentioned yesterday, requires a protocol change, so good time to bikeshed)

sharp zealot
# sharp zealot I needed that exact pattern for my `daggy` module (LLM does the stitching via CL... 
DAGGER_MODULE=github.com/shykes/daggerverse/supercore \
dagger call \
 fs \
 load \
 directory --snapshot=$(dagger call \
  git \
  repository --url=https://github.com/dagger/dagger \
  ref --name=main \
  tree \
  save \
 ) \
 entries

Notes:

  • I had to create custom wrapper types in order to get the ID. Since ID is reserved by the SDKs, I had to call it something else: save
  • Since I had to write custom wrapper tools anyway, I took the liberty of experimenting with improvements to the core API 🙂 Hence the name supercore. I'm also going to push core which is a straight 1-1 pass-through: it basically only exposes the top-level funtions as is, with strings instead of IDs. You have to call sync from the CLI until we remove the "hidden ID" issue
agile saddle
#

Unrelated is there a way in a module to look for a device file on the HOST, and then mount it on a container passed as an argument

I’m thinking about generalizing GPU device support to incorporate the weird accelerators that I use at work.

#

Another use case would include a module that mounts /dev/kvm to allow VMs to run on Linux for other operating systems

#

Both of these would need to be “privileged” modules

sharp zealot
#

mount custom devices

sharp zealot
restive shore
sharp zealot
#

The above 👆 is a full bash-compatible interface to the dagger engine. I'm experimenting with the syntax for it. Can be used to pass all those repetitive flags , cc @latent trellis 🙂

#

Imaginary scenario:

$ jq -r .scripts.build.lang dagger.json
bash
$ jq -r .scripts.build.source dagger.json
kpenfound/golang --proj=. \
 build --args=./cmd/foo --arch $DAGGER_ARCH --os $DAGGER_OS \
 export --path=./bin/foo
$ dagger run build
$ ./bin/foo
restive shore
#

I sort of get it. Would love to see more. It's like a package.json basically.

sharp zealot
#

It's a combination of:

  1. A builtin bash-compatible shell that executes dagger functions instead of local commands
  2. A simple way to save scripts in dagger.json and run them with a short commands - a-la package.json. Unlike regular npm scripts, those would be interpreted (and sandboxed) by the builtin shell. So more reproduceable, safer, etc. We can allow environment variable access, but do static analysis of which variables are accessed, so that the user can decide whether to allow it or not; we can virtualize the variables; we can scope local filesystem access to the boundaries of the current module; etc
latent trellis
restive shore
nova hazel
#

Is it possible to dynamically add/call a dependency at runtime to a zenith module?

deft rain
#

If I export a directory using export, have we considered having a way to replace the contents that were previously there? (instead of overlaying everything that was there, so old files wouldn't get cleaned up)

latent trellis
rocky harbor
#

Is it possible to dynamically add/call a

#

If I export a directory using export,

#

Chatting about Zenith for the next hour in dev-audio for anyone who wants to join!

sharp zealot
sharp zealot
rocky harbor
mossy hazel
#

@rocky harbor I tried what you suggested earlier for calling the dagger cli from a module:

func (m *Ci) DaggerCall(ctx context.Context) (string, error) {
    ctr := m.getBaseImage(m.WorkDir.Directory("ci")).WithExec([]string{"sh", "-c", "apk add --no-cache docker"})
    ctr = ctr.WithExec([]string{"sh", "-c", "curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/bin sh"})
    ctr = ctr.WithExec([]string{"sh", "-c", "dagger functions"})
    out, err := ctr.Stdout(ctx)
    if err != nil {
        return "", err
    }
    return out, nil
}

I get 1: [0.59s] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? - Given the CLI still depends on the docker cli, I guess I'll have to wait for supporting mounting sockets in module to make that use case work

thorn moat
# sharp zealot 9. Let me pass subdirectories to a git repo in the command-line. eg. `--source h...

the tl;dr on how this related to v2 refs: I wanted to turn module refs into just a way to refer to a Directory, so dagger -m github.com/vito/daggerverse/apko would just be how you referred to a Directory that came from a git repo. (or git://git.sr.ht/foo/bar//baz to show an example of not special-casing github.com). So all module flags would actually just be Directory flags, and you could use those same refs anywhere that took a Directory

rocky harbor
#

RFB (Request For Bikeshedding) on this PR for supporting directory arg filtering: https://github.com/dagger/dagger/pull/6857

Specifically, it expands dagger config with subcommands (currently just dagger config include and dagger config views) that each have their own set of "standard" CRUD subcommands, which has all sorts of naming and CLI design things to decide on

sharp zealot
#

Any chance we could relax the rule of "no ID arguments or functions"?

#

At least argument name, seems excssive

sharp zealot
#

Quick DX feedback on the topic of .gitignore. I contributed to someone's module, and got this review on the PR:

Look into potential, unnecessary autogenerated code added (internal/dagger) checked in.

sharp zealot
#

@rocky harbor confirmed that module find-up doesn't work for me in any command:

dagger init --sdk=go repro
cd repro
dagger functions # this works
cd dagger
dagger functions # this fails
rocky harbor
# sharp zealot <@949034677610643507> confirmed that module find-up doesn't work for me in any c...

Ahh okay right we do have findup logic but it's for finding the context root (i.e. .git), but we also want it for finding dagger.json in these cases.

It should be a super simple fix, but one question: if you there was a module in ../some/dir and you ran dagger functions -m ../some/dir/some/subdir would you expect to see the functions for ../some/dir? Basically, should the find-up always kick in, or should only be if you don't specify an -m?

I think it should just always kick in, but just ambiguous enough to double check

sharp zealot
#

My first instinct is that explicit -m means "override whatever you were going to do to find the module: this is the one I want"

rocky harbor
sharp zealot
#

Oh, in your scenario, ../some/dir/some/subdir does not have its own dagger.json then?

rocky harbor
sharp zealot
#

In that case, then yes that makes sense to me. -m is always applied. And findup is always applied.

sharp zealot
#

I think Container.terminal() hides the error message on error

rocky harbor
#

In that case, then yes that makes sense

sharp zealot
#

Does Dagger have a native time type?

thorn moat
onyx remnant
#

How does one create a file and pass it to a container inside a dagger function?

sharp zealot
#

Container.WithNewFile

quick wind
#

@kind carbon Any chance you could update https://github.com/helderco/daggerverse/tree/main/poetry to 0.10.x? Getting The "main" module could not be found. Did you create a "src/main.py" file in the root of your project? for a project structured as follows:

.
└── dagger
    ├── pyproject.toml
    ├── poetry.lock
    ├── sdk
    │   └── src
    │       └── dagger
    │           ├── client
    │           ├── _codegen
    │           ├── _engine
    │           └── mod
    └── src
        ├── constants.py
        └── main.py

and the following in pyproject.toml:

name = "lambda"
packages = [{ include = "main.py", from = "src" }]
sharp zealot
#

Questions on dependency management.

  1. Does dagger ever write git tags to the dependencies source entry in dagger.json? Or only ever raw git commit?

  2. If I manually write a dependency to dagger.json which is pinned by tag rather than commit, what happens? Error?

rocky harbor
sharp zealot
rocky harbor
sharp zealot
# rocky harbor Yeah more or less, in practice its trivial for the engine to resolve any git ref...

Yeah you're right that enforcement is airtight at execution. I only called it "best effort" because you can still write an invalid dagger.json, and publish it over git, we won't (can't) prevent you from doing that since we don't control source distribution.

It's good to have clarity on what is invalid, though, because then we can add little checks and warnings over time, to guide users. Like an error on publish, warning on load, stuff like that

#

FYI @dense canyon @wintry prism 👆

void widget
#

Do you know how to add function arg descriptions on the SDKs ? I can't seem to make it work

sharp zealot
void widget
sharp zealot
#

In python, I'm guessing it's docstrings, that's the standard way

void widget
sharp zealot
# 
$ dagger call -m ../supergit remote --url https://github.com/asciinema/asciinema branch --name main commit tree docker-build
✘ initialize 1.2s
  ✔ ModuleSource.asModule: Module! 0.3s
Error: unsupported list of input type "BuildArg" for flag: build-args

Translation: can't call Directory.DockerBuild from the CLI 😭 😭 😭

sharp zealot
#

I'm tempted to try 1-repo-per-module for my next few modules... Instead of picking weird repo names like shykes/the-dagger-module-for-X, I kind of want to call it.... shykes/X.

#

As in:

  • shykes/git: git as a dagger module.
  • shykes/docker: docker as a dagger module.
  • shykes/tailscale: tailscale as a dagger module

Since it appears Dagger Functions can encapsulate pretty much any software that I'll build, download, deploy, run from the terminal, or integrate with: why bother with intermediary namespaces? My Github is my Daggerverse. Increasingly I will have no other purpose for GIthub, than hold my Dagger Functions, conveniently grouped into modules.

The other practical benefit is that, hopefully, it will make cold load times just slightly shorter - less git material to clone the first time 🤞

plucky ermine
#

I am running dagger in dagger to test a module and running into a strange error that I have not seen before.

Connected to engine 25de772207a8 (version v0.10.2)
1: starting session 
1: [1.01s] OK!
1: starting session [0.22s]
1: connect DONE

2: initialize
2: initialize ERROR: failed to get configured module: failed to get module ref kind: input: moduleSource resolve: get git tags: failed to run git: exit status 128


4: dagger call
4: [0.00s] Error: failed to get configured module: failed to get module ref kind: input: moduleSource resolve: get git tags: failed to run git: exit status 128
4: [0.00s] 
4: dagger call ERROR: failed to get configured module: failed to get module ref kind: input: moduleSource resolve: get git tags: failed to run git: exit status 128


Error: failed to get configured module: failed to get module ref kind: input: moduleSource resolve: get git tags: failed to run git: exit status 128

Does anyone have any ideas?

rocky harbor
#

I am running dagger in dagger to test a

visual hedge
#

Question about the Zenith Doc.

https://docs.dagger.io/quickstart/292472/arguments#pass-a-directory-as-argument

In this section we call the tree command from a local dir.

It's gonna sound stupid but I did it in my home dir and I was wondering why it took minutes to run. I quickly figured out that dagger was copying/caching my whole $HOME.

I completely forgot about it, but last week I introduced dagger to a colleague and he ran through the exact same issue and it took me a couple min to remember about the issue.

I don't think it is particularly obvious for a newcomer that calling tree on ./ Will cache my entire local dir especially when we are not focused on the command itself but rather on understanding what dagger does.

I would highly encourage a slight update encouraging the reader to create a dumb dir with 3 random files. It makes for a much nicer experience 🙂

Pass arguments to a function | Dagger

Pass arguments to a function

#

Or mentioning that passing the --dir Implies that you copy out the local content and that you should avoid running it on any folder structure close to /

sharp zealot
#

OK so... Is there any way to force the allocation of a tty in Container.withExec?

#

Basically an equivalent of ssh -t

rocky harbor
sharp zealot
plucky ermine
#

I don't want to hijack the thread above but I had a similar use case, I am working on some smoke tests for our doc examples inspired by the panic from terminal in 0.10.1 -- I got most of the examples to run but ironically cant get the terminal one to work since it says "terminal not supported in TUI", ironic because it does not panic, but fails in its own way 😄

ruby night
#

Trying to publish a module (again, but with a different org), but getting crawl error: ["error" "get module for ref: no rows in result set"]. First time I see it

sharp zealot
plucky ermine
#

@Helder Correia would you consider

sharp zealot
#

ATTENTION: I am going to rename this channel:

  • Name: #developers
  • Category: USING DAGGER
  • Description: A place to talk about developing for Dagger

Last call for feedback... We can change it if doesn't work.

plucky ermine
#

Is there some nice pattern to make things feel clean that folks have found when you have a function that takes a lot of parameters (like 5+)

plucky ermine
#

I wrote a module that uses another third party go library and basically 0 dagger code and everything "just works™" is this because its go and dagger knows how to execute it?

sharp zealot
#

I just experimented with livestreaming software, to livestream myself daggerizing things 🙂

#

I tried it on my own, but I think it would be more fun as a duo. Also, I got stuck and didn't understand the software I was daggerizing well enough to get unstuck on my own 😛

#

Who wants to join me and daggerize something together live on air? 🙂

plucky ermine
plucky ermine
#

Im cooking up a dagger module that knows how to send toots on Mastodon. excellent

#

Im cooking up a dagger module that knows

plucky ermine
honest hearth
#

Where do I go to follow the work on function/module caching? Cold start time is still a bit spicy these days.

sharp zealot
plucky ermine
#

one thing that is a bit curious is why *Secret has what seems to be an empty default value in when you call dagger --help

Here is the definition

    // mastodon access token
    accessToken *Secret,

But when I run dagger --help for my function I see this

      --access-token Secret    mastodon access token (default :)
#

Does anyone happen to have a more advanced example of using Dagger constructors or interfaces with the go sdk?

plucky ermine
#

I think its really awesome how daggerverse.dev shows you usage examples in shell automatically, have we considered adding this same behavior to --help so that you don't have to explicitly document the examples?

sharp zealot
#

How does everyone feel about allowing file:... for string arguments?

#

I've encountered a few times the situation where an argument could reasonably be an inline string or a file (eg. the contents of a script); but I don't want to have to split my function into fooString and fooFile. I would rather have an easy way for the CLI load a string argument from a file

kind carbon
restive shore
kind carbon
#

Problem is if your string value needs to start with file: for some reason 🙂

sharp zealot
rocky harbor
#

Chatting about modules for the next hour in dev-audio for anyone who wants to join!

kind carbon
#

Oh, you’re 1h early 😦

rocky harbor
kind carbon
#

Yeah 😅

rocky harbor
#

okay, be back in an hour! 😄

rocky harbor
# kind carbon Problem is if your string value needs to start with `file:` for some reason 🙂

I think we could cover all corner cases if we did:

  • --string-arg string:foo - value is foo
  • --string-arg file:foo - value is contents of file foo
  • --string-arg foo - no prefix defaults to string: so value is just foo

Then the corner cases would be handled by:

  • --string-arg string:file:foo - value is the literal string file:foo
  • --string-arg string:string:foo - value is the literal string string:foo

I don't love that but it would allow you to cover every case on a per-arg basis I think

sharp zealot
#

what if my argument is string:string:foo 😛

rocky harbor
#

The logic would be "strip any string: or file: prefix and interpret the rest of the value as indicated by that prefix, otherwise if those prefixes aren't present just interpret it all as a string"

rocky harbor
sharp zealot
#

@kind carbon OK the real topic for our first episode together: HELDER SHOWS SOLOMON HOW TO MAKE A DAGGER SDK 🙂

restive shore
#

oh man! was hoping to join today 😦

rose hinge
#

Hi, I'm playing with dagger function since some weeks, at the beginning I was a little disturbed with the transition from sdk style to function, but now I very like it.
I would like to have a feedback on my node module (https://github.com/Dudesons/daggerverse/tree/main/node).
This module allow to call a lot of functions to do test build ... Also I tried to add "lazy" functions, one for an automatic setup based on the package.json and another one which will try to build a basic pipeline, I would like to know if you think it's a good idea or not
automatic setup: https://github.com/Dudesons/daggerverse/blob/main/node/dagger/auto.go#L9
basic pipeline: https://github.com/Dudesons/daggerverse/blob/main/node/dagger/pipeline.go#L9
On the Pipeline function I was hesitating if it's a good idea to return (string, error) or if I should return my Node struct or a container for more flexibility
Also I have the feeling I didn't see modules around a language which purpose to build and push images, is there any reasons ?

rose hinge
#

Another thing I'm developing a terragrunt module and another one for drift detection but executing this call: dagger call terragrunt with-source --path=/terraform --src=../testdata/terragrunt/ plan --work-dir=/terraform/stacks/dev/europe-west1/staging/bar --detailed-exit-code do
I had in the following output:

...
Stderr:
2024/03/15 08:55:44 traces export: Post "http:///dev/otel-grpc.sock": http: no Host in request URL
2024/03/15 08:55:44 traces export: Post "http:///dev/otel-grpc.sock": http: no Host in request URL
...

Terragrunt add recently an opentelemetry integration: https://github.com/gruntwork-io/terragrunt/issues/2919
https://terragrunt.gruntwork.io/docs/features/debugging/#opentelemetry-integration

In the case a tool have an opentelemetry integration what do you recommend to do ?
If I need to use this kind of feature for debugging, there is a risk to break something in dagger ?

#

Environment of my container:

 dagger call terragrunt with-source --path=/terraform --src=../testdata/terragrunt/ shell
/terraform # env
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
SHLVL=1
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_DAGGER_PARENT_CLIENT_IDS=zxcr8rjez0dw0n2ujeg2ntph0
/terraform # env
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
SHLVL=1
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

(I didn't set any environment variables)

sharp zealot
#

Node module

verbal quest
#

this channel name chefkiss

thorn moat
#

watch out world, i'm writing SDKs for my

plucky ermine
#

I'm still a bit lost on how tagging is supposed to work for a monoerpo of modules, I have this tag for my mastodon module https://github.com/levlaz/daggerverse/tags and for a moment daggerverse did show this version as latest, but now it says my latest version is "main"

https://daggerverse.dev/mod/github.com/levlaz/daggerverse/mastodon@5fe79dd124aab15264773e103d77f038c9779183

GitHub
Tags · levlaz/daggerverse

Personal Collection of Dagger Modules . Contribute to levlaz/daggerverse development by creating an account on GitHub.

mastodon :: Daggerverse

A Dagger module for interacting with Mastodon

plucky ermine
#

Tags · levlaz/daggerverse

plucky ermine
#

Does anyone happen to have a more

rose hinge
#

I'm trying to build a module for publishing my modules.
This module is taking a git repo + a module name
Then this one is checking if the module has git tag or not
According to the function call it will increment major/minor/patch
After that it will git push the tag and run the dagger publish.
Actually I'm blocked on the git push part, my key have a passphrase and for the moment I didn't find a way to mount the ssh agent socket.
Is there a way to do that or for the moment it's a limitation ?

kind carbon
# rose hinge I'm trying to build a module for publishing my modules. This module is taking a ...

Sockets aren't supported yet. See https://github.com/dagger/dagger/issues/6726 and https://github.com/dagger/dagger/issues/6747.

GitHub
Add socket support to `CurrentModule` · Issue #6726 · dagger/dagger

We missed adding support for host sockets to CurrentModule when removing Host from module codegen. We should re-add the equivalent support, it should be pretty straightforward.

GitHub
Issues · dagger/dagger

Application Delivery as Code that Runs Anywhere. Contribute to dagger/dagger development by creating an account on GitHub.

rose hinge
kind carbon
#

Not that I know of \cc @rocky harbor

rocky harbor
# rose hinge ok so waiting that you recommend me to disable to git push part and doing it out...

Yeah unfortunately at this exact moment there's no way to get a host ssh agent hooked up with pipelines running in a module. Supporting sockets as arguments should make that possible; https://github.com/dagger/dagger/issues/6747 Helder linked to is the one to follow for that. It will require explicitly passing the socket as an argument when using dagger call, but that's probably the best way since it will ensure that only the functions you've explicitly passed it to have access (relative to any module always having access to your ssh agent)

sharp zealot
#

Just in case it's useful: I published a module that lets you call core functions directly from the CLI:

https://daggerverse.dev/mod/github.com/shykes/core

(
 export DAGGER_MODULE=github.com/shykes/core
 DIR=$(
  dagger call \
   git --url github.com/dagger/dagger \
   branch --name=main \
   tree \
   sync
 )
 dagger call \
  load-directory-from-id --identifier="$DIR" \
  directory --path=cmd \
  entries
)

Limitations (both are bug reports on dagger)

  • dagger call won't let you call id() which is frustrating. You can call sync() on some types, but on others, like GitRepository, there is no workaround

  • id is a forbidden argument (I don't understand why). So I use identifier instead for load functions

rocky harbor
# sharp zealot Just in case it's useful: I published a module that lets you call core functions...

id is a forbidden argument (I don't understand why). So I use identifier instead for load functions
Yeah I don't think this restriction was necessary, it just got caught up in adding protections against naming fields/functions id. Was an extremely quick fix so sent out a PR here: https://github.com/dagger/dagger/pull/6912

dagger call won't let you call id() which is frustrating. You can call sync() on some types, but on others, like GitRepository, there is no workaround
This one is a lot more nuanced, left thoughts on it in the issue https://github.com/dagger/dagger/issues/6710#issuecomment-2012902063

sharp zealot
rain hedge
#

Hi All,
I have really noob questions.. As a Service Platform, we would like to have a centralized repository to develop our functions.
In Github Actions, how each engineering teams can access to functions that we created via dagger? Should they checkout the centralized repository in their repo or is there any there suggested way?

I also would be really happy to get some suggestion how the folder structure looks like.

plucky ermine
#

When I'm running a service with up is there any way for me to enter the container that is running with terminal?

strange arch
#

Hello, I’m playing with interfaces in the Go SDK and I expected something like this to work:

package main

type Test struct {
}

type TestOption interface {
    DaggerObject

    Do(m *Test) *Test
}

func New(
    opts ...TestOption,
) *Test {
    m := &Test{}

    for _, opt := range opts {
        m = opt.Do(m)
    }

    return m
}

But it doesn’t seem to be the case:

❯ dagger call help
✘ initialize 1.6s
  ✔ ModuleSource.asModule: Module! 1.1s
  ✘ Module.initialize: Module! 0.3s
    ✘ exec go build -o /runtime . 0.3s
    ┃ # dagger/test                                                                                                                                                                                                                           
    ┃ ./dagger.gen.go:598:19: (&Test{}).WithGraphQLQuery undefined (type *Test has no field or method WithGraphQLQuery)                                                                                                                       
Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "test" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1

Stderr:
# dagger/test
./dagger.gen.go:598:19: (&Test{}).WithGraphQLQuery undefined (type *Test has no field or method WithGraphQLQuery)

Is it planned to make this kind of thing work in the future (would be awesome!)?

#

More on the use case: the idea would be to have something like this

container := dag.RedhatUbi(dag.Nodejs().RedhatUbiConfiguration(), dag.Golang().RedhatUbiConfiguration()).Container()

to get a container based on RedHat UBI with Node.js and Go installed and configured (with mounted cache volumes for example), all of this in the more efficient way. That is, installing all the packages at once to limit the number of layers in the image for example.

sonic vessel
rocky harbor
#

When I'm running a service with up is

#

Hello, I’m playing with interfaces in

#

Hi All,

austere latch
#

Ahoy! Someone I know heavily suggested me to come by 🤣

austere latch
#

Will try to make my very very first dagger module

#

Cross fingers 🤞

sharp zealot
#

Welcome 🙂

sharp zealot
#

@thorn moat @rocky harbor @deft rain @kind carbon a not-urgent but pretty fundamental topic we've discussed in the past: whether to allow "upstream modules" as a first-class citizen.

An "upstream module" being a module embedded with upstream software, which it can implicitly access.

In other words, should we consider cautiously re-opening the door we closed when disabling "root": and host().Directory() ?

restive shore
#

default OTEL env vars

restive shore
#

I am running into something strange. I can't figure out a workaround. I have a container that has a service attached (basically docker-in-docker). Everything from that container runs fine.

When I try to pull a file out of that container and use it in another container, that other container also tries to look up the service of the first container and fails with a 125

host alias: lookup t6sjv1ru3serq on 10.87.0.1:53: no such host
  ┃ lookup t6sjv1ru3serq.c1mh4mk35essm.dagger.local on 10.87.0.1:53: no such host

This may be related to - https://discord.com/channels/707636530424053791/1220265352030584834. I am doing all of this within a function. Is there a workaround?

#

Basically run (container A + service).

file = get file from (container A + service)

Try to run (container B + file) fails

warped canyon
#

Basically run (container A + service).

sharp zealot
sharp zealot
#

@rocky harbor is there an issue for the "can't compose multiple calls from the CLI" problem? ie. can't pass an ID as argument etc

rocky harbor
# sharp zealot <@949034677610643507> is there an issue for the "can't compose multiple calls fr...

This one is the closest, but the title is technically slightly more specific than that https://github.com/dagger/dagger/issues/6710

GitHub
CLI: don't hide `id` from core types · Issue #6710 · dagger/dagger

Problem When calling a function that returns a core type, like Container or Directory, I can then call any function from that core type... except id. This makes it unnecessarily difficult to stitch...

sharp zealot
restive shore
#

It took me way too long to figure this out. When you are creating a daggerverse within an existing go project, and you create a new module and move things out of the "dagger" subfolder (not a fan of that default btw), it modifies the root directories go.work to add the module to be under mymodule/dagger. But init and develop both broke for me until I realized and fixed my go.work. This is probably worth documenting.

sharp zealot
#

Also: agree that the ./dagger default subfolder is not working out.

sharp zealot
deft rain
quick wind
kind carbon
warped canyon
deft rain
#

(of course, it could be much more subtle)

sharp zealot
warped canyon
sharp zealot
#

Nice 🙂 We should start a "cool database modules" collection, I have my clickhouse one too

#

Is there a sqlite module yet? I feel like that could be particularly awesome since you get "database forking" for free

warped canyon
#

I'll make that one next if nobody else does by the time I get to it!

sharp zealot
#

I wonder if some real-world data management pipelines could be made 10x easier with the combination of a postgres +slqite module. Basically you'd handle export/import to/from the postgres DB; then instead of carrying a raw SQL text file, you'd carry it as a sqlite DB, and make arbitrary changes along the way - then re-export to postgres on the other end

#

(this assumes there is excellent sqlite/postgres bridges out there, but I'm pretty sure there are)

#

@warped canyon while we're showing off, I'm in the process of spinning out parts of my daggerverse repo into standalone repos (getting fed up with the go.work issues)

#

So I figured I'd make a reusable Function for spinning out a subdirectory of a repo into its own repo 🙂

warped canyon
sharp zealot
#

How do I cleanly check if a directory exists again?

upbeat herald
#

Hey! For those who use Dagger to manage CI or local dev of applications that have environment stored in .env or .envrc and struggle exporting all variable, I made a super simple module that can help you:
https://daggerverse.dev/mod/github.com/quartz-technology/daggerverse/magicenv@627fc4df7de8ce3bd8710fa08ea2db6cf16712b3

The idea is simple -> load the environment from your file
Example in Typescript:

this.ctr = dag.magicenv().loadEnv(this.ctr, { path: ".envrc" });

It checks on .env by default and this supports env expansion (e.g Foo:$BAR:HA)

For a more complete use case of this module + how to deploy local infrastructure with dagger, you can check a demo I published there: https://github.com/quartz-technology/daggerverse/tree/main/demos/bun-server

I hope this helps some of you 😉

magicenv :: Daggerverse

Magicenv provides utility functions to inject environment from various sources.

GitHub
daggerverse/demos/bun-server at main · quartz-technology/daggerverse

Daggerverse modules made by Quartz. Contribute to quartz-technology/daggerverse development by creating an account on GitHub.

#

I'm also super open to comments for improvement, feel free to ping me or open an issue on my repo

warped canyon
sharp zealot
kind carbon
sharp zealot
restive shore
#

Potential bug?

If I return a *Container from a function that has a WithExec that runs a long running/blocking process (like a server) and I do a terminal in the CLI, I have no way to exit out of it. q or esc does not work. I have to kill the terminal.

agile saddle
#

Also depending on the use (eg creating in private a container filesystem namespace is safe b/c of copy write semantics), but on the host or in shared cache volume, you actually want to check if the directory exists and atomically create it a the same time with mkdir or tmpdir because it avoids a time of check vs time of use bugs. In both cases these functions set errno to E_EXISTS to show the directory exits

restive shore
#

When I do a dagger up. The TUI doesn't show the full log even with --debug. I have an entrypoint script that does a lot of work. That script outputs stuff to stdout but I can't see any of that in dagger up. How do I get to that data? It's pretty important when troubleshooting the script.

sharp zealot
restive shore
#

oh no I said dagger up because it's succint 🙂 I am using dagger call ... up. Same behavior

sharp zealot
#

Ah!

restive shore
#

And I am on 0.10.3

sharp zealot
#

The entrypoint script is what the service itself executes?

restive shore
#

yeah.. It does a bunch of bootstrapping and then runs a java app (service)

sharp zealot
#

Might be worth filing an issue- maybe service start maybe doesn't stream standard output the way it should?

#

might be worth splitting out the bootstrapoing into separate withExecs in the meantime?

restive shore
#

hmm... You mean instead of running the script, do the work in the script via withExec? Right now I am running the script with withExec (as I don't know if there's an automatic way to run the entrypoint)

#

It's quite a big script. I don't really want to translate all that into dagger code yet

#

Even if we put the entrypiont aside, I can't see the running app's log. I only see a continuous "tail" of it.

#

When I exit out of the service, it truncates the historical stdout

sharp zealot
#

Showing off my new git module 🙂

This used to be the most annoying git task for me: spinning out a subdirectory of a repo, as its own repo, with history properly translated. It is now a simple dagger call 🙂

# Let's spin out the dagger docs as their own repo
dagger call -m github.com/shykes/git \
 clone --url https://github.com/dagger/dagger \
 subdirectory --path docs \
 inspect

And that's it!

#

To spin out any module from your daggerverse repo into its own repo:

REPO=https://github.com/shykes/daggerverse
MOD=tailscale

dagger call -m github.com/shykes/git \
 clone --url=$REPO \
 subdirectory --path=$MOD \
 checkout \
 export --path=./$MOD
restive shore
#

This is probably a really silly question. I want process something in a file. But I want to do it using Go modules (like process env vars with dotenv). How do I load that file into my function? Or access it with Go? If I load it as a *File, do I mount it into a container and then export it? Feels very cumbersome. I'm probably missing something basic. Is there a location in CurrentModule where I have access to that file?

sonic vessel
#

This is probably a really silly question

sharp zealot
#

I was just randomly trying a module from daggerverse, and got a weird error:

$ dagger call github.com/purpleclay/daggerverse/ponysay@v0.1.0
✘ initialize 0.0s
Error: failed to get configured module: no dagger.json found in directory . or any parents up to git root

There is a dagger.json, and daggerverse crawled it

This is with Dagger 0.10.3. Anyone else able to repro?

warped canyon
#

Did you miss the -m?

sharp zealot
#

Copied the dagger install line from daggerverse, replaced install with call

warped canyon
#

Should we have another copy option for call? I do this a lot too

sharp zealot
#

Well this is why I want dagger install to work all the time

#

So we can all build the muscle memory to install first, always

#

(or almost always)

sonic vessel
#

I see we can get function description via dagger functions but how to get description of the module from the cli? It maybe a stupid question, but I can't find it. T_T

sharp zealot
sonic vessel
void widget
sonic vessel
quick wind
#

Big difference in Gitlab CI in terminal output from 0.10.3 -> 0.11.0, much prefer the new, cleaner output 👍

wintry prism
restive shore
#

With 0.11.0 I am not seeing any of the docker pulls. Starting with the new engine pull. I see some references to http GET but that's it.

#

remotes.docker.resolver.HTTPRequest is what I'm seeing

wintry prism
#

or TUI?

restive shore
#

TUI

wintry prism
#

cc @thorn moat 👆

thorn moat
#

we'll probably send PRs upstream to bring back all that goodness

restive shore
#

Ah ok makes sense. Not a huge deal but I will miss those till they are back. They also made it nice to demo dagger to others 🙂

naive mason
#

Hey just kind of curious on like overall guidance on using Dagger.. it feels like the dagger functions was a pretty big shift. I felt like before dagger functions you'd have a simple dagger program, likely in the language of your project, that would pipeline together the necessary steps to build your app. Now post dagger functions, it feels like the docs are pushing you more for either using one or more dagger call to run some module functions. Is that the future direction or is the older style programs that the docs had pre-functions still a good pattern?

wintry prism
# naive mason Hey just kind of curious on like overall guidance on using Dagger.. it feels lik...

Hi Ian 👋 today I'd use functions, but you can use them in basically the same way.
Write a simple Dagger function for CI in the language of your project that pipelines steps together.
You kick off your CI function with dagger call like you used to with dagger run go run or dagger run node or dagger run python (don't need those runtime tools on your machine anymore).

You can call other functions in code from your CI function from the Daggerverse or that you write yourself to get the best combo of composition, abstraction, DRY, re-use (you can call functions written in any Dagger SDK language, btw. That's new!).
This workshop (from today) shows how to do that really well:
https://www.linkedin.com/events/7170662442614636544/
https://github.com/kpenfound/kubesimplify-dagger-workshop
https://www.youtube.com/watch?v=1vHrf7-D3oI

sharp zealot
#

What @wintry prism . Calling functions with dagger call is the recommended way. You can still embed a dagger client library in your own custom tool, and execute that on the host machine, but we consider that to be a niche use case. Most software teams don't actually want to develop a custom CLI tool just to run their pipeline: they would prefer to develop the individual building blocks, and use the dagger CLI to call them. Less boilerplate to maintain, less work for the user.

naive mason
restive shore
#

On 0.11.0, every time I run the TUI I see this 7:23 WRN failed to get commit object err="object not found" . It's persistent at the bottom of the TUI console. Didn't see that before.

sharp zealot
#

Sometimes you have the luxury of Daggerizing almost everything all at once. Sometimes, it might be a gradual process stretching years.

#

The point is: every step of the way, you decide how much glue script is involved. The important part is that the trend is reversed: instead of more glue accumulating over time, now things get simpler over time.

#

Concretely, the lifecycle of your dagger call glue will be:

  1. Ha ha, I can replace 100 lines of awful glue with just 1 line: dagger call. Dagger rules!
  2. OK, my dagger call pipeline is getting a little longer, I'm chaining a few calls together. 10 lines.
  3. This is stupid, I'm writing a whole shell script just to wrap dagger call. Time to write a new function with the new logic!
    -> You just expanded the boundary of your DAG. Repeat the cycle with the next function.
sharp zealot
#

@warped canyon your demo this morning gave me an idea for a new blockbuster module 🙂

warped canyon
sharp zealot
#

Will keep it a surprise for now 😛

wintry prism
quick wind
shy ledge
#

@quick wind - There are few Trivy modules.

quick wind
#

I just threw together the simplest kics module + test file, it's only for demonstration purposes

rocky harbor
#

Having modules office hours for the next hour in the dev-audio channel for anyone who wants to join!

plucky ermine
restive shore
rocky harbor
# restive shore same! I need to put a recurring meeting on my calendar. Keep missing it.

In case you were gonna ask about CA certs+proxies, that's been my main focus and I'm fairly hopeful the next release will have enough baseline fixes that should unblock you 🙂 It's been taking way longer than expected due to the fact that CA certs are a never ending hellhole of different cases to handle, but I think we'll be able to support enough common cases to start to cover the majority 🤞

restive shore
rain hedge
#

I would like to develop some modules via TS but folder structure a bit confused to me. Should I really need dagger folder? For instance i'd like to develop a module for docker with scan and sign so should folder structure looks like below?
./docker/dagger/src/index.ts or is it also ok to have ./docker/src/.index.ts?

wintry prism
rain hedge
#

Btw, what might be the reason of dagger call takes too long? Am I doing something wrong? :/

image.png
wintry prism
restive shore
#

is --progress=plain supported with dagger call ... up? It's stuck for me in Connected to engine .... Dagger v0.11.0

thorn moat
restive shore
#

ah.. for me it doesn't look like it's even running. For the service up use case that is. It's just stuck. I gave it plenty of time.

quick wind
#

Any info on the roadmap for Daggerverse publicly available? With the update from 0.10.3 -> 0.11 it's even more important that a module's engine version is displayed/filterable. Other things off the top of my head: filter criteria: partner/community/dagger-created modules, sort on publish/update date

naive mason
#

Anyone have some good examples of unit testing their dagger functions?

wintry prism
#

Any info on the roadmap for Daggerverse

plucky ermine
#

Are we not able to pass in dir as a constructor?

I tried this but the CLI yells at me

@object_type
class MyClass:

    dir: Annotated[dagger.Directory, Doc("Directory containing source code")]

    @function 
    def ci(self) -> str:
        """Run entire CI pipeline"""
        # run tests 
        return self.test(self.dir).stdout()

then

dagger call ci --dir=. 

((.venv) ) levlaz@Levs-MacBook-Pro fedidevs % dagger call ci --dir=.
✘ initialize 3.2s
! required flag(s) "dir" not set
  ✔ Module.initialize: Module! 1.0s
    ✔ exec /runtime 1.0s

Error: required flag(s) "dir" not set
amber pilot
thorn moat
amber pilot
amber pilot
#

Any idea about to fix the color lost issue in CI?
I feel lost light when upgraded to dagger v0.11.0 =。=

thorn moat
amber pilot
thorn moat
#

it's possible that env var was ignored by the old UI, so just want to confirm it's not set; that should be the only way to disable colors, unless there's a bug (also likely)

amber pilot
#

so when is not tty, here to force drop color?

amber pilot
#

"CI=true" it may the issue

func (o *Output) isTTY() bool {
    if o.assumeTTY || o.unsafe {
        return true
    }
    if len(o.environ.Getenv("CI")) > 0 {
        return false
    }
    if o.TTY() == nil {
        return false
    }

    return isatty.IsTerminal(o.TTY().Fd())
}
#

Need to NewOutput with termenv.WithUnsafe()

#

color is back

image.png
rain hedge
# wintry prism It could be that your `docker ` directory has many MBs of files being copied int...

@wintry prism is this what you need?

8.0K    ./docker/sdk/context
 28K    ./docker/sdk/provisioning
 52K    ./docker/sdk/common/errors
 56K    ./docker/sdk/common
4.0K    ./docker/sdk/introspector/decorators
4.0K    ./docker/sdk/introspector/utils
8.0K    ./docker/sdk/introspector/registry
 28K    ./docker/sdk/introspector/scanner/abtractions
 44K    ./docker/sdk/introspector/scanner
 60K    ./docker/sdk/introspector
4.0K    ./docker/sdk/graphql
 24K    ./docker/sdk/entrypoint
200K    ./docker/sdk/api
412K    ./docker/sdk
4.0K    ./docker/src
436K    ./docker
 40K    ./docs
 76K    ./.github/workflows
 12K    ./.github/ISSUE_TEMPLATE
100K    ./.github
4.0K    ./.git/objects/3b
4.0K    ./.git/objects/04
152K    ./.git/objects/pack
4.0K    ./.git/objects/7e
  0B    ./.git/objects/info
8.0K    ./.git/objects/13
172K    ./.git/objects
4.0K    ./.git/info
4.0K    ./.git/logs/refs/heads/feat
8.0K    ./.git/logs/refs/heads
4.0K    ./.git/logs/refs/remotes/origin
4.0K    ./.git/logs/refs/remotes
 12K    ./.git/logs/refs
 16K    ./.git/logs
 64K    ./.git/hooks
4.0K    ./.git/refs/heads/feat
8.0K    ./.git/refs/heads
  0B    ./.git/refs/tags
4.0K    ./.git/refs/remotes/origin
4.0K    ./.git/refs/remotes
 12K    ./.git/refs
292K    ./.git
908K    .
wheat river
#

Hey guys, is there a way to capture the stdout but avoid exposing it as part of the logs shown when a given command (WithExec()) is being evaluated?
Taking this snippet as an example

    out, err := dag.Container().
        From(image).
        WithMountedCache("/go/pkg/mod", mod).
        WithMountedCache("/root/.cache/go-build", build).
        WithMountedCache("/root/.terraform", dotTerraform).
        WithExec(t.getTFInstallCMD(tfVersion)).Stdout(context.Background())

If I run it, it's going to show the standard output, but what I'd like to do is to capture it, and show this one and others later on, in a sort of summarised view.

warped canyon
wheat river
amber pilot
#

How to remove those resolver logs? (when plain mode)

image.png
wintry prism
#

cc @thorn moat

restive shore
plucky ermine
#

Imagine a I have a function called cli that is a "proxy" for a CLI that sends string commands to some CLI that I am wrapping and returns a *Container.

Is it possible to chain things like with-directory before executing the cli function? I basically want to pre-pend a few steps before the function executes but I don't see any way to do that.

kind carbon
#

Can you put those steps in another function?

wheat river
#

Hey guys, what's your experience debugging Dagger functions in something like Goland? (let's assume we're using mostly the Go SDK, and we prefer to code in Goland — my case).
I'm curious since there's a lot of code-generation, and the actual code that performs the magic is mostly generated, hence placing a breaking point in any part of the 'wrapper' code doesn't work. Going into the generated code and creating a breaking point there isn't a good Developer Experience. I'm looking for experiences and perhaps a better approach that works while developing and debugging functions. The same question applies for modules.

sharp zealot
#

debugging a Go function

strange arch
#

Hello, is there a way to simplify the following code by directly returning the Installed function as an anonymous function from the Packages function?

type RedhatMinimalPackages struct {
        Names []string
}

func (*RedhatMinimal) Packages(names []string) *RedhatMinimalPackages {
        packages := &RedhatMinimalPackages{
                Names: names,
        }

        return packages
}

func (packages *RedhatMinimalPackages) Installed(container *Container) *Container {
        return container.WithExec([]string{"microdnf install --nodocs --setopt install_weak_deps=0 --assumeyes " + strings.Join(packages.Names, " ") + " && microdnf clean all"})
}
sharp zealot
strange arch
sharp zealot
#

Simplify code with custom type

remote magnet
#

Hey there, with the big push with Dagger Modules and Functions is the "legacy" (< v0.10.0) way of defining Dagger Pipelines/DAGs (e.g. any examples here) still be supported in the long term? Is the goal to deprecate it?

kind carbon
sharp zealot
noble oracle
kind carbon
noble oracle
#

Thanks for the clarification!

kind carbon
#

On one hand, code in dagger run can't run modules using dag from the larger modules ecosystem. On the other, dagger call has a more stricter access to host resources. Having good support in dagger run scripts to call module functions in order to bridge that gap is high on the list of things to fix, but I hope people try out modules and give it a chance.

sharp zealot
#

Yeah it's fair to say: in doubt, use dagger call. But no plans to deprecate run.

And as @kind carbon said, we want to add a built-in scripting feature. If we do a good job, that may end up being everyone's favorite way to run dagger 🙂

bronze lance
#

When coming back to Dagger, I did get thrown for a loop by everything being Function-centric. I jumped straight to the docs to refresh myself on how to build a full CI pipeline with dagger, but felt a bit confused. Maybe because I was used to the old way dagger worked the last time I played with it. I thought it was replacing the CI provider's whole bespoke engine, but this time around it seemed more like it was replacing just individual bash scripts or commands within the engine(?). I no longer just invoke dagger once and let it drive the whole process, but call many individual modules/functions from the CLI serially? So if I want to parallelize the work and perform frontend build, backend build, and test coverage tasks all at once (what would normally be scripts all in the same gitlab CI stage), I should define a single CLI-callable function for this? And rather than dagger run, for some other less savvy user to execute it, I should provide a wrapper script to abstract away the individual function calls? What's the recommended practice here?

I'm rubber ducking more just by typing a lot of stuff out, and I can't quite put my finger on what I didn't like about the quickstart docs just yet. Something about not starting at the big picture with a whole non-trivial pipeline demonstration from which I could anchor my understanding of the use of functions, maybe. Or maybe that some of the examples were just toys (like cowsay) that I have to translate to actually useful alternatives in my head to understand the benefit. I may come back and say more if I figure that out. In the meantime, I'll be trying to put together at least part of a pipeline I use at work 👀

sharp zealot
sharp zealot
bronze lance
sharp zealot
# bronze lance I'm sure I'll grok it better once I've dug into it sufficiently. I definitely se...

Yeah exactly 🙂 The transition hasn't been perfectly smooth, especially in the docs which we basically rebooted. Sorry about that. But I think it's worth it, the possibilities are truly endless now.

Also note that the "old" way still works: you can still write a custom Dagger client, and run it directly on your host machine. We think that will become more of a niche use case, as most people don't actually want to write a custom tool from scratch just to have good CI. But on the other hand, if you already are developing a custom tool, we want to make it very easy for you to embed dagger functions into it.

warped canyon
#

Dockerd services issue

bronze lance
#

Yeah exactly 🙂 The transition hasn't

bronze lance
#

In the docs here: https://docs.dagger.io/manuals/developer/go/882081/module-structure#file-layout

It says that dagger init should create a dagger.json and all the regular go files in the same directory. However, when I use that command, I get a directory by the name I expected, and inside that there is dagger.json and dagger/. Inside that dagger/ subdirectory I then find my regular go files. Should these all be one level up with the dagger.json presumably generated for the module? Or should every dagger module generated with init just contain dagger/ dagger.json at root level?

Module Structure | Dagger

It's essential to understand a few key concepts about Dagger Modules created for use with the Go SDK, for a better fit with your normal development workflow.

sharp zealot
sharp zealot
sonic vessel
#

I just see that the telemetry folder in internal didn't get ignore. Is that intention?

Screenshot_2024-04-21_164344.png
sonic vessel
sharp zealot
#

Do we have a primitive for unpacking a tarball, or generally interacting with tarballs in any way?

#

Use case: I have a tarball url, and want a directory. Trying to figure out how much I need to do myself vs. using core API

sharp zealot
#

Also: I would love a core function that gives me an interactive terminal on a directory...

#

very common that I want to quickly inspect the result of a function that returns a directory, and have to manually orchestrate setting up a debug container with that directory mounted, so I can call terminal... I would prefer for the engine to just have a call to do that

sharp zealot
sharp zealot
deft rain
plucky ermine
#

I did something that felt really dirty today to work around the lack of host environment knowledge.

This works for Alpine, but Im sure there is a version for all distros.

alpine can load stuff from /etc/profile.d/script.sh whenever you have a sh --login shell, so if I pass in my entire local .zprofile file into the function I have access to all my local environment variables

func (m *MyModule) Run(env *Secret) *Container {
    return dag.Container().
        From("alpine:latest").
        WithMountedSecret("/etc/profile.d/env.sh", env).
        WithExec([]string{"sh", "-c", "--login", "echo $SECRET_FROM_PROFILE"})
}

dagger call run --env file:/Users/levlaz/.zprofile stdout

Note that this would not work, because os still refers to the engine and not the container runtime. So its still only half way to being as nice as the previous way things worked.

WithExec([]string{"sh", "-c", "--login", "echo", os.Getenv("SECRET_FROM_PROFILE")}

At the end of all this, I am more convinced than ever that I really should use Infisical for this locally and in CI 🙃

latent trellis
latent trellis
restive shore
#

Seems like a good way to organize tests. Maybe eventually dagger develop can generate a sample test like this

latent trellis
wheat river
latent trellis
wheat river
#

I think that falls into the grey area of functions/modules. For instance, I have modules that are very small and few of them are just exposing one or two functions. Those functions on itself are small enough to be considered as a single unit, hence a test for that specific function makes sense in that context. A more integrated API as part of the SDK(s) will be great to facilitate the unit/other sort of tests. It's kinda related to this discussion here: #daggernauts message

thorn moat
sharp zealot
thorn moat
thorn moat
sharp zealot
#

Auto-complete in Go IDE

sharp zealot
#

Actually @thorn moat - more broadly, the generated Go bindings were the problem for me. They make those arguments mandatory.

thorn moat
#

yep

#

it's because from a codegen POV the presence of a default has no bearing on that, which is wrong imo. instead it relies on the type being marked optional, which isn't necessary for an arg to actually be optional in GraphQL, and isn't really what you want, because you don't want to have to deal with a null value

#

I'll comment on 6749 to check what the resolution is, i'm not completely sure based on the convo

plucky ermine
#

New testing strategy for daggerverse rep...

sharp zealot
#

If I understand the issue you linked, the problem is that +optional is actually implemented to mean "nullable"

#

leaving everyone unhappy: both people who care about nullable, and people who don't

thorn moat
sharp zealot
#

I don't have a better suggestion. Just observing that people are getting consistently confused, which is a design problem

#

I think we're only partially reflecting how GraphQL works. To more fully reflect it, we would use the term "nullable" instead of optional. +optional would become +nullable. And "optional" is just what happens when you either have a default value, or a nullable value.

#

I'm not saying we should do that necessarily

#

But at the moment we define the word "optional" differently from the GraphQL spec

thorn moat
#

yeah, nullable vs optional conflation has come up before. it also kind of ties in to type inference; you could imagine in Go something is implicitly optional if it's a pointer. but we can't do that because all objects are pointers regardless

#

I think we do do that for pointers to primitive types like string but not sure

#

it seems like what has happened in practice is individual GraphQL server implementations have taken it upon themselves to allow non-null-type args with default values

sharp zealot
#

There's another distinction at the GraphQL level, which is that "nullable" applies to any type, including return values, whereas "optional" (in its graphql definition of "either nullable, has a default value, or both") only applies to arguments.

thorn moat
#

yup yup

#

although - where is that actually defined in graphql? or is it just terminology?

#

afaik it's not part of the schema, fields just have a type and default

#

which is where the nullable confusion comes from

sharp zealot
thorn moat
#

the term "optional" regarding arguments specifically

sharp zealot
#

But as you say, they don't specify the case non-nullable+default-value

#

There's a section called "input coercion" in the spec that seems relevant

thorn moat
#

If the value null was provided for an input object field, and the field’s type is not a non-null type, an entry in the coerced unordered map is given the value null. In other words, there is a semantic difference between the explicitly provided value null versus having not provided a value.
that's interesting, since if you extend that to args, from the first example we referenced that implies you can pass a null value for unit: - what would that even do?

#

oh wait, it doesn't mention a default there

sharp zealot
#

Another clue in the introspection schema https://spec.graphql.org/October2021/#sec-The-__InputValue-Type:

The __InputValue type represents field and directive arguments as well as the inputFields of an input object.
defaultValue may return a String encoding (using the GraphQL language) of the default value used by this input value in the condition a value is not provided at runtime. If this input value has no default value, returns null.

thorn moat
#

nice

#

ok yeah that's basically how I want it to work, and how DagQL already works, it's just the codegen that's out of sync with it i suppose

#

glad it's actually defined in the spec

sharp zealot
#

Didn't even notice that, pretty cool

sharp zealot
thorn moat
#

isn't it just something like argIsRequired = !argType.Optional && !arg.Default? (to determine whether e.g. Go makes it a required arg or puts it in the Opts struct)

#

plus maybe renaming Optional to Nullable (or flipping it to NonNull to match GraphQL more closely)

#

since codegen is driven by an introspection schema JSON, I think each SDK would have to do that, unless we switch them to be based on our own typedefs, at which point we could expose a higher level check for that (like FunctionArg.isOptional)

sharp zealot
thorn moat
#

right, the semantics are unclean in that case, just because we're using that word

#

so we want something like argIsRequired = argType.NonNull && arg.Default == null then?

sharp zealot
#

Does it even make sense to have an explicit non-null marker though? Either it can be infered by the native type already (eg. a Go function that returns a string is for sure String!), or it can't.

thorn moat
#

that part is probably SDK-specific

#

that argIsRequired bit would be in codegen running against a generated schema though, so that decision is already made by the time we're generating code for it

#

for Go code => schema, I think you'd still need a comment to explicitly mark an object arg as optional (or nullable)

#

...unless we do some sort of cursed double pointer thing

sharp zealot
#

OK better late than never: I'm starrting a thread 🙂

noble oracle
#

Random ?: Is there a general recommendation or leaning on building and publishing containers - via feeding a Dockerfile in or going all-in on Dagger native to build a publish containers?

plucky ermine
#

Random ?: Is there a general

sharp zealot
#

I'm writing up a proposal that I think will make you happy @wintry prism

willow carbon
sharp zealot
shy ledge
#

Anything in combination with Dagger becomes friggin cool. But, yeah. Some tools solve similar problems as Dagger does. Dagger is just doing it better IMHO. 😄

latent trellis
wheat river
#

For more complex functions and/or modules, with the principle in mind that they should be composable and self-contained, I've been thinking in cases that require complex logic beyond simple commands in WithExec — currently, my workaround involves writing the logic in —let's assume I'm writing it in Go— a Golang program, creating a container image, and initializing the Dagger function with dag.Container().From(mycustomimage:latest). This process is cumbersome as it involves multiple steps.

I'd love to have an API similar to WithExec, perhaps something like WithCodeEntryPoint("main.go"), that allows running code directly in a pre-existing base image as it's currently possible nowadays (e.g., golang:alpine), without the need to create and push a new container image 🤔

Have you considered this scenario, and what are your thoughts on its feasibility given the current capabilities of Dagger functions?

quick wind
#

Contextual modules

willow carbon
#

I admit tho I did not think about a dagger module for docker and I have not used it, if anyone has a reference example for building containers and shipping to a registry I would be happy to see it

restive shore
# willow carbon I admit tho I did not think about a dagger module for docker and I have not used...

This is in the core API. https://archive.docs.dagger.io/0.9/quickstart/730264/publish. These core API docs needs to bubble up to the current module docs. You could create a wrapper module to publish something and I am doing that currently at my work (dagger -m module call publish). The module is just using the core API to build and publish. Unfortunately I can't share that code here but you get the gist.

You can call publish on any Container object that gets returned in a module.

willow carbon
#

awesome thanks for sharing!

sharp zealot
restive shore
willow carbon
#

is there a recommended solution for the CD portion of application delivery with dagger? I realize that is application dependent and the answer might be "whatever you want, Argo, Flux, etc"

bleak nest
#

@willow carbon @restive shore https://docs.dagger.io/manuals/user/577201/publish/ has an example of building and publishing a container with publish. I hope that helps? Thanks also for the feedback on the search...I checked quickly just now searching for "publish" as you did and the page linked above appears in the drop-down list of search results (4th in the list)

willow carbon
#

that is unbelievably cool. I love wolfi too.

sharp zealot
#

@chrome pilot I'm still digesting your idea that all modules could be contextual - ie. always look in .dagger 🤔

wheat river
#

Curoius, why a map isn't supported for the type inference when writing functions?
For example

type MyModule struct {
    Src *Directory
    CfgFile string
    // EnvVars map[string]string
    EnvVars map[string]string
}```
The `EnvVars` breaks the code-generation.

Error: generate code: template: alias.go.tmpl:74:3: executing "_dagger.gen.go/alias
┃ " at <ModuleMainSrc>: error calling ModuleMainSrc: failed to parse field type: unsu
┃ type for named type reference *types.Map```

sharp zealot
sharp zealot
latent trellis
spiral whale
#

is this code generation intentional? using pointer and non-pointer refs causing warnings by IDE.

func (r Gh) MarshalJSON() ([]byte, error) {
    var concrete struct{}
    return json.Marshal(&concrete)
}

func (r *Gh) UnmarshalJSON(bs []byte) error {
    var concrete struct{}
    err := json.Unmarshal(bs, &concrete)
    if err != nil {
        return err
    }
    return nil
}
image.png
latent trellis
spiral whale
latent trellis
spiral whale
deft rain
dense canyon
#

👋 Dagger devs! Just created a new proposal issue here https://github.com/dagger/dagger/issues/7217 with the intent to improve our Daggerverse modules quality by allowing module creators to write examples for their modules that will be surfaced in Daggerverse. Any feedback or ideas are always welcome. Please don't heistate to ask if you have any questions 🙌

GitHub
Improve module quality by examples · Issue #7217 · dagger/dagger

Problem In many cases it's not clear for module consumers how to use them once they find them in the Daggerverse. Even though https://daggerverse.dev has detailed docs about the module function...

sharp zealot
#

I didn't really understand the difference between "solution 1" and "solution 2"

dense canyon
dense canyon
# dense canyon I'll clarify the difference

the most important difference is that solution 1 will only translate the GraphQL queries made by the example code. So if the user writes something like:

func (e *JavaExample) ExampleJava() *Container {
    c:= dag.Java().WithJdk("17").
        WithMaven("3.9.5").
        Container().
        WithExec([]string{"mvn", "--version"})
        if "foo" == "bar" { fmt.Println("test") }
        return c
}

then the if statement won't be part of the generated example code since that is not part of the GraphQL pipeline query

#

as we're capturing the underlying graphql query from the returned *Container.ID(ctx) dagger.ID type

sharp zealot
#

yeah that doesn't seem worth it to me

#

Is the idea that it might be a shortcut for generating multi-language snippets?

dense canyon
sharp zealot
#

I may be wrong, but doesn't seem worth it

#

Non-trivial engineering, and limiting experience for the user.

#

Better to invest the engineering in gradually improving a LLM-based snippet translation IMO

#

ie. assume LLMs will do it perfectly. Then plan gradual steps from there. Don't invest any engineering in something that you'll need to rip out later when LLMs do it perfectly

#

Also support for CLI example is key (not sure how your current plan addresses that, but I'd be curious to learn more)

dense canyon
sharp zealot
dense canyon
sharp zealot
#

I liked the idea I saw floating around, of an examples/ directory with random files, which you can reference at will

#

But curious to hear other ideas as well

dense canyon
sharp zealot
# dense canyon the most important difference is that solution 1 will only translate the GraphQL...

First naive attempt at translating with GPT4 🙂

Python:

from dagger import dag

class JavaExample:
    def example_java(self) -> dagger.Container:
        c = (dag.java()
             .with_jdk("17")
             .with_maven("3.9.5")
             .container()
             .with_exec(["mvn", "--version"]))
        if "foo" == "bar":
            print("test")
        return c

Typescript:

import { dag, Java, Container } from '@dagger.io/dagger';

class JavaExample {
  exampleJava(): Container {
    const c = dag.java()
      .withJdk("17")
      .withMaven("3.9.5")
      .container()
      .withExec(["mvn", "--version"]);
    if ("foo" === "bar") {
      console.log("test");
    }
    return c;
  }
}
void widget
#

Hello everyone 👋

For the past few months, several discussions have occurred around the best way to allow other SCM than GitHub for modules and Daggerverse. This is now a top-priority on the backlog 👼.

As no perfect solution exists, your opinion is extremely valuable to assess what behavior you would expect as a user, especially as it might touch the developer experience

Feel free to comment / take a look: https://github.com/dagger/dagger/issues/7218 🙏

willow carbon
#

is there any webassembly modules?

void widget
latent trellis
spiral whale
#

OpenSSL cannot read SSH keys mounted as ...

willow carbon
dense canyon
# dense canyon 👋 Dagger devs! Just created a new proposal issue here https://github.com/dagg...

👋 just updated my proposal with some feedback from our previous interactions @sharp zealot and also added a new UX section with an idea of how that could work. Would love some 👀 if anyone has the time 🙏 https://github.com/dagger/dagger/issues/7217#ux

GitHub
Improve modules quality by examples · Issue #7217 · dagger/dagger

Problem # In many cases it's not clear for module consumers how to use them once they find them in the Daggerverse. Even though https://daggerverse.dev has detailed docs about the module functi...

sharp zealot
#

Experimenting with a possible scripting DX: embed a bash-compatible interpreter in dagger, bind it to dagger without argument. Basically making dagger a repl.

Even with bash-compatible syntax, command execution would be resolved dynamically based on the current pipeline. So, we could take advantage of the familiar pipeline syntax:

dagger install github.com/shykes/git github.com/shykes/daggerverse/wolfi

#!/usr/bin/env dagger

base() {
 wolfi | container --packages git,openssh,go | without-entrypoint
}

repo() {
  git | clone --url https://github.com/dagger/dagger | checkout --ref=v0.11.2 | directory
}

container() {
  base | with-mounted-directory --path=/src --source=$(repo) | with-workdir --path=/src
}

ctr | terminal
sharp zealot
sharp zealot
#

Feedback on latest Traces UI changes:

  • I love that the "error path" is highlighted
  • I find it confusing that the error path is shown twice: first in the special "errors" section, then again in the "general" view.
  • It would be better IMO to collapse the two: only show one view (the usual spans view), but when there is an error, collapse everything except the error path. This would get the best of both worlds
#

Also the structure of the call tree is less clear, there is more noise:

  • "Spans" repeated each time: noisy
  • "Logs" and "Spans": there's both. What is their relationship? Logs are shown first, so were they chronologically first? Or are logs and spans interleaved, but if so why not render them as interleaved?
  • Getting mixed signals on the hierarchy: are "Logs" and "Spans" nested inside the clone() call? They're not inside the call's "box". There is a branching line that says maybe yes; but the branch happens after the clone() box, so maybe it's unrelated? Confusing. Multiply by each span, and add the "duplicate error path", I no longer am confident I know what's going on
Screenshot_2024-05-01_at_12.36.29.png
latent trellis
#

@sharp zealot what's the rationale behind keeping git state and workdir in separate directories in your Git module?

sharp zealot
latent trellis
#

Apparently there is no WithoutSecretVariable and WithoutEnvVariable doesn't clear those env vars. Is that intentional?

latent trellis
#

Quick fix: WithSecretVariable("VAR", dag.SetSecret("reset", ""))

warped canyon
#

That sounds like an oversight to me, I can create an issue

sharp zealot
#

Yeah oversight

warped canyon
sharp zealot
latent trellis
spiral whale
latent trellis
#

Dagger just started randomly doing this without me changing anything:

📎 message.txt
wintry prism
#

Quesion on rendering of comments today and examples in future in modules from @acoustic forge

So if you see the code I do have the function descriptions written here: https://github.com/okteto/dagger-module/blob/main/dagger/main.go#L44-L47 but ever since I add a gap between the description and the example command it stopped showing the description. My problem is that if I don't add that gap line then it renders the description and example in the same line (see screenshot)

// Deploys a preview environment in the specified Okteto context

// example usage:
// dagger call preview-deploy --repo=https://github.com/RinkiyaKeDad/okteto-dagger-sample --branch=name-change --pr=https://github.com/RinkiyaKeDad/okteto-dagger-sample/pull/1 --context=yourinstance.okteto.com --token=$OKTETO_TOKEN
func (m *OktetoDaggerModule) PreviewDeploy(ctx context.Context,
    // Repo to deploy
    repo string,
...
Screenshot_2024-05-09_at_12.png
GitHub
dagger-module/dagger/main.go at main · okteto/dagger-module

A Dagger Module for Okteto. Contribute to okteto/dagger-module development by creating an account on GitHub.

#

cc @dense canyon @void widget

latent trellis
#

Has anyone seen this?

Failed to connect; retrying... make request: Post "http://dagger/query": rpc error: code = Unknown desc = failed to verify client: client ID "***" registered with different secret token

Happens after upgrading to 0.11.3

deft rain
rocky harbor
#

Ah wait, I saw the registered with different secret token error message now, that is possibly related to some changes. I'll see if I can repro

latent trellis
#

I figured it out in the meantime (setting the same secret in parallel) and it kinda makes sense that it doesn't work, but I wouldn't expect Dagger to hang.

rocky harbor
# latent trellis I figured it out in the meantime (setting the same secret in parallel) and it ki...

Yeah that will coincidentally avoid the problem, but I think I know the underlying cause and it's a bit more generic than just setting secrets.

Fortunately the fix is pretty straightforward and already part of something else I was working on, so I'll just extract that out and we can get it in for the release tomorrow. cc @jed funnily enough this is a problem that ftp_proxy unintentionally masked but now was revealed by removing that hack lol. Thankfully we have better ways of fixing it now via all the recent improvements, don't need to go back to that 😮‍💨

rocky harbor
#

Fix here: https://github.com/dagger/dagger/pull/7335 The fix was what I was imagining originally but while testing some more I did realize that we aren't getting cache hits as expected when providing args of type Secret to functions, which is quite odd, but a separate issue (more details in the commit message). The fix there is still good to go either way.

rocky harbor
#

Ah, I may have found the problem with the caching, @thorn moat these OTEL values appear to be random every time the same function is invoked with the same inputs: https://github.com/sipsma/dagger/blob/47a8a80c1109b1032cd167f011bda3aafda36904/core/modfunc.go#L209

So they end up invalidating the buildkit cache every function call. Fortunately dagql caching masked most of that, but if there's an impure ID (such as Secret) then we don't hit dagql caching and get buildkit cache misses.

Fortunately the new way of passing random IDs without busting cache that I also used in the above PR should be perfect for this too, just FYI in case it's not expected those values are different every time.

rocky harbor
sonic vessel
#

The Daggerverse use markdown format to render function doc on the web?

wintry prism
sonic vessel
wintry prism
#

@void widget it might be in the comments, right?

void widget
latent trellis
willow carbon
#

does dagger utilize gpu's when running modules?

thorn moat
wintry prism
latent trellis
sharp zealot
latent trellis
#

But now that you mention it... dagger install -u would be easier 😄

latent trellis
kind carbon
wintry prism
#

@kind carbon a user today with hatch was having trouble with his VS Code IDE with the pip install -e ./sdk instructions.... not sure if there is something special he needs

kind carbon
dense canyon
warped canyon
#

This looks like it's with trivy and not dagger, unless i'm missing some context 🙂

celest osprey
sharp zealot
#

Trying out a random idea: how would you feel about a dagger embed command, to help you embed dagger functions in your own native software project? Basically the "classic" use case, but legitimized by a clear framing ("embedding dagger in your code") and a proper UX (dedicated subcommand).

Things you could do with dagger embed (and its subcommands, tbd):

  • Manage the installation of a Dagger client library for your language
  • Manage generation of native client bindings for dagger modules you wish to call

Thoughts?

If the discussion yields anything, I'll write it up in an issue

spiral whale
#

Trying out a random idea: how would you

spiral whale
#

I'm currently trying to gale to v0.11.x and trying some non-sense combinations with new features. But it's too hard to find what is breaking to code. Is there any better way to diagnose what is breaking my code?

dagger develop
✘ ModuleSource.asModule: Module! 0.4s
! failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "/usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json" did not complete successfully: exit code: 1
  ✘ Module.withSource(
      source: ✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
    ): Module! 0.4s
  ! failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "/usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json" did not complete successfully: exit code: 1
    ✘ Container.directory(path: "/src"): Directory! 0.3s
    ! process "/usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json" did not complete successfully: exit code: 1
      ✘ exec /usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json 0.3s
      ! process "/usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json" did not complete successfully: exit code: 1
      ┃ generating go module: gale
      ┃ writing dagger/gale/dagger.gen.go
      ┃ writing dagger/gale/go.mod
      ┃ writing dagger/gale/go.sum
      ┃ running post-command: go mod tidy
      ┃ running post-command: go work use .
      ┃ needs another pass...
      ┃ Error: generate code: template: alias.go.tmpl:76:3: executing "_da
      ┃ .go/alias.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc
      ┃ ed type spec to be a struct, got *ast.SelectorExpr

Error: failed to get module SDK: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: process "/usr/local/bin/codegen --output /src --module-context-path /src/dagger/gale --module-name gale --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stdout:
generating go module: gale
writing dagger/gale/dagger.gen.go
writing dagger/gale/go.mod
writing dagger/gale/go.sum
running post-command: go mod tidy
running post-command: go work use .
needs another pass...
Stderr:
Error: generate code: template: alias.go.tmpl:76:3: executing "_dagger.gen.go/alias.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: expected type spec to be a struct, got *ast.SelectorExpr
latent trellis
#

I find it kinda weird that I have to name secrets: dag.SetSecret("name", "value").

Is there a way to skip that?

It makes APIs kinda bloated when a function creating a secret is called repeatedly.

sharp zealot
# latent trellis I find it kinda weird that I have to name secrets: `dag.SetSecret("name", "value...

It's been a while, but that API was the result of a long and relatively painful design process. Here's the context: https://github.com/dagger/dagger/issues/4426

I would have to refresh my memory on how we landed here, and whether we learned anything new since, that would justify changing.

One thing I do remember, is that we thought of this API as an intermediary step towards pluggable secrets providers. Having a caller-supplied secret name would give a path to this future design

sharp zealot
latent trellis
#

From what I can tell, what's new since is the introduction of functions. Modules can provide an API for accessing secret management solutions.

#

I'd also argue that with the introduction of functions, the current design just became a lot more confusing.

For example: what if I have a function that creates secrets? Can I reuse the name? Do I have to expose an option and make sure each secret has a unique name.

#

The alternative is generating random UUIDs I guess?

#

But that might mess with caching.

sharp zealot
#

I agree that we should revisit the API in the context of functions.

sharp zealot
latent trellis
#

I mean, assigning a name to the secret is mandatory, right? And it has to / should be unique.

sharp zealot
#

Yes. I honestly don't know if we could/should do that, but it's a good question. Now is a good time to revisit it

#

Would you mind creating an issue? 🙏 Otherwise I can do it monday (need to wrap up a few other things this week-end, with the crumbs of time I get in between family time 😛

latent trellis
sharp zealot
#

Basically like cache volume names

latent trellis
sharp zealot
#

Oh, meaning there might be a namespace conflict? Similar to cache volume namespace conflict?

latent trellis
#

yep

sharp zealot
#

Yes I guess you need to interpolate into the secret name each variable that might change in the context of creating the secret (for a given session)

latent trellis
#

I'm generating the secret name based on the content. I think that should be enough for now.

sharp zealot
#

Yeah, and I see your point that, if that works better for you, and has no downsides, it may be a wake up call to that name argument not being needed or helpful at all

kind carbon
#

That's basically what the CLI is doing when you provide a secret (hash based on the content). Helps with cache as well, but leaves an opening for a brute force attack, however difficult that is.

void widget
#

At a hackathon and it would really help to hack the compose / api service. We're hacking on a benchmark tooling of LLMs based on Dagger (pretty cool) @mossy hazel if you're around, seing you connected

void widget
#

PS: can we expose as service a function that also receives services ? I am getting an Invalid host header/ error which does seem to come from us.

PS: was my fault - The terminal DX loop is amazing

void widget
#

@dense canyon saved me 😍

sharp zealot
#

@void widget how's the hackathon going?

void widget
#

But, Dagger functions fit very well as the engine for their python based DSL. Pretty bullish 😇

plucky ermine
#

I seem to be in daggerverse publish limbo - has anyone ever seen this error?

trying to publish this via the site: github.com/levlaz/daggerverse/docusaurus@v0.2.1

get name: input: moduleSource.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to diff generated code: select: TODO: cannot diff with different relative paths: "" != "/"
wintry prism
#

Can you update your module's engine version from 0.11.2 and retry?

#

That being said, we need to update Daggerverse to 0.11.4 (currently at 0.11.3)

plucky ermine
#

I seem to be in daggerverse publish

latent trellis
#

Which API do you like better?

Option A:

func (m *Module) Func (
    // +optional
    // +default=true
    cache bool,
)

CLI invoke: dagger call func --cache=false

Option B:

func (m *Module) Func (
    // +optional
    noCache bool,
)

CLI invoke: dagger call func --no-cache

sharp zealot
sharp zealot
latent trellis
#

I get the point though

wintry prism
sharp zealot
#

I have accepted that they are all bad in some way

latent trellis
wheat river
#

What's the current status of Go workspaces support while developing dagger functions? — There are some utility functions in Go that most of my modules shared, and I truly hate DRY 😅 .

latent trellis
#

In my experience Go workspaces cause more trouble in Dagger than not. Why not just import it as a module?

agile saddle
wheat river
# latent trellis If you hate DRY, does that mean you like repeating yourself? 🤔

I don't consider them as modules. When I refer to utility functions, I'm talking about those that perform mundane tasks, such as converting a comma-separated string into a slice or performing very basic validations. I prefer to use the natural capabilities of the language of choice. In this particular case, that means utilizing Go workspaces instead of creating an entire module when it's not justified (I don't need a module :), I need to just share reusable code).
If the Go workspace + dagger combination wouldn't be problematic, would you create a module for sharing code?

latent trellis
wheat river
#

That's precisely my scenario —that it wasn't working properly with Dagger modules in the past, hence my question. I have a go module that encapsulates a set of reusable Go functions that normally I use in all my dagger modules. For that, in my Daggerverse mono repo I've configured a Go workspace to precisely allow my Dagger modules (Go modules) to reuse it. It works fine, but it fails when I try to publish (at least the last time I tried) without recognizing the go modules used in some of my dagger modules (more details here: https://discord.com/channels/707636530424053791/1232967596366757911)

Discord
Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

thorn moat
latent trellis
#

Like pushing to a remote service

sharp zealot
#

@thorn moat I guess this is a gap in our TTL discussion: we discussed how the callee can specify its caching TTL, but not how the caller might want to change it

thorn moat
#

yeah, makes me wonder if there's some generalized 'cache key' thing instead, and TTL is automation for 'generate one based on now().truncate(ttl)'

sharp zealot
#

@latent trellis the other discussion is a way for your function to specify its own caching behavior:

"I want my function to always be cached, with a TTL of 1 second"

// +ttl=1000
func (m *Module) Func () {

}

"I want my function to never be cached (ttl=0)"

// +ttl=0
func (m *Module) Func () {

}
sharp zealot
#

(that I could find)

thorn moat
#

yea not sure. it'd be sort of a formalization of the withEnvVariable("BUSTER", time.Now().Truncate(...)) pattern i guess?

sharp zealot
#

But whether TTL is enough or not, either way we're missing a mechanism for the caller to change it at runtime

sharp zealot
latent trellis
sharp zealot
latent trellis
#

Couple things I probably never want cached:

  • git clone
  • git push
  • helm push
sharp zealot
#

I guess TTL could be considered an operator concern, rather than something you write in the code? 🤔

latent trellis
thorn moat
latent trellis
#

Higher level commands like push/pull rely on that interface.

latent trellis
sharp zealot
sharp zealot
#

What's the story behind remotes.docker.resolver.HTTPRequest in the TUI? Is that the actual path of a function? But then why not show the call tree eg:

remotes()
  docker()
    resolver()
      HTTPRequest()

Any way we could make this message more user-friendly? It comes up a lot

thorn moat
sharp zealot
thorn moat
deft rain
#

i mentioned this to @thorn moat yesterday coincidentally

#

this is generally part of trying to make the plain progress better, some of this noise gets lost in the fast tui, but when it's in the logs forever, it sticks out a lot more

deft rain
#

there's a fairly horrendous hack in there, so need to go do some little bits of work in buildkit+containerd

indigo lichen
latent trellis
indigo lichen
sharp zealot
#

I think exposing custom otel spans so quickly, without much discussion on use cases and tradeoffs, was a mistake

#

"no is temporary, yes if forever"

#

I am seeing confused users trying to use this bonus feature correctly, when we don't even have the basics fully figured out

#

And scarse SDK dev cycles going into parity for this, when we have more urgent issues

thorn moat
# sharp zealot I think exposing custom otel spans so quickly, without much discussion on use ca...

the thought process was:

  1. this is inevitable (why integrate with otel without integrating with otel)
  2. we get this for free (buildkit already handled 90% of the plumbing) - one of the benefits of embracing an existing ecosystem in the first place
  3. users don't need to solve for "dagger + otel" - they just need to understand otel, and only if they want/care to, there's no special aspect of dagger, it's just otel like usual
    so:
  • where is the confusion coming from? isn't it all just standard otel integration?
  • what are the "basics" that we haven't fully figured out?

from my POV it was a tiny amount of effort in the Go SDK to point to plumbing that we already needed there for other reasons and everything "just worked" the same way otel always does, so I'm confused to see this tbh

#

there is an extra catch for emitting live telemetry data, but it degrades gracefully (telemetry gets flushed on span completion like otel normally does), so I saw that as a sort of 'progressive enhancement' option for SDK authors with little time

#

we also knew there would be kinks to iron out with swapping out the entire telemetry infra in such a short timeframe, but figured that was the lesser evil to doubling down on 100% bespoke telemetry infra that was also not really figured out. i know that's not the part you contend with, but compared to the other kinks this was assumed to be a more certain thing, given all of the above

thorn moat
#

maybe we just un-document it until we have the cycles to expose it properly? the entire tracing functionality has been framed as 'beta' so it's nothing is really 'forever' yet

sharp zealot
#

yeah switching to otel was an excellent decision for sure

#

yes I think we should hide the custom spans feature completely from the docs until we can clearly explain when how and why you should use it.

It is about dagger+otel, because we show spans alongside function calls in our Traces feature, and in the TUI. So to understand our visualization you have to understand otel spans. And if you emit custom spans from a dagger function you're changing how that function and its subfunctions will be visualized

thorn moat
# sharp zealot yes I think we should hide the custom spans feature completely from the docs unt...

Right, by that I only meant there are no special "dagger otel" APIs to learn, it's at least supposed to be as if you just integrated otel into any other project. But there definitely is an aspect of how it interacts with Dagger's lazy/async APIs that you need to be familiar with, and our docs really need to drive that point home - they're not there for you to use constantly in the same breath as Dagger APIs, since Dagger functions should be handling 99% of that. They're more of a "you should only use this if you are already pretty sure you need to" (e.g. measuring individual test performance)

thorn moat
latent trellis
#

Why isn't there a dag.NewFile method?

I find myself writing code like this a lot lately:

dag.Directory().WithNewFile("file.json", content).File("file.json")
kind carbon
latent trellis
wintry prism
#

💯 I was doing a GraphQL experiment the other day and noted it.

#

I'll open an issue

wintry prism
sharp zealot
#

I asked myself the same question last week 🙂 Agree

#

I think there may be a technical limitation - in buildkit, files can only exist in the context of a directory - they have a name property. So a file in a vacuum is not possible in buildkit I believe. But perhaps we can implement it in dagger anyway

wheat river
#

Just collecting random feedback — It'd be awesome to have more human-readable sort of errors. When facing errors during the development of a module such as:

Stdout:
invoke: input: container.from.withEnvVariable.withMountedDirectory.withWorkdir.withMountedCache.withEnvVariable.withExec.withExec.withExec
.stdout resolve: process "xyz command" did not complete successfully: exit co
de: 125

If there's an invalid parameter passed or unexpected by the Dagger APIs, would be awesome to have something like the environment variable passed is invalid due to or the mounted directory could not be found or you haven't set enough permissions, etc. I think it'll reduce the initial friction for new module builders who aren't so familiarised with the Dagger APIs. 🤔

sharp zealot
#

(also: I completely agree)

sharp zealot
#

How does everyone feel about the dagger develop command? Do you find it clear and useful? Or confusing and cumbersome? 🙂

deft rain
#

How would the IDE support work without dagger develop?

#

I use it to actually get my IDE to do anything at all to detect issues

warped canyon
deft rain
#

but imo, we shouldn't remove dagger develop until we have a fully working solution like that (manual is better than nothing right now)

restive shore
sharp zealot
#

Yes I'm not suggesting we remove it - was wondering if we should consider either renaming it, or splitting its different features in more focused commands

#

(both ideas are speculative)

#

Things dagger develop does today:

  • Install and initialize an SDK (if needed)
  • Set some SDK configuration (where to store source directory)
  • Have the SDK one-time generate boilerplate (if needed)
  • Have the SDK generate client bindings
  • Have the SDK build the runtime containers and print any errors
wintry prism
wheat river
# sharp zealot How does everyone feel about the `dagger develop` command? Do you find it clear ...

I'd say that it's also a sort of static checker, or at least is another usage that I give to it. If there's something 'wrong' in your code, the code generation part will complain and you'll notice it. Having a dedicated command for static analysis or a more advanced linter would be helpful. It's more or less similar to what @deft rain commented.
I think having more scoped/focused commands will be more readable, like dagger develop check (for static analysis, code-generation validations, a canonical fmt that we'd like to enforce as part of a more standardise way of developing modules? thinkies , if the module has dependencies on another modules, check basic things such as the dependency path, versions, etc. ). By the way, I use mostly Jetbrains IDE(s) (Goland), and I hook it to my On Save/File Watcher settings, it's a sort of a hook that helps me to get quick feedback whenever I'm developing a module.

sharp zealot
sharp zealot
sharp zealot
sonic vessel
sharp zealot
#

dagger sdk install -> install an sdk
dagger sdk generate -> ask the sdk to regenerate bindings
dagger sdk update -> update sdk
dagger sdk config -> configure sdk

maybe?

honest hearth
#

I personally like how explicit the above verbiage is ^^

#

and not terribly verbose

sharp zealot
#

also need to figure out how to differentiate "I'm developing dagger functions" and "I'm developing my own project and want to embed generated client bindings into it" ( #1238585212523384832 )

wheat river
# sharp zealot How would you feel about `dagger sdk`? Since all these features have in common t...

That makes sense. Provide nice (dev) tooling for quick feedback, rapid troubleshooting, and hopefully, a seamless integration with the IDE will help in reducing a lot of the friction, and encourage more folks to develop modules. What about dagger sdk lint or dagger sdk validate, so it's easily hooked into git hooks, or even CI made for Dagger modules?
Personally —I think it's a scenario reached by folks that have developed modules already— is how to enrich the debugging experience #1230026862860963920 message — I'd love to have an integrated way to hit my breaking points in my IDE while debugging functions, instead of just relying in plain logs that aren't necessarily human-readable.

sharp zealot
#

Calling for comments on an important design decision.

The decision is: are we serious about making the Dagger CLI scriptable?. Do we want to support users implementing logic in a shell script beyond a unique dagger call?

This is important because it will influence the structure of our docs, and our best practices in general. It will also affect our priorities.

  • Docs: does it make sense to keep "user manual" and "developer manual" separate? If we want the CLI to be scriptable, then it doesn't.
  • Best practices: is it realistic to daggerize your project without writing Go, Typescript or Python? Today the answer is "no". If we make the CLI truly scriptable, it might become possible in the future.
  • Features: if we want the CLI to be scriptable... We need to prioritize work to make it happen. The longer we wait, the harder it will be to change best practices and docs later.

Related design discussions:

GitHub
`dagger call` is tedious with too many flags+args · Issue #6723 · d...

Somewhat related to #6112 but a bit more general Passing a large number of flags and args when using dagger call can become way too tedious to type out by hand. One example would be support for fil...

GitHub
Contextual modules · Issue #7199 · dagger/dagger

Problem There are two types of Dagger modules: those that are standalone software projects, and those that exist in the context of another software project. Let's call those standalone modules ...

GitHub
Allow calling core functions from the CLI · Issue #6947 · dagger/da...

Problem There is no way to call core functions from the CLI Solution Allow calling core functions from the CLI. See https://daggerverse.dev/mod/github.com/shykes/core for a userland implementation.

GitHub
CLI: don't hide `id` from core types · Issue #6710 · dagger/dagger

Problem When calling a function that returns a core type, like Container or Directory, I can then call any function from that core type... except id. This makes it unnecessarily difficult to stitch...

GitHub
Dagger Scripts: a more convenient way to call Dagger Functions from...

TLDR: dagger call is very safe, but lacks the convenience of artisanal scripts. There is a way to get both: safety and convenience. Problem Now that Dagger Functions are generally available, the st...

#

Separate topic: what happens if I write a go module that uses graphql interfaces? Can python and ts clients consume that module?

rocky harbor
celest osprey
#

is posible to use something like:

@dagger.task()
some async function

to make task like in Taskfile.yml on newest version?

warped canyon
#

is posible to use something like:

sharp zealot
#

(not completely fleshed out, but wanted to get the discussion going asap)

sharp zealot
leaden pagoda
#

hey all, if i have a tool inside the container WithExec but it returns a non 0 status code, i seem to be having issues exporting a file that is generated from the tool (its exit code = bad if something fails validation)

#

is there a pattern on returning or grabbing a file from a container with a bad exit code, which is expected behaviour in some ways

quick wind
#

is there a pattern on returning or

sharp zealot
#

Conversation starter. How does this script snippet feel?

#!/usr/bin/env dagger

# Build a base image
base() {
 wolfi | with-packages git openssh go | container
}

# Build the app
build() {
 base | with-directory /app $(
    base |
    with-directory /src "$(context directory)" |
    with-workdir /src |
    with-exec go build -o ./bin ./... |
    directory ./bin
 )
}

build
plucky ermine
#

I am trying to pass a Github URL as a default arg for *Directory but it's not happy. Is there any way to get this to work?

this:

func (m *MyModule) Build(
  ctx context.Context, 
  // Source code Directory
  // +optional
  // +default="https://github.com/golang/example#master:hello"
  dir *Directory,
) string {
  // build app 
  builder := dag.Container().
    From("golang:latest").
    WithDirectory("/src", dir).
    WithWorkdir("/src").
    WithEnvVariable("CGO_ENABLED", "0").
    WithExec([]string{"go", "build", "-o", "myapp"})

results in:

levlaz@Levs-MacBook-Pro build % dagger call build
✘ initialize 0.1s
! query module objects: returned error 400 Bad Request: failed to get schema for mod


Error: query module objects: returned error 400 Bad Request: failed to get schema fo
r module "MyModule": failed to get field spec: failed to decode default value for ar
g "dir": decode "Directory" ID: failed to decode base64: illegal base64 data at inpu
t byte 5
sharp zealot
#

@plucky ermine I don't think you can set objects as default values

#

Scalars only

#

The solution is to implement the default value in code:

if dir == nil {
 dir = dag.Git("https://github.com/golang/example").Branch("master").Tree().Directory("hello")
}
plucky ermine
deft rain
#

Conversation starter - scripting snippet

uneven shadow
#

So daggerverse only supports github.com urls for publishing but is it possible to host modules on your self hosted git instance and get them executed with dagger call?

upbeat herald
# uneven shadow So daggerverse only supports github.com urls for publishing but is it possible t...

Hey 😄

For now we only support GitHub url but we're actively working on it, you can follow our progress in the following issue https://github.com/dagger/dagger/issues/7218

GitHub
Add more VCS (gitlab, gitea, bitbucket) support to Zenith and Dagge...

Problem Dagger modules and Daggerverse do not work with other SCM than GitHub. For the past few months, several discussions have occurred around the best way to solve it. This is now a top-priority...

uneven shadow
#

awesome, thanks. Would be really nice to have that working.

echo spindle
#

Has there been any thought to using something other then git as a repo/registry for dagger modules? I know that is what go does, but it isn't awesome IMO. It makes things not very portable. You end up using something like Athens that ends up pulling down the equivalent of an artifact. And it also expects weird hacks from git providers (github, gitlab) to support discovery. Most other languages, even non-compiled ones, do this better. Even docker/helm has the concept of a registry that I can pull a distrinct bundle from.

sharp zealot
#

Git only?

leaden pagoda
#

A lot of the cookbook seems to be using dagger in quite a different way. Any examples of "container status code failure, fail CI Platform Pipeline, but export results"?

restive shore
#

Is there a way to tell dagger to ignore a flag when it's passed in but the value is the default null value? For example if I have a string arg in Go which is // +optional . I want it to still work work when I pass in the flag but empty value. The reason I ask is when I use dagger in a CI orchestrator I want to be able to do the bare minimum within the CI system. I don't want to conditionally modify the dagger call. just want to pass in a basic set of user inputs regardless of them being null or "".

Right now I get flag needs an argument: error.

wintry prism
#

Is there a way to tell dagger to ignore

sharp zealot
sharp zealot
#

Has anyone seen dagger init --sdk=go fail with Error: load package "<foo/bar>": package name is empty ?

sharp zealot
#

I wish we had gone the route of calling construtors ModuleName() rather than New()

#

It would give me more control on the name of my top-level type

#

For example in a Go module, I want my top-level type to be a project. But I'm stuck having to call it Go

#

So I have:

func New(version string, source *Directory) *Go {}

// A Go project
type Go struct {
  // The project's Go version
  Version string
  // The project source code
  Source *Directory
}

But would prefer:

func Go(version string, source *Directory) *Project {}

// A Go project
type Project struct {
  // The project's Go version
  Version string
  // The project source code
  Source *Directory
}
sharp zealot
#

@thorn moat @chrome pilot I have a trick otel question 🙂

#
  • I want my test function to expose test results as structured data, instead of a flat wall of logs + exit code
  • Right now I need to parse a report file to achieve this
  • But if my tooling already instrument tests with otel spans, and dagger magically gobbles those otel frames for visualization... Then perhaps I don't need to parse a report file?
  • Is there any way the Dagger API could give my function access to the custom otel spans generated by the containers I'm running? 😇
thorn moat
sharp zealot
#

I didn't even realize dagger watch existed 😄

#

Nice thanks

#

Do you think that could be made a super-easy pattern in the future? Just wondering if it could become the "standard" way of wrapping testing tools in a module

thorn moat
thorn moat
# sharp zealot Do you think that could be made a super-easy pattern in the future? Just wonderi...

hmmm maybe! right now the pub/sub is fire-and-forget, so you'd need to subscribe before running the suite, and then process the stream of spans, so it's a bit involved. We could perhaps add persistence and allow querying spans in the API, sort of like how Buildkit has a History API now. I'm not sure about super-easy though. You'd probably want to figure out how to identify the test spans in particular for example. Or maybe just look for all spans beneath the command's span, but then how do you identify the command's span. There's probably something there but it'd take a lot of noodling

sharp zealot
#

yeah sounds interesting. I think buffering the spans and letting me query them would be killer. unlike buildkit it wouldn't need to be persisted at all. my function just needs its spans while it's running.

#

I don't know enough about otel format to know what to query the spans for, or how to figure out which span is which test. but seems like a solvable problem

thorn moat
# sharp zealot I don't know enough about otel format to know what to query the spans for, or ho...

it'd probably mean querying by the ScopeName field (aka instrumentation library, not sure what the most accurate terminology is), e.g. the Ruby RSpec instrumentation sets it to OpenTelemetry::Instrumentation::RSpec: https://github.com/open-telemetry/opentelemetry-ruby-contrib/blob/main/instrumentation/rspec/lib/opentelemetry/instrumentation/rspec/formatter.rb#L32 - and then it creates a span for each test ("example")

wheat river
#

Hey guys, I'm exploring some alternatives and crazy ideas about how to effectively test my modules. I've thought in these two approaches — I'm planning to include them both actually, but I'd like to get your thoughts about them first:

  1. For each module, ship a test (vanilla) Go module that's going to have Go tests using testcontainers. With this, testing the module from the CLI interface (it entails installing the Dagger CLI into the testcontainer container, and ensure the dagger engine works properly) will ensure that the dagger module as commands works as expected.
  2. For each module, similarly to what @latent trellis suggested I'll create a test module where the module's APIs will be tested as code (dag.MyModule().WithMyAPI()).

Also, is there any plan to include a built-in testing feature for quickly test modules/functions in future releases?

sharp zealot
#

in other words your test module can handle both testing the code and CLI interface of your module

wheat river
lime comet
# wheat river Hey <@488409085998530571> , about the first point, that'd be ideal, indeed. Do y...

Here is quick example of a go module that makes a dagger call in a container that has Dagger's CLI installed using Dagger's SDK. Given its dagger in dagger, it is using the dagger engine of the underlying host (my computer). The important parameter is within the last WithExec, dagger.ContainerWithExecOpts{ExperimentalPrivilegedNesting: true}. This will allow the underlying container to access the engine:

package main

import (
    "context"
    "fmt"
    "log"

    "dagger.io/dagger"
)

func main() {
    c, err := dagger.Connect(context.Background())
    if err != nil {
        log.Fatal(err)
    }
    daggerVersion := "0.11.4"
    out, err := c.Container().From("ubuntu:lunar").
        WithExec([]string{"sh", "-c", "apt update && apt install -y curl"}).
        WithExec([]string{"sh", "-c", fmt.Sprintf(`cd / && DAGGER_VERSION="%s" curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION="%s" sh`, daggerVersion, daggerVersion)}).
        WithEntrypoint([]string{"/bin/dagger"}).
        WithExec([]string{"call", "-m", "github.com/shykes/daggerverse/hello", "hello"}, dagger.ContainerWithExecOpts{ExperimentalPrivilegedNesting: true}).
        Stdout(context.Background())
    fmt.Println(out, err)
}
image.png
restive shore
#

I can't wait to test this https://github.com/dagger/dagger/pull/7272 when released. It looks great! Dagger pipeline console output within our CI orchestrator (Jenkins), left a lot to be desired as it was confusing and borderline incomprehensible. IMO the plain output experience is equally important as the terminal TUI output as I expect a huge portion of dagger workloads to run on CI orchestrators due to various reasons within enterprises.

GitHub
feat: improve plain progress by jedevc · Pull Request #7272 · dagge...

Fixes #7137.
With the changes as part of #6835 (in v0.11.0), all old TUIs were removed, including the old --progress=plain output. With #7069 (in v0.11.1), a limited version of the plain TUI was re...

wheat river
shy ledge
#

Ooohhh......

sharp zealot
#

Yeah 🙂

deft rain
#

question - could we potentially support the file/env prefixes for normal string arguments? i've gotten caught several times where i change an arg to not be a secret, and then the args that are passed to it work entirely differently

restive shore
#

is the only way to "upgrade" dagger engine version of a module to manually edit the dagger.json right now?

deft rain
#

it should automagically update when you run dagger develop with a new engine

#

but fyi right now, that number doesn't affect the behavior of any dagger command - yet

restive shore
#

ah... then is the install script maybe installing 0.11.4 on my CI, let me retest

#

When testing the new plain progress, I was seeing a bunch of HTTP stuff. I only noticed later it pulled engine 0.11.4

deft rain
#

aha, yes, so v0.11.5 of the engine is required to eliminate the repetitive http stuff

#

sadly can't just be done with a client side fix

restive shore
#

funny enough, HTTP stuff is still showing in the TUI. Now the "plain" progress is cleaner than the TUI 😄

restive shore
#

Ok now I'm definitely on 0.11.5, but I am still seeing the HTTP spans when doing a Publish. Did that get missed? CC: @deft rain

deft rain
#

do you also have a matching v0.11.5 cli?

#

it's potentially possible publish might have been missed

restive shore
#

Yes matching CLI, engine and SDK version

#

Everything else but Publish seems to be showing the new output

deft rain
#

okay, do you have a screenshot?

restive shore
#

btw, these colors aren't great in light backgrounds

deft rain
#

sigh yeah that'll do it

restive shore
#

I had to switch to dark mode to properly see them

deft rain
#

do you mind if i open an issue with your screenshot?

restive shore
#

Nope please go ahead. I only shared what I could 😄

restive shore
deft rain
#

hm, so we only use the 16 color pallete configured by your terminal

#

that said, default configurations of a lot of these colors is excessively terrible - i remember when working on similar things in buildkit, it's hard to strike a balance

#

the good thing is that if you don't like the colors, you just to configure your terminal to get the results you want

deft rain
#

thanks for the quick report

restive shore
#

Nice! Thanks for the quick fix!

restive shore
thorn moat
# restive shore Re: colors. The above output is from Jenkins console log. I don't know/think the...

honestly there's not a lot that can be done in the CLI for this - like @deft rain said, we're already restricted to a 16 color palette, it's the outer software's responsibility (terminal emulator, Jenkins) to ensure those colors are all readable. unfortunately, many of them suck at that. 😦 for example the default 'blue' color is unreadable on dark backgrounds on macOS defaults, so I've already learned to stop using it, but there's only so much you can do... (not being able to use blue is already a crazy thing to me)

deft rain
#

buildkit's tty output uses that blue color, and it's fairly sad, yup 😦

sharp zealot
#

If I want to install Dagger from main, and use that as my every day install, is there a set of reproduceable dagger calls you can give me?

Is this enough? Or do I need to set additional env variables to make sure the right engine container is downloaded:

# Mac-specific
dagger call -m github.com/dagger/dagger --source https://github.com/dagger/dagger \
  cli \
  file --platform darwin/arm64 \
  -o ~/bin/
restive shore
deft rain
#

ideally. we should have our install.sh script install from an arbitrary main commit i think

#

and then have a dagger function that can wrap that / vice versa

thorn moat