daggernauts

[restive shore]

Ah didn't know this was an option. Let me try it. Maybe it should be documented too?

[thorn moat]

probably yeah! it's "sort of" a standard, but worth calling out that we support it: https://no-color.org/

@restive shore would you expect to see that in --help or somewhere in the docs? (both make sense, just wondering what your first instinct would be)

[restive shore]

I was thinking of only docs, but now that you said it. It makes sense to be in --help too! Probably part of the --progress flag?

I don't want to clutter --help too much. As I don't think this is an option that will be widely used. But that's just my guess

[sharp zealot]

Or, we make our registry 100% just-in-time, backed by the build function. Then the CLI just pulls its tag or commit, and it always gets the correct image back

[sharp zealot]

Call for bikeshed: https://github.com/dagger/dagger/issues/6947#issuecomment-2135814264

[sharp zealot]

My module A imports another module B. Can a function from A return a type from B, using interfaces?

[plucky ermine]

I noticed a nice improvement in 0.11.5 - instead of printing out <nil> functions with no return just return nothing, I like it!

[latent trellis]

Kudos to @kind carbon for basically fixing it within minutes after the issue was open: https://github.com/dagger/dagger/issues/7369

[amber pilot]

Hi
github.com/dagger/dagger v0.11.5 dep github.com/dagger/dagger/engine/distconsts v0.11.4, which is not go get-able?
now I have to copy engine/distconsts in local project and with go replace directive to fix.

[deft rain]

yeah a go-replace is needed for this for now

will be fixed by https://github.com/dagger/dagger/pull/7346

this was added because of a dependency issue between our ci and core code

[strange arch]

If it’s possible, I’m greatly interested in how to do it, this would allow me to simplify my modules 😉

[leaden pagoda]

So, i started to get weird behaviour, i thought maybe i was building my container wrong, so i decided to back to the basic docs just to re-inforce my understanding but even the basic doc examples arent working for host filesystem access

https://tenor.com/view/futurama-fry-hmm-squinting-gif-4973356107513966870

The weird part with -d is that this file seems to be "uploaded to the engine" fine..

✔ upload . from PATRICK-PC (client id: jhyf9iajxjh4gcitjjjlelp1i) (include: C:\Projects\example\README.md) 0.0s
✔ load cache: copy upload . from PATRICK-PC (client id: jhyf9iajxjh4gcitjjjlelp1i) (include: C:\Projects\example\README.md) 0.0s

[sharp zealot]

@kind carbon is your codegen dagger module the best place to look for a reference implementation of introspection? If I want to write a custom tool that inspects available functions, their arguments etc.

(not in the context of developing an SDK)

[kind carbon]

Yes. That's actually Go's introspection but turned into a module so it's more easily consumable.

[sharp zealot]

nice thank you

sorry if I already asked, but can I easily go from introspecting a function to actually calling it, in the same session?

[kind carbon]

Yes

It's what the CLI does.

[sharp zealot]

I'm experimenting with a web UI for Dagger, in case that's interesting to you @manic lynx @hallow gust 😇

[manic lynx]

yo

[sharp zealot]

But the CLI isn't using that introspection module. Does what the CLI does translate when using that module?

[kind carbon]

The CLI uses this system though: https://daggerverse.dev/mod/github.com/helderco/daggerverse/codegen@357dfd7f97eae9a62a2a13454b48cb7c69c44e34#Experimental

[kind carbon]

No, there's an hardcoded "introspection" query in the CLI, and a custom struct data structure to map that result into.

[manic lynx]

@sharp zealot some of the more beginner PHP devs are wildly different to the senior PHP software engineers (same level as golang/python/rust devs) .. so a Web UI is a smart move @sharp zealot

To execute dagger run calls, if that's what you've got in mind

[kind carbon]

But it uses the same source for the information as the codegen module in that experimental type.

[sharp zealot]

I guess what I'm realling hoping for (but nobody has built yet, so looking at maybe trying myself as a side project), is a React component for easily querying the Dagger API (including modules) and rendering a UI from it.

Or replace React with another web framework, I'm equally terrible at all them, so agnostic in that way

[manic lynx]

How much do you like Rust ? 😉

[sharp zealot]

Yeah no

I'm trying to move up the stack, not down here 😛

[manic lynx]

I've got something to you

I can't find it. It's like Next.js but for rust - it's mental LOL 🙈

@sharp zealot found it 😛 it's called https://leptos.dev/

Look how great it is. You can write API and SQL calls and your frontend ReactJS-style stuff, in one file

#sarcasm

[sharp zealot]

Yeah no thank you

I get the appeal for Rust devs though

[manic lynx]

night night @sharp zealot - onward 💪

[plucky ermine]

I am getting a bit of a cryptic error and I could use some help.

I have this function in python that works just fine.

https://github.com/levlaz/dagger/blob/docs-268-multi-arch/docs/current_docs/manuals/developer/cookbook/snippets/builds/multi-arch/python/main.py

I translated it to typescript and I get the following error

https://github.com/levlaz/dagger/blob/docs-268-multi-arch/docs/current_docs/manuals/developer/cookbook/snippets/builds/multi-arch/typescript/index.ts

levlaz@Levs-MacBook-Pro typescript % dagger call build --src="https://github.com/golang/example#master:hello"
✘ MyModule.build(
    src: ✔ Directory.directory(path: "hello"): Directory! 0.4s
  ): String! 2.0s
! call function "build": process "tsx --no-deprecation --tsconfig /src/tsconfig.json /src/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1
  ✘ exec tsx --no-deprecation --tsconfig /src/tsconfig.json /src/src/__dagger.entrypoint.ts 2.0s
  ! process "tsx --no-deprecation --tsconfig /src/tsconfig.json /src/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1
  ┃                                                                                                     
  ┃       { id   }                                                                                      
  ┃                                                                                                     
  ┃                                                                                                     
  ┃ error: parse selections: parse field "id": Query has no such field: "id"                            

Error: response from query: input: myModule.build resolve: call function "build": process "tsx --no-deprecation --tsconfig /src/tsconfig.json /src/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

Stderr:
Error: query:

      { id   }
    

error: parse selections: parse field "id": Query has no such field: "id"

Here is the trace: https://dagger.cloud/levs-test-org/traces/41d6c525a94c75f7efd37980b8645c38

[sharp zealot]

does it work with an older version of dagger? perhaps a regression in typescript sdk?

[hallow gust]

oh this is interesting
what's the main goal/need of the UI?

[sharp zealot]

https://tenor.com/view/i-do-it-because-i-can-jack-donaghy-30rock-i-do-what-i-want-im-capable-of-doing-it-gif-21884978

[hallow gust]

oh just read this

is the idea something similar to the GraphQL playground but with the installed modules added?

so you could query all the functions available?

and the UI I guess would be to autocomplete/save queries?

this is similar to a VSCode plugin I had in mind once

[restive shore]

Any chance this could evolve to show traces like dagger cloud does? That would be so much win! Even if it's just a "basic" version of it.

[wintry prism]

I am getting a bit of a cryptic error

[leaden pagoda]

https://tenor.com/view/atlanta-lakeith-stanfield-creepy-smile-creepy-gif-2838775962290957578

When you add a PR to a open source repo with "dagger" for the dockerisation and say "hey would you guys consider publishing a docker image?"

[wintry prism]

When you add a PR to a open source repo

[plucky ermine]

Trying to run dagger/dagger linter locally and seeing this very weird error all of a sudden

dagger call --source=.:default sdk typescript lint

✘ Container.from(address: "tonistiigi/binfmt@sha256:e06789462ac7e2e096b53bfd9e607412426850227afeb1d0f5dfa48a731e0ba5"): Container! 0.0s
! failed to resolve source metadata for docker.io/tonistiigi/binfmt@sha256:e06789462ac7e2e096b53bfd9e607412426850227afeb1d0f5dfa48a731e0ba5: failed to do request: Head "https://registry-1.docker.io/v2/tonistiigi/binfmt/manifests/sha256:e06789462ac7e2e096b53bfd9e607412426850227afeb1d0f5dfa48a731e0ba5": context canceled

Can anyone else reporduce this?

Hm, seemed to be a transient issue that went away.

[wintry prism]

too many people pulling Tonis' image for qemu 🙂

[plucky ermine]

I recall someone saying there were some GC setting we could tweak to make cache a bit better, is that true for Docker on MacOS? -- if so, any pointers?

[plucky ermine]

Feels like this is something that should be caught locally when I run dagger develop

Instead I only see it after publishing to daggerverse

Stdout:
generating go module: containerd
Stderr:
Error: bootstrap package: existing go.mod has unsupported version 1.22.3 (highest supported version is 1.22.2)

https://daggerverse.dev/mod/github.com/levlaz/daggerverse/containerd@22b261cb47b6ea9d5b2c3bf051437522e6103e51

[strange arch]

Hello, question about dagger develop for a Go module: was it idempotent, but not anymore or hasn’t it ever been? Do you guys commit the generated files or not?

[leaden pagoda]

I ignore them and just use dagger develop cmd when doing a fresh git clone

in the new sdk it adds gitignore of the generated code. so I'm pretty confident you don't check them in

[strange arch]

Actually, it’s configurable by adding

  "codegen": {
    "automaticGitignore": false
  }

to dagger.json.
I was checking generated files in, but I’m considering removing them and calling dagger develop when I need them in CI (for example to do static code analysis) due to this idempotency issue.

[sharp zealot]

Yes dagger develop is meant to be idempotent. If you're seeing that it's not, then it's a bug.

[latent trellis]

daggerverse is still running v0.11.4. Any ETA on when it's going to be upgraded to v0.11.6? @wintry prism

https://daggerverse.dev/crawl/github.com/sagikazarmark/daggerverse/go@v0.3.0

get name: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "go" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1

Stderr:
# dagger/go/internal/telemetry
internal/telemetry/proxy.go:50:14: cannot use proxySpan{…} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: proxySpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)
internal/telemetry/proxy.go:59:20: cannot use proxySpan{} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in variable declaration: proxySpan does not implement "go.opentelemetry.io/otel/trace".Span (missing method AddLink)

[strange arch]

Thanks for your answer, so I’ve created a bug report: https://github.com/dagger/dagger/issues/7524.

[daring vine]

Hey everyone, hope you're all doing great! I've just created my first module and I wanted to ask for advices and if that's possible. Which channel should I use please? https://daggerverse.dev/mod/github.com/Smana/daggerverse/kubeconform@98af039dec822c24eae239cd1c3ff6df3a4d1828

[sharp zealot]

here is good for asking advice 🙂 Congratulations on building your first module!

[daring vine]

Thanks. Actually I was aiming to build another one that fetches the open api schemas from sources and return a *Directory for the kubeconform module. Indeed, as you can see it is made to handle a list of directories to load. Kubeconform needs OpenAPI schemas for custom resources. I was wondering if there's already a similar module because the source urls could be git repositories, archives ...

[sharp zealot]

I don't know of such a module, but it's a big wide daggerverse out there, not impossible that someone has done it.

Have you searched daggerverse.dev yet?

I've been thinking about the best way to do this:

- Add support for calling modules from an external client

It makes sense to expose a command to generate a client library for a given module, or set of modules, in the language of your choice. The output would be a directory, which you (the client developer) would be free to save at the best place within your project, then import locally.

That made me realize, I kind of wish I had that too when developing a module... Right now the client library is dumped in the same directory as my client (in this case the client is a dagger module). But I wish they were more cleanly separated.

For example in Go, the generated go client library has its own go dependencies. Because the generated client library is mixed with my own code, their dependencies are mixed together. It's weird to have a project with zero dependencies, and all of the sudden your go.mod is 100 lines long because a generated client library you happen to use, has tons of dependencies

[latent trellis]

If I understand correctly, something like this could work :

dag.Directory().WithFiles(
    dag.HTTP("https://example.com/openapi.yaml"),
)

[sharp zealot]

Are you asking if it works? Or saying that it doesn't work, but should?

[latent trellis]

That piece of code downloads a file (or more) and adds them to a directory.

Sorry, I probably should have answered directly to the question above.

If that's the requirement, it should be enough, maybe no need for a separate module.

But I could have misunderstood the requirement

[sharp zealot]

I found a cool bug in the Go SDK 😛

If you change the name of your module to the name of a core type, it will load the core type as the entrypoint of your module

I present to you: support for calling core functions from the CLI, in bash 🙂

function core() {
 (
  set -ex
  cd $(mktemp -d)
  echo "{\"name\":\"$1\", \"sdk\":\"go\"}" > dagger.json
  echo 'package main' > main.go
  shift
  dagger call "$@"
 )
}

Then:

core container \
 from --address=alpine \
 with-mounted-directory --source=https://github.com/dagger/dagger --path=/src \
 with-workdir --path=/src \
 terminal

https://tenor.com/view/teteakanpe-gif-21347890

https://github.com/dagger/dagger/issues/7528

[sharp zealot]

My function implementation:

func Foo(ctx context.Context) error {}

The generated binding:

func Foo(ctx context.Context) (Void, error) {}

🤔

[wintry prism]

daggerverse is still running v0.11.4.

[daring vine]

Thank you, I'm gonna give it a try, see if that fits my needs. I was about to use modules chaining, but now that I mention it I can't find an example. I was pretty sure that was possible... 🤔

[latent trellis]

You can also use dag.Git for cloning a Git repo.

You can use arc to extract the contents of an archive: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/arc@b45dbd7448bb967aca4a538af9ce7f042abf0316#Arc.unarchive

The nice thing about these is that you don't have to wait for any of them to complete, you can just add them directly to a directory:

dag.Directory().WithFiles(
    dag.HTTP("https://example.com/openapi.yaml"),
    dag.Git("https://github.com/you/repo.git").Commit("HASH").Tree().File("openapi.yaml").WithName("openapi2.yaml"),

    dag.Arc().Unarchive(dag.HTTP("https://example.com/openapi.tar.gz")).File("openapi3.yaml"),
)

[daring vine]

oh interesting, thank you

[leaden pagoda]

I feel like dag.HTTP, dag.Git and any others need more awareness, when you do searching on the docs, i dont think they have a section on any page specifically? So would people assume they need to write something like WGET etc in a container?

[sharp zealot]

cc @bleak nest we'll add it to the docs backlog

https://github.com/dagger/dagger/issues/7529

[bleak nest]

Yes, will do

[plucky ermine]

My blog publishing pipeline had three redundant files in three different formats that I was maintaining by hand.

This weekend I wrote a dagger module that interacts with the miniflux sever api and removes all this manual work for me.

The module also takes advantage of the CurrentModule API. This is my first time using it, its super powerful!

I wrote a bit about it here: https://levlaz.org/miniflux-dagger-module/

[leaden pagoda]

daggerverse best practices?

[restive shore]

I've been creating modules for a while but didn't know about dag.HTTP 😯 . I've been using curlimages/curl for my api calls. That being said, dag.HTTP always returning a file is a bit confusing to me. It would be better if it returned the raw results of a HTTP call so the end user can decide to put it in a file if needed. Then it could be used as a client for calling any API.

[leaden pagoda]

Maybe there could be variations of this? I like the simplicity of Http returning a file for things like an OpenAPI Spec, when i know for sure a simple HTTP GEt will return a yaml file or json file.

[latent trellis]

I'm satisfied with how dag.HTTP works. It's perfect for 90% of the use cases. For the rest, you can use curl.

[wheat river]

Inspired by this answer from @lime comet , I've published a DaggerInDagger module #daggernauts message that suits my requirements, and perhaps it'll help others that need to run dagger in dagger as well (for running CI on their modules, or publishing from GitHub/GitLab, etc.).
https://daggerverse.dev/mod/github.com/Excoriate/daggerverse/dagindag@9783ed1cc5a3c50165c861895d113f4c8612ef47
@sharp zealot 😸 your hello module is so handy (for testing purposes)

dagger -m github.com/Excoriate/daggerverse/dagindag@v1.15.0 call use-fn \
--mod-name=github.com/shykes/daggerverse/hello \
--fn="hello" \
--fn-args="--greeting=Yoooo, --name=Alex"

[latent trellis]

Anyone else seeing this error: get fn arg value: returned error 400 Bad Request: {"errors":[{"message":"transport not supported"}],"data":null}?

I've seen it at least 3 times in the last 10 hours in different projects using different versions (0.11.4, 0.11.6).

[sharp zealot]

Do you get the error consistently? Or intermitently?

And what's the argument in question?

[sharp zealot]

Nice! By the way I also published it at github.com/shykes/hello to save everyone keystrokes

[latent trellis]

It seems random. I haven't actually changed anything in the workflows and I haven't been able to figure out yet which function it complains about.

[thorn moat]

strange, seems like it'd be coming from here: https://github.com/99designs/gqlgen/blob/fbf73ee1f0daecd78fc5f65634a4a9ae59d20255/graphql/handler/server.go#L119

[plucky ermine]

I have this same issue, is this indeed a bug, and if not is there some other way to get logs to show up in the TUI?

https://github.com/dagger/dagger/issues/7510

[sharp zealot]

I feel like this deserves more eyeballs. You can call any core function from the CLI

[wintry prism]

Pretty cool indeed!

core container \
from --address postgres:latest \
with-env-variable --name POSTGRES_PASSWORD --value postgres \
with-exposed-port --port 5432 \
as-service up

Yeah, it's funny, I was testing some things in the GraphQL playground the other day and remembered how nice and elegant that is.
This is the same experience, shell style.

[sharp zealot]

btw @wintry prism there is a proposal to literally make this the official UX, with dagger core

[wintry prism]

Pretty cool indeed!

[sonic vessel]

The --sdk usage is a bit weird when using sdk from local path. I have sdk module like this

wingyplus@WINGYMOMO ~/s/g/w/dagger-dotnet-sdk (module)> ls -lah
total 84K
drwxr-xr-x  5 wingyplus wingyplus 4.0K Jun  5 23:14 ./
drwxr-xr-x 16 wingyplus wingyplus 4.0K Jun  2 22:33 ../
drwxr-xr-x  8 wingyplus wingyplus 4.0K Jun  5 23:12 .git/
-rw-r--r--  1 wingyplus wingyplus  160 Jun  2 23:31 .gitattributes
-rw-r--r--  1 wingyplus wingyplus   97 Jun  3 23:26 .gitignore
-rw-------  1 wingyplus wingyplus  25K Jun  2 23:31 dagger.gen.go
-rw-r--r--  1 wingyplus wingyplus  246 Jun  2 23:31 dagger.json
-rw-r--r--  1 wingyplus wingyplus 1.4K Jun  2 23:31 go.mod
-rw-r--r--  1 wingyplus wingyplus 6.4K Jun  2 23:31 go.sum
drwxr-xr-x  5 wingyplus wingyplus 4.0K Jun  2 22:39 internal/
-rw-r--r--  1 wingyplus wingyplus 1.3K Jun  2 22:44 introspection.graphql
-rw-r--r--  1 wingyplus wingyplus 4.1K Jun  5 23:12 main.go
drwxr-xr-x  8 wingyplus wingyplus 4.0K Jun  5 22:49 sdk/

And I want to use sdk from this path. I initialize module with dagger init --sdk=. potato , its got the error

✔ connect 1.8s
  ✔ starting engine 1.4s
  ✔ connecting to engine 0.2s
  ✔ starting session 0.2s
✘ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
! failed to collect local module source deps: failed to collect local sdk: local module at "/home/wingyplus/src/github.com/wingyplus/dagger-dotnet-sdk/potato" has a circular dependency

Error: failed to generate code: input: moduleSource.withName.withSDK.withSourceSubpath.resolveFromCaller resolve: failed to collect local module source deps: failed to collect local sdk: local module at "/home/wingyplus/src/github.com/wingyplus/dagger-dotnet-sdk/potato" has a circular dependency

So I thought the engine might do something with the sdk path. So I change the init command to dagger init --sdk=.. potato, the module is created with correct sdk.

[sharp zealot]

Making my first contribution to a Python module. Staring at a 500 line python stack trace wrapped in a buildkit error. Thanks I hate it

[sharp zealot]

The docs about interfaces show how to receive an interface as argument. But can my module define an interface and return a concrete implementation of it?

Functions defined in interface definitions must match the client-side API signature style. If they return a scalar value or an array, they must accept a context.Context argument and return an error return value.

What does this mean?

I was hoping to do this:

// My interface
type Check interface {
    DaggerObject
    Pass() bool
    Details() string
}

// My concrete type
type DummyCheck struct{}

func (c DummyCheck) Pass() bool {
    return true
}

func (c DummyCheck) Details() string {
    return "This is a dummy check which always passes"
}

// My function returning concrete implementation
func (m *MyModule) Lint() []Check {
    return []Check{
        new(DummyCheck),
    }
}

[sharp zealot]

Looks like it's not possible 😦

[strange arch]

Hello, is there a way to put a link to another function of the module in a function documentation published on Daggerverse? I tried adding a Markdown relative link to the function anchor (https://daggerverse.dev/mod/github.com/camptocamp/daggerverse/presentation@7389a62f140d142082d7f0bb0941818fc944564a#BuildResult.container) in the comment of my function, but it ends up linking to GitHub sources (https://github.com/camptocamp/daggerverse/raw/7389a62f140d142082d7f0bb0941818fc944564a/presentation/#BuildResult.container):

// Get a service serving the presentation
//
// See [container()](#BuildResult.container) for details.

[daring vine]

Hi Mark, sorry to bother you with that but I'm trying to run that

dir := dag.Arc().Unarchive(dag.HTTP(target_url))

With the url "https://github.com/fluxcd/flux2/releases/latest/download/crd-schemas.tar.gz"
But I got the error

Stdout:
invoke: failed to verify the script existence: input: arc.unarchive resolve: call function "Unarchive": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.Directory: input: container.from.withExec.withExec.withWorkdir.withMountedFile.withExec.directory resolve: process "sh -c arc unarchive 211ddf895dd412a333317279704b3fb470dd732e90259a0006d9691b5ee15062 211ddf895dd412a333317279704b3fb470dd732e90259a0006d9691b5ee15062" did not complete successfully: exit code: 1

Stderr:
format unrecognized by filename: 211ddf895dd412a333317279704b3fb470dd732e90259a0006d9691b5ee15062

I'm missing something which will certainly be more obvious to you 🙂 Could you please give me a hand?

[latent trellis]

Try this: dir := dag.Arc().Unarchive(dag.HTTP(target_url).WithName("crd-schemas.tar.gz"))

Chances are dag.HTTP doesn't preserve the file name

[leaden pagoda]

odd question maybe but... one thing you cannot "put a breakpoint" on is a pipeline. moving to code does give a sense that devs can breakpoint code. hit it with a debugger. this isn't actually the reality though right? since its submitted and running elsewhere it would probably need a complex way of hooking up a debugger. I don't know why devs would be writing crazy code in a language sdk though. buuuut. Is it clear that it's not supported? why would someone put a breakpoint and "step through the code" of a module they are building. container terminal solves the my container isn't configured properly let me play around in the container interactively but that's not a language debugger

[sharp zealot]

Opened an issue about it 🙏 https://github.com/dagger/dagger/issues/7582

[rocky harbor]

(cross-posting #typescript message for more visibility)

Just FYI for any TS module users, if you see an error containing Transforming JavaScript decorators, there is a workaround needed until our engine release next week, described here: https://github.com/dagger/dagger/issues/7583

[sharp zealot]

Does this ring a bell? (Go SDK, pretty vanilla code):

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "go" to get functions: call constructor: process "/runtime" did not complete successfully: exit code: 2

Stdout:
get fn name: returned error 400 Bad Request: {"errors":[{"message":"transport not supported"}],"data":null}

[sharp zealot]

Oh no. I'm hitting the same error as you... Did you ever resolve it?

Went away. So far only a fluke.

[daring vine]

That worked thank you again 🙂

[leaden pagoda]

https://github.com/dagger/dagger/blob/main/.goreleaser.yml#L32 There is completion scripts for pwsh but its not mentioned here (and pwsh isnt unique to windows). But i'm also not so heavy into the goworld myself yet. Is this an old artifact and not used or should it also have pwsh here?

[latent trellis]

Nope. It still happens from time to time.

[latent trellis]

https://github.com/dagger/dagger/issues/7592

[hazy turret]

Is there an alternative way to document the function parameters other than making them multi-line?

[sharp zealot]

I think it depends on the language, each SDK handles this in the most native way possible.

If you mean Go, I don't think so. No clean way to do it, or if there is - the Go SDK devs will be interested to hear it 🙂

[hazy turret]

He. I assumed so. I've got some colleagues not liking the multi-line style for the go SDK 😛 Thanks!

[sharp zealot]

I know... I do wish there was another way. Ask your colleagues if they have any suggestions, we're open to trying anything. The goal is to be as Go-native as possible (in the Go SDK)

I guess we could parse the function comments, and look for a bullet list there.

// Build the app
// - source: the source code of the app to build
func Build(source *Directory) *Directory {}

But that would be a lot of work, so can't promise we would get to that quickly, more urgent improvements to make first, we do accept PRs though 😛

Btw, SDKs are themselves Dagger modules, so anyone can create their own, or fork an existing SDK, and just dagger develop --sdk ADDRESS_OF_MY_CUSTOM_SDK_MODULE. Just in case you or your colleagues are feeling adventurous

A fun corollary is that when you install someone else's module, you may be using their custom SDK along with it, without having to know it. Which I find pretty cool

[hazy turret]

Oh, that is quite cool indeed. And quite easy to explore and iterate.. I would be more than happy to dive into contributing to the devEx as soon as we get the project up to speed 😉

[sharp zealot]

Does Directory.glob() support include/exclude? Something like this in gitignore:

*
!**/dagger.json
docs/**/snippets/**/dagger.json
**/testdata/**/dagger.json

To achieve the equivalent with glob, I'm stuck doing Glob("**/dagger.json") then doing the rest with handrolled go logic.

[sonic vessel]

How do I know which files are copy to context after applying view defined in dagger.json?

[sharp zealot]

I would love to have a new Directory.walk() function that works like this:

extend type Directory {
  """
  Recursively walk a directory and return a flattened list of its contents.
  Optionally, gitignore-style patterns can be applied
  """
  walk(ignore: [String!]): [String!]!

Example:

paths, _ := mydir.Walk(DirectoryWalkOpts{Ignore: []string{
  "*",
  "!**/dagger.json",
  "docs/**/snippets/**/dagger.json",
  "**/testdata/**/dagger.json",
}})

[sharp zealot]

You mean, how to debug the view to make sure it works properly?

[sonic vessel]

Yes. The current TUI report only it's copying but it's not report which files are copy.

[sharp zealot]

What is the return value of the function that receives your directory view as argument?

[sonic vessel]

I do use together with New constructor function like this:

func New(source *Directory) *MyModule { return &MyModule{Source: source} }

Then in dagger call, I use dagger call --source=.:default.

[sharp zealot]

You could write a small debug function, for example in Go:

func (m *MyModule) DebugView() *Terminal {
  return dag.
   Container().
   From("alpine").
   WithMountedDirectory("/dir", m.Source).
   WithWorkdir("/dir").
   Terminal()
}

Then call the function:

dagger call --source=.:default debug-view

You will get an interactive shell in a container with the view mounted

[sonic vessel]

Ahh, and you use some tools (df, dust) to find it, right? I'm looking for convinient way to find which files I need to exclude it. I come from Earthly and found the similar use case like this but they has --verbose option to debug which files it copy to the context.

[sharp zealot]

Maybe dagger -vvv will help?

[sonic vessel]

I tried but it still report only upload.

upload **REDACTED** from **REDACTED** (client id: vkg86mq7h4aflqwinkag3s4f5) (exclude: **/.node_modules, **/deps, **/_build, **/.lexical, **/target, **/.github)

[sharp zealot]

can you try running:

dagger call -m github.com/shykes/core container from --address=alpine with-directory --path=/data --source .:default directory --path=/data glob --pattern='*'

[sonic vessel]

can you try running:

[deft rain]

oh absolutely ❤️ this would be very appreciated. i wrote up the design thoughts behind why we went with pragmas over anything else in https://github.com/dagger/dagger/issues/6651#issuecomment-1940933051

[restive shore]

The new constraint Error: incompatible engine version: version v0.11.6 does not meet required version v0.11.7 is creating a chicken and egg situation for me. I am using a _EXPERIMENTAL_DAGGER_RUNNER_HOST with my custom dagger engine. I have a pipeline that builds and pushes this image to my custom registry. My pipeline always pulls the latest dagger CLI. Now I can't pull v0.11.7 CLI and use my existing engine image because the versions don't match. I can't pin to an engine version v0.11.7 because that's not available yet.

The only solution I see is to run the pipeline twice. Once pinned to v0.11.6 and build and push the v0.11.7 and then unpin it and run with v0.11.7. Anyone else see a better solution?

[honest hearth]

The new constraint `Error: incompatible

[sharp zealot]

@kind carbon @rocky harbor another note from dogfooding: support for 1) returning interfaces, and 2) using another module's type, have moved to the top of my list after contributing to our CI the last week

As well as a first implementation of context directory of course 🙂

If we had that I could make our testing & linting experience awesome

[rocky harbor]

returning interfaces requires the same work that is needed to support Self api calls, so while not a small effort per-se, it will get us a lot of bang for our buck https://github.com/dagger/dagger/issues/7582#issuecomment-2156227500

[sharp zealot]

What about using another module's type, for argument or return value? Is that harder or easier than returning interfaces?

[rocky harbor]

We've ruled that out as non-viable because it creates dependency hell problems that are impossible to hide from users. That's actually what motivated adding the existing support for interfaces. You can use them as arguments right now, which supports callers passing in types from their/other modules as args. We just need to lift the limitation on returning types that implement your own interface and it should invalidate any need for accepting/returning foreign types

[sharp zealot]

OK good to know!

Yeah major bang-for-buck as you mentioned

By the way @rocky harbor we talked about context dir today with @thorn moat @warped canyon @wintry prism and my idea of having the full git ref in the pragma, eg. +default="github.com/dagger/dagger#:sdk/python/runtime" was utterly crushed

[sharp zealot]

We settled on local path only:

• Relative path means "relative to this file". Eg +default="../runtime"
• Absolute path means "rooted to the current scope, aka your git repo or whatever" Eg. +default="/docs"

https://tenor.com/view/arod-alex-rodriguez-gif-24184203

(I think it will work beautifully, can't wait to use it)

[kind carbon]

Is that for a Directory argument in a function?

[sharp zealot]

Yes, with magical pragmas to default to a directory from the context. AKA a first implementation of "contextual modules"

Add +ignore pragma for extra magic 🙂

[sharp zealot]

Hey I might have a workaround: what if I took an interface as argument, then modified it with WithXXX calls (persisted in my own state), then my caller can get it back out of my state?

kind of convoluted but could work right?

func (sdk *MySDK) Lint(report LintReport) *MySDK {
  myLintIssues := do_my_thing()
  for _, myIssue := range myLintIssues {
    report = report.WithIssue("my sdk", myIssue.Text, myIssue.Position)
  }
  return sdk.withLintReport(report)
}

Would that work?

[amber pilot]

go: dagger.io/dagger@v0.11.7: parsing go.mod:
module declares its path as: github.com/dagger/dagger
but was required as: dagger.io/dagger

Ops

[rocky harbor]

Possibly, depends on specifics, happy to look at the more specific situation too. The only thing not possible atm is for the caller to convert back to its specific type once it has become an interface, i.e. no equivalent of x, ok := foo.(*someType). But not sure if that’s what you needed exactly. Otherwise it sounds possible

[sharp zealot]

I found a simpler way 🙂 Going to keep it all in the same module for now. Won't be able to reuse it in other projects (yet) but buys me several weeks

[plucky ermine]

Hm, I just noticed that when using python date.today() on my computer is not the same as date.today() inside of the python runtime - I assume its because timezone is not set or passed in by default, but this feels like it has potential to lead to a lot of bugs in modules

[wintry prism]

especially around things that use certs or other time-based credentials. Is UTC not the same?

[plucky ermine]

yeah UTC is the same - so perhaps its only a problem for more edge casy things -- but in my case I was comparing two dates to match month an day

the functions works perfectly most of the day 😅

[uneven shadow]

I've been using dagger cloud since yesterday. But have a couple of runs where I do get a trace url in my CI output but where the traces take a very long time to show up and don't get marked as finished. So far only seems to happen when the CI run fails(not using modules/functions). Is this a known issue?

[latent trellis]

I'm not a sure what's the situation with 0.11.7. Is the retagged version safe to use? I've already upgraded to 0.11.7 before it was retagged, so I'm not sure what I'm supposed to do now. Is 0.11.8 around the corner?

[deft rain]

0.11.7 is good to use - but if you're manually importing the go sdk (from non module code), then i would use v0.11.8-rc2 for that

[latent trellis]

Thanks!

[thorn moat]

traces not showing up / finishing

[restive shore]

How do you run golanglint-ci on a daggerverse repo? The linter seems to not work with go.work. The only solution I can find currently is to to cd into each module folder and run it. I see this https://github.com/golangci/golangci-lint/issues/2654 but there's no real resolution there. If I run go list -f '{{.Dir}}/...' -m | xargs golangci-lint run it actually lints the generated code, which I don't want it to do.

[deft rain]

ah, yeah, this is a bit annoying agreed - the right way to use golangcilint is to go into each go module, and run it in there as you've worked out

you'll need to generate code to be able to run the linter - otherwise, the code will just entirely fail to parse
if the linter is producing warnings from the generated code, you can exclude those warnings from golangcilint: https://golangci-lint.run/usage/false-positives/#exclude-issues-by-path

alternatively, i'm not entirely opposed to introducing //nolint directives to our generated files

[restive shore]

Oh I didn't even consider having to force the code generation in order to do linting. It was working fine locally but I didn't realize I have all the codegen already in place.

so basically, the process is

• cd into directory
• dagger develop
• golangci-lint run --skip-dirs=internal --skip-files=dagger.gen.go
• repeat for every module

[deft rain]

yup

[restive shore]

Can the Dagger CLI do something to help here? Like a way to do codegen for all modules from the parent folder? Maybe dagger develop can look for dagger.json recursively and generate code if invoked from the parent folder?

[sharp zealot]

we were just discussing that 👍

[restive shore]

created an issue. Feel free to close it if you already have one 🙂 https://github.com/dagger/dagger/issues/7668

[sharp zealot]

Thank you! I've been meaning to create it

[hoary geyser]

I had the pleasure of meeting Paul (zoneinfo) Gannsle at PyconUS last month. Timezones are dark magic. We may even want to consider special linting rules to look for anti patterns in Python modules.

[restive shore]

Running codegen programmatically

[latent trellis]

a justfile (or any other similar tool) can work nicely around that.

[restive shore]

For sure. I'd rather not have any other dependencies though. If I can create a dagger function or use the CLI I'd rather do that. Daggerize all the things.

[sharp zealot]

we have the same problem in the dagger CI 😁

[elder kiln]

suppose you´d want to implement passing containers to other dagger functions through bash pipes

where would you even begin?

[quick wind]

Question I've just had that I couldn't find an answer for: are there any SDK benchmarks?

[strong ingot]

Question I've just had that I couldn't

[sharp zealot]

Like this? 😇 #daggernauts message

[elder kiln]

beautiful

[leaden pagoda]

Sooo, forgot to mention on weekend i was playing around and did a dagger install with my docker-desktop with a helm install, i managed to configure the dagger client to hit my docker-desktop k8s single node dagger engine. but it did not play/perform anyhting like the container just sitting in normal docker-desktop. Should I be doing additional configuration here or is something sounding fishy? It also seemed to have some networking issues, but i have no idea why that would happen. My install is pretty basic: https://github.com/pjmagee/fleet-infra/blob/main/infrastructure/docker-desktop/dagger/release.yaml.

im gonna try and run something simple on it and hook it to cloud and see if ican share smth there to help debug my issue

[elder kiln]

(is this on some rev?)

[sharp zealot]

I wish... It's in my head for now

I did start a POC of a bash-compatible interpreter for the Dagger API. But didn't have time to finish it

[restive shore]

I wanted to ask this question here before I open an issue. In go, the convention for creating a "constructor" is New + "Struct name". For example if my module struct is named CI, then my constructor would typically be NewCI. Would it be possible for dagger to also identify and invoke the constructor in the form of "Name"+"Struct"? I couldn't really find reference docs for that convention but I see it used as a convention in most Go projects.

[sharp zealot]

So something like: func CIStruct() *CI ?

[restive shore]

It's for the constructor. Right now it's just New so something like func NewCI() *CI

[sharp zealot]

OK got it. I was trying to parse "Name"+"Struct"

[restive shore]

No worries, I didn't know how best to explain it

[strange arch]

Actually, that’s not exact. In https://go.dev/blog/package-names, it’s stated:

Simplify function names. When a function in package pkg returns a value of type pkg.Pkg (or *pkg.Pkg), the function name can often omit the type name without confusion:
[…]
A function named New in package pkg returns a value of type pkg.Pkg. This is a standard entry point for client code using that type

[restive shore]

Thank you for pointing me to this. That makes sense

[leaden pagoda]

is an 18gb directory okay to pass to dagger

[restive shore]

"okay" is subjective here. Can you pass in such a directory to Dagger? Sure! Should you? That depends on whether dagger behaves in a satisfactory way when you do. It might get slow. IDK if the dagger team has tested such big dirs.

[latent trellis]

Here is an interesting challenge: in many cases, we use CLIs to interact with third-party APIs from Dagger, but with Functions, we can use native SDKs instead.

These SDKs often exchange a set of credentials for a temporary access token. CLIs usually cache these token in some local store (which may or may not be safe), but the point is they do not reauthenticate in every single call.

That's a bit of a challenge with functions, because you cannot "save" these preconfigured SDK client instances (they are not json serializable).

So I see two paths forward:

1. Save the credentials as Secrets and use them every single time to reauthenticate for a temporary access token.
2. Save the temporary access token (if possible) and use it to configure the client.

The challenge with option 2 is these tokens may expire which means you either solve token refreshing or just don't care about it (eg. the expectation for the function is to be used within a single CI/CD run and the token should be valid for that duration).

Any thoughts?

[deft rain]

So to me, this feels a bit like some logic you'd want to wrap in an object and pass it around (using an interface i guess).
You'd have a method like GetTempAccessToken() *Secret, which would get a temporary secret.
The problem is, how do you know when to reauth? You need some more persistent state for the object - but I guess that's the issue? If you modify the object in GetTempAccessToken, it won't persist, because you're returning a Secret.

I do wonder if we should maybe have semantics for "mutable" methods - ones that can change the base object, instead of chaining.

Or maybe there should be a "scratchpad" - a way to get/set arbitrary persistent data in a module

Having multiple returns would also solve this (sort of, GetTempAccessToken could return the base object, as well as the secret)
But this feels much more fiddly

[latent trellis]

It's not obvious to me how much internal semantics of a library should leak into a Dagger library. At the moment I'm leaning towards just reauthenticating in every call.

[restive shore]

force re-auth every call is what I am doing currently as well.

[plucky ermine]

I feel like we ran into this with working with infiscial lately @vast igloo can you remind me which pattern did you end up settling on?

[latent trellis]

That's exactly where I ran into this question, but I imagine other providers are also affected (AWS, GCP, etc)

[vast igloo]

I currently have my root module take a Secret that I use to authenticate to Infisical.
That secret is then passed into every downstream module that I have.
I created a singleton instance around the Infisical client, and I initialize it in each @func

class Root {
  @field() auth: Secret
  constructor(auth: Secret) { this.auth = auth }

  @func() doStuff() {
    Infisical.initialize(this.auth)
    // do stuff with secrets here
    return new SecondModule(this.auth)
  }
}
class SecondModule {
  @field() auth: Secret
  constructor(auth: Secret) { this.auth = auth }

  @func() report() {
    Infisical.initialize(this.auth)
    // do stuff with secrets here
    return 'done'
  }
}

[latent trellis]

Yeah, that's option 1 from the above list.

[vast igloo]

yup ... its tedius, but it works

[restive shore]

Is anyone running either renovate or dependabot on their dagger modules? Since a bunch of those dependencies are controlled by dagger how do I go about it? When I run renovate, it identifies packages that needs updating but do I update them or wait for next dagger release? I tried to update dependencies and when I run golangci-lint to validate, I get an error

Can't run linter goanalysis_metalinter: inspect: failed to load package : could not load export data: no export data for \"github.com/99designs/gqlgen/graphql\""

I wonder if this is because only some of my modules have the most up to date dagger develop dependencies.

[latent trellis]

What's the policy for updating dependencies in Dagger?

Vulnerability scans are getting triggered by https://github.com/vektah/gqlparser

[chrome pilot]

Dependabot triggers the upgrade for core and all the SDKs -- I'm trying to figure out which version is affected and why it hasn't been upgraded

[thorn moat]

Is it this? https://github.com/vektah/gqlparser/commit/d457fc08189db3b7bee997060cfcb01717cdbbec - nothing shows up in our engine scan CI job atm

[chrome pilot]

Yeah, somehow the engine was updated to latest, the Go SDK is still running on an older version

I'm trying to figure out why dependabot didn't kick in for the Go SDK

[chrome pilot]

https://github.com/dagger/dagger/pull/7674

[thorn moat]

oh cool, so it's already dealt with (edit: just read the other msgs, i see, still need it for Go SDK)

[chrome pilot]

Yep. I see the SDKs have a ignore: update-types: ["version-update:semver-patch"]. @strong ingot I see you added that, was it because of noise?

[strong ingot]

Yes.

[chrome pilot]

Ok we might want to re-assess for security patches (AFAIK we can set a separate dependabot policy for security updates, we could remove the ignore on that one). In the meantime, I've manually bumped the dependency https://github.com/dagger/dagger/pull/7695 /cc @latent trellis

@latent trellis we have the v0.11.8 Go SDK coming out tomorrow, this should make it inside

[strong ingot]

I'm all for doing these bumps weekly, as long as we group the bumps per language / component, rather than end up with individual PRs for every single dependency bump.

[chrome pilot]

@strong ingot I think @deft rain already made the change to group them together a few months ago: https://github.com/dagger/dagger/pull/6811

looks like we can get rid of the ignore with that?

[strong ingot]

Let's do it!

[chrome pilot]

https://github.com/dagger/dagger/pull/7697

[chrome pilot]

set the milestone to v0.12, we don't want to mass-update dependencies right before the v0.11.8 release

[restive shore]

This conversation is related to my question. How do consumers (dagger modules) handle dependency updates?

[sharp zealot]

For discussion: https://github.com/dagger/dagger/issues/7707

[sharp zealot]

Small papercut: https://github.com/dagger/dagger/issues/7709

[restive shore]

This seems to be generating a LICENSE file for every dagger develop. Based on the PR description that is not intended? https://github.com/dagger/dagger/pull/7658. I run dagger develop a lot and every time I have to delete the LICENSE file that it creates. v0.11.8. I am not using the --sdk flag

[sharp zealot]

Yeah at the very least there should be a flag for disabling license creation. And possibly that should be the default

[thorn moat]

ah that PR was for changing dagger init to not generate the license, not dagger develop. If your module configures an SDK, develop will indeed create a license file still, regardless of whether you passed --sdk on that specific invocation

[restive shore]

That is not desirable for my use case. I don't push LICENSE files per module. I assume a lot of others doing "Daggerverse" won't either. Need a way to disable that behavior.

I actually see an issue for this already and a discussion! I'll post there too 🙂 https://github.com/dagger/dagger/issues/7703

[kind carbon]

Correct, that's a bug. It should create it only when --sdk is used.

[storm lark]

Hello everyone... I'm trying to deploy the image to my internal registry and I'm getting an error:
us-central1-docker.pkg.dev/project/repo/app: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://us-central1-docker.pkg.dev/v2/token: 405 Method Not Allowed

I've checked the buildkit code and it does seem like it should handle 405 in cases where we do provide a user, right?

        resp, err := authutil.FetchTokenWithOAuth(ctx, httpClient, nil, "buildkit-client", to)
        if err != nil {
            var errStatus remoteserrors.ErrUnexpectedStatus
            if errors.As(err, &errStatus) {
                // Registries without support for POST may return 404 for POST /v2/token.
                // As of September 2017, GCR is known to return 404.
                // As of February 2018, JFrog Artifactory is known to return 401.
                if (errStatus.StatusCode == 405 && to.Username != "") || errStatus.StatusCode == 404 || errStatus.StatusCode == 401 {
                    resp, err := authutil.FetchToken(ctx, httpClient, nil, to)
                    if err != nil {
                        return nil, err
                    }
                    return toTokenResponse(resp.Token, resp.IssuedAt, resp.ExpiresIn), nil
                }
            }
            return nil, err
        }

My code is just doing this:

        WithRegistryAuth("us-central1-docker.pkg.dev", "_json_key", credential).
        Publish(ctx, registry)```

credentials being the json secret key.

anyone having the same issue? thanks 🙂

[storm lark]

Nevermind everyone... seems like I can't use my own credentials generated with gcloud auth application-default login as a credential we need to use an actual service account's credentials. False alarm, keeping it here for future ref is it's ok 😉

[sharp zealot]

@chrome pilot is your debug PR merged in main? 🤩

[chrome pilot]

Only 1 out of 2

[sharp zealot]

Best practice note: I'm going to adopt the following best practice for my own contributions (including to Dagger's CI):

• When daggerizing a project, create a dagger module in ./dev
• Make calls with dagger call -m ./dev
• Open a PR to dagger so that dagger call with no argument falls back to ./dev if . is not a module.

A possible stopgap, if dagger call -m ./dev is not acceptable:

• Initialize the module at the root of the repo, with "source": "dev"
• Once the PR is merged, revert the stopgap: remove the "source" field, move dagger.json to ./dev/dagger.json

--> For example, in dagger/dagger this means renaming ci to dev. I will open a PR to do this.

[sharp zealot]

As I daggerize the various linting tools in our CI, I am discovering the wonderful world of JS development, where packages can install binaries, and in the case of eslint (the linter we use), the binary cannot lint anything unless you also install the eslint javascript package. Leading to inextricable entanglement between system tools, and your project's libraries. A mess.

In theory you can load a project-independent config file into the tool, but you have to be careful... as that configuration file is itself a javascript file! so it could have random dependencies. Amazing.

Once we fix CI/CD we may have to take a look at web tooling because wow

My dilemna now is that I'm not sure how to encapsulate eslint into a clean dagger module.

• I can run a globally installed eslint binary in a container. But it won't work unless the source directory also has the @eslint/js package installed
• Do I just pass that error, and require that you install the package yourself before using the dagger module? That makes the dagger module less useful, it's a weird abstraction leak
• Do I automatically install the package in the mounted source dir? This makes more sense, but I would have to handle version mismatches, in the case where the package is already installed.

Three ways to handle versioning mismatches:

1. Always install my own version of the library, possibly overwriting pre-installed one. In this case, my module might not yield the same result as the dev's inner loop. calling eslint directly
2. Call my own version of the binary, but honor the pre-installed library if it's there. This creates the risk of mismatches between the binary and library (I'm going to make a wild guess, and assume JS tooling does not reliably guarantee that you can decouple binary and library from a package)
3. Always use pre-installed binary+library if it exists, only install my version as a default. This means I'm essentially wrapping an unknown version of the tool. So I can't guarantee that it will work.

Seems like 3 terrible choices to me. All because JS tooling tightly couples libraries and binaries in a way that makes cross-platform integration super annoying

[leaden pagoda]

LOL

The node/js world is soooooooo bad, i really dislike what happened to the web in general with js development in the last 12 years.

[honest hearth]

Looks like the Deno guys also felt some of your pain: https://github.com/bartlomieju/eslint_binary

[sharp zealot]

https://tenor.com/view/am-i-dreaming-rand-ridley-inside-job-pinch-me-dream-come-true-gif-23666359

[honest hearth]

It does look like a bit of a hack in its current form (albeit a pretty simple one) but if you could reliably compile it, maybe it could work?

[leaden pagoda]

@latent trellis ive been using your github module, it's awesome, thank you. I was going to also add it into another project which I am helping contribute on. But i thought for a moment that in the dagger.json, its fixed to a commit. And i am thinking likely at some point, dagger install would also need like a dagger upgrade command? So people can see if modules have been updated recently? Is anyone else thinking about module dependency updates and how to be notified better if there are changes there?

[latent trellis]

Thanks!

For the moment, I just do this:

dagup () {
    cat dagger.json | jq -r '.dependencies[].source' | cut -d '@' -f 1 | xargs -L1 dagger install
}

[daring vine]

Hi everyone, I'm currently studying running on Kubernetes and the different options. Just wondering if that could make sense to mount an EFS volume into the dagger-engines (Daemonset)? Looking at the Helm chart I can see these volumes:

          volumeMounts:
            - name: varlibdagger
              mountPath: /var/lib/dagger
            - name: varrundagger
              mountPath: /var/run/buildkit
            {{- if .Values.engine.config }}
            - name: dagger-engine-config
              mountPath: /etc/dagger/engine.toml
              subPath: engine.toml
            {{- end }}
      terminationGracePeriodSeconds: {{ .Values.engine.terminationGracePeriodSeconds }}
      volumes:
        - name: varlibdagger
          hostPath:
            path: /var/lib/dagger
        - name: varrundagger
          hostPath:
            path: /var/run/dagger

What are the ones that should be shared? Wdyt? Are there limitations that I don't see?
Here's the big picture

[sharp zealot]

My guess is that if you mount the Dagger buildkit cache in EFS, 1) it will be slow, and 2) buildkit doesn't support concurrent writers to the same state directory, so you will probably have data corruption issues

[daring vine]

Ok thank you Solomon 🙂

[sharp zealot]

I wish this were possible. We're working on solutions to distribute cache data across machines.

[leaden pagoda]

has anyone had any odd behaviours with git submodules? that one is hard to test easily.

obviously locally, you do a build and it works. you were coding, doing things, and the submodule is cloned. But you run it in CI and you have to run a git submodule

also the dir is just swallowed up, so should i be removing this and doing a git clone of the submodule in dagger, that would likely be the best way to simulate it in CI environments

[daring vine]

Hello, it's me again, hope you're having a great week-end. While waiting for an improvement on distributed caching, here's another proposal that is not part of the options in the docs (maybe for a good reason...):

• A deployment made of a unique dagger engine
• Make use of the nvme storage mounted as ephemeral storage into the pod.
• Dagger engine exposed as a service.
• The pod will be scheduled on an "on-demand", with the annotation "karpenter.sh/do-not-disrupt" and a pdb.

Pros:

• A unique cache
• The fastest storage
• Simple architecture

Cons:

• Vertical scaling
• In very rare cases (node crash), we assume loosing the cache.

My first test is successful but I might be missing something.

kubectl port-forward -n tooling svc/dagger-engine 8080
_EXPERIMENTAL_DAGGER_RUNNER_HOST="tcp://127.0.0.1:8080"
export _EXPERIMENTAL_DAGGER_RUNNER_HOST

dagger query <<EOF
{
    container {
        from(address:"alpine") {
            withExec(args: ["uname", "-a"]) { stdout }
        }
    }
}

EOF
{
    "container": {
        "from": {
            "withExec": {
                "stdout": "Linux buildkitsandbox 5.10.218-208.862.amzn2.x86_64 #1 SMP Tue Jun 4 16:52:10 UTC 2024 x86_64 Linux\n"
            }
        }
    }
}

This is worth noting that the node is configured with a script "/usr/bin/setup-local-disks raid0" which builds a raid0 array and mounts the container's filesystem on it. This allows to define ephemeral storage with access to a high capacity and performant storage.
For more detail please check my PR here: https://github.com/Smana/demo-cloud-native-ref/pull/311

Note that I'm currently writing a blog post on Dagger, I hope I'll have a first version before the summer break 🤞 https://blog.ogenki.io/

[lime comet]

Hey @daring vine! Its an interesting idea! We've been discussing how to run/scale dagger in production in this issue: https://github.com/dagger/dagger/issues/6486. I think you will find Gerhard's comment particularly insightful: https://github.com/dagger/dagger/issues/6486#issuecomment-1910551524. It describes very well the differences between a horizontally and vertically scaled approach for dagger engines. We are working on a few initiatives that will help both types of deployment so some of the comments there might be just a bit out of date

[daring vine]

thank you, will do @lime comet

[sharp zealot]

@daring vine I think the TLDR is that seems like a sound architecture. A few things to keep in mind:

• The interface between CLI and engine is privileged. So make sure untrusted processes (whether on the same machine or not) can't access that tcp port.

• How well a single machine (even vertically scaled) can handle N CI jobs, really depends on your workload. So I recommend keep an close eye on CPU and IO stats of that machine during your experiment

• Are you considering adding a 2nd engine at some point? And if so, do you have a plan for spreading the load?

[daring vine]

The interface between CLI and engine is privileged. So make sure untrusted processes (whether on the same machine or not) can't access that tcp port.
I'll define fine grained network policies

Are you considering adding a 2nd engine at some point? And if so, do you have a plan for spreading the load?

A second or more engines would mean having different caches, duplicating things. I think that this architecture is sufficient for many small companies (to be confirmed though), I know that depends on the biggest instance in AWS. Does it make sense? (I didn't read the issue yet :p, I'll do that when I'll be back from the office )

[daring vine]

Just to keep you updated, I finally succeeded in testing the architecture above 🎉
https://discord.com/channels/707636530424053791/1254887505162141766
I read the issue and, there's a point I didn't considered carefuly: The pricing of the long running instance. For my first experiment, I think that this is acceptable but I'll follow closely the topic 🙂
Thanks for your support 🥰

oh, one last question: Would that be hard to make the dagger engine exposing a tls certificate? I'm gonna dig that way to enhance my setup (even though I'll have fine grained network policies)

[wintry prism]

@rose hinge when you get a chance, have you seen errors like this with your node module?

return dag
.node()
.withSource(source, {persisted: true})
.withYarn()
.install()
.run(["setup"])
.run(["dev"])
.container()

[sharp zealot]

Today I used dagger from main, hoping to use the new Container.terminal() and Directory.terminal() calls, but it didn't seem to work. Are these known to work at the moment?

Oh I guess it is working. I was just confused by the different prompt

[rocky harbor]

I saw people freaking out on Twitter about how good Claude 3.5 is at writing code in combination with it's 200k context window size and gave it a quick try myself by uploading our entire core/schema/ package from the dagger codebase and asking it to add an implementation of a new engine API under query with stubbed out implementations of functions and it did indeed pretty much nail it 🦾 Saved a lot of boilerplate writing.

This is screaming for a dagger module. Our primitives for loading/exporting directories in dagger call should work really well with it I think.

Before this the only AI coding tool I've found genuinely useful is Copilot, but only when it comes to autocompleting single lines of code at a time (90% of my usage is just autocompleting error message annotations or log statements). I've been really wanting something that is capable of handling larger chunks of boilerplate and the types of refactors that are like "change the type of these args on these functions" or "add this new arg to these functions" which don't require that much deep thinking but are incredibly tedious+time-consuming when done across many files. I think this may finally be that.

My next two weekends are booked but some weekend I'll give modularizing this a try if no one else has already 😄 Think it could genuinely be a huge development time save for dagger and any one writing code

[sharp zealot]

Amazing. I've been wanting to try something like that, but was deterred by the required hacks. Like how do you go from a directory of code, to a flat prompt with context, to a flat response, to a changed directory? Upload a text archive format of some sort, and have it print a diff?

And 1000x yes to Dagger being the perfect tool to wrap the hacks into repeatable functions 🙂

[rocky harbor]

Yeah for my manual experiment I used this feature Anthropic just added called projects that let you upload a bunch of files to use as context, which helped on the input side. On the output side I was only asking for a single new file so it was easy here, but would need to figure something out for outputting whole directories.

One possibility would be to ask for a git diff output of the changes and/or patch file. Provided it knows that format well enough that might do the trick? The dagger module can ask for the diff and then either return that to the caller or apply the diff to the loaded Directory and export the dir back to them. I'm a noob in this whole space so I'm sure there's other tricks possible too

[sharp zealot]

i've used diff output with gpt4 and yeah it works great

for input I'm thinking: https://pkg.go.dev/golang.org/x/tools/txtar

[rocky harbor]

Yeah I mean it seems like these tools understand all sorts of file formats and even understand base64 if you provide it in the prompt, so there's probably a bunch of approaches like that

[sharp zealot]

@thorn moat @rocky harbor I'm about to embark in my first serious experiment with interfaces... Now that I've discovered the wonderful feature that is "loosely coupled interfaces" (not sure what to call them), and the magical Asxxx.

I do have one question. If my interface defines functions that return other interfaces, will Asxxx still work?

[thorn moat]

I'm pretty sure Erik went hard on this and it checks for subtype compatibility and all the fancy things you'd expect

i.e. Foo() Bar implements Foo() Baz if Bar is a subtype of Baz, so whatever implements Foo() will satisfy an interface that expects Baz for its return value

[sharp zealot]

Yeah but what about:

type Foo interface {
  bar() Bar
}

type Bar interface {}

Does MyFoo have a chance of implementing Foo?

[rocky harbor]

Yep that should work, there's a big module with all the various cases of interfaces as arg/return types here: https://github.com/sipsma/dagger/blob/0290df5df67b2514a0dd045652b0083017a86b14/core/integration/testdata/modules/go/ifaces/test/main.go#L8-L8

Which should show what's currently possible. It includes an interface returning other interfaces (and returning itself, etc.)

[sharp zealot]

(sorry using regular Go interface notation for simplicity)

FYI @thorn moat I'm taking a crack at a Checker and Fixer interface, to generalize the linter / code formatter / codegen checker use cases, and try to deliver a neat dagger call fix and dagger call check that rolls it all up

[sharp zealot]

https://tenor.com/view/pumped-woooo-ric-flair-go-hard-go-home-gif-15455267

"Erik went hard" another T-shirt opportunity right there 🙂

[twin atlas]

Hi folks! quick question, is there a way to see my Dagger engine status to see the details of which Dagger engine I'm connected to or what are my other Dagger engines I configured that I can connect to? I'm looking for a feature similar to a docker context.

Let's assume that I've configured my Dagger engine to connect to the pod on Kubernetes, so, next step I'd like to ensure that the connection works, but I don't have a command to do that in Dagger CLI

[twin atlas]

the way I use to do that is just by calling the @sharp zealot hello module dagger -m github.com/shykes/daggerverse/hello@v0.1.2 call hello

[wintry prism]

Proposal for discussion:
https://github.com/dagger/dagger/issues/7777

[deft rain]

🎉 let's do it 🙂

[wintry prism]

When I was making a simple test module for 👆 I used two try catch statements to guess main and then master for the git branch of a repo (versus parsing a format with #main or @main). I found out a few things:

• I had to sync() my Container state to see if I'd hit an error in order to catch it at the right time.
• if the try statements were at the same level of depth, then Dagger tried both paths in parallel and sometimes the second one finished and threw an error that ended everything. (maybe?)
• so I nested the second try in the catch of the first one to "chain" them: https://github.com/jpadams/testgit/blob/main/dagger/src/index.ts#L22-L39

I'll look to expand on this article: https://docs.dagger.io/manuals/developer/error-handling

[spiral whale]

I wish I could see more information about module when I publish it to daggerverse. I just tried to publish my new moduleto daggerverse and it's stuck in crawling state(here: https://daggerverse.dev/crawl/github.com/aweris/daggerverse/k3s@c45c2f2ea68ba3a7ca035d3c5b2b7585a3e82fa0) and i can't see why it is not show-up or what is wrong.

[sonic vessel]

I found the display issue when using attachable terminal on dagger main with the Windows Terminal. The text seems to do some indenting every times it has (, make a little bit hard to read but it's not a blocker issue.

[wintry prism]

I wish I could see more information

[sonic vessel]

Is DAGGER_SESSION_PORT and DAGGER_SESSION_TOKEN are forward to the container in the function automatically? I try to make dotnet sdk unit test that connect to dagger though those variables works inside the dagger function and discover that it can run without requiring dagger cli install.

[wheat river]

Hey folks 👋 -
I usually release my modules using release-please, it works relatively nicely (and I've been using it for a while already), however, for a mono-repo as many of you likely are using as well it's more handy to release the modules independently. I've noticed that @thorn moat already is using this approach (module/vx.y.z) so i'm curious if you're using a particular tool to tackle the versioning/releasing of dagger modules.
In my case, release-please fits great in my workflow. The changes are merged, a new pull-request is created and updated if there are more changes that got mergerd, and when I'm ready, I can release and publish (https://github.com/Excoriate/daggerverse/blob/f920b34ad9e208b5216b77f72e78995ed506497c/.github/workflows/release.yaml#L39-L39) — the only drawback is that I'm unable —yet— to make it works with the daggerverse mono-repo approach. Release-please seems to support it, but there's no much documentation of this feature (specially for Go - it's more extended for NPM/NodeJS-related repos).

[twin atlas]

Today is the day where Koray Oksay is going to present Dagger in KCD Munich ^^ We are super excited about this and this is the demo we created for this presentation, hope you like it https://github.com/developer-guy/kcd-munich-2024-demo

[restive shore]

Are there examples of testing dagger modules in the dagger repo? I don't see any in the dev folder.

[wheat river]

I'm not sure if you're referring to this, but I usually test my modules with an approach inspired by @latent trellis 's. I have a tests/ dagger module that's used exclusively for testing purposes. Works quite well. In addition, I've built a tiny rust code-gen (clap-based CLI) tool that generates a module (https://github.com/Excoriate/daggerverse/tree/main/module-template) and its test with some common (I'm opinionated here) functions that most of my modules needs.

[restive shore]

Thank you for sharing that! Yes I am exploring Mark's testing strategy. I was wondering if the dagger team had an official strategy/guidance for testing modules. Module testing is a bit tricky. By doing the module tests by executing a separate tests module, you are essentially doing functional tests and you may be missing out on some of the benefits of the go testing framework. Unless someone has figured out a way to tie the two together. You also have to handle and implement concurrency.

For generating the module Mark uses a Justfile. Your rust CLI based approach is also interesting. However, I think eventually dagger CLI would bet the best tool to generate tests.

[wheat river]

I also use JustFile (I was previously using extensively Taskfile, but I prefer Justfile simplicity) - check the GoTest module (https://github.com/Excoriate/daggerverse/tree/main/gotest) for example. I also miss a bit the GO (built-in) approach for testing specific aspects of my code. My tests are indeed a bit more functional (testing my commands and API(s) E.g.: WithSource()). Perhaps @sharp zealot you can tell us more if you guys are thinking in a dagger-validated module's approach that can cover a recommended way to write tests for Dagger modules? — if there's a proposal in that direction it'd be awesome to contribute to it. I'm building Dagger modules quite frequently, and I'm using them in production these days 🙂 .

[sharp zealot]

Testing modules (part 2)

[dense canyon]

@restive shore here's another pattern I've been playing with recently that uses the native SDK testing framework for that (thread)

[craggy elbow]

How do folks decide when to use a native language sdk vs “shelling out” to a container/CLI? Right now I’m working on something where I want to hit the GitHub API and collect a list of PRs, do some filtering, and then possibly create a new PR. This would be pretty straightforward to do with the gh CLI but when it comes to interacting with the response, unmarshaling it, etc. seems like using go-github is probably a more reliable choice?

My thinking right now is if I want to fire off a command and return stdout/stderr but not do much to it, I probably would lean towards the CLI/container execution. But if I’m trying to read in data, modify it, and then act on it, I probably want to use a native sdk?

[plucky ermine]

How do folks decide when to use a native

[daring vine]

Hi everyone, I don't know if this is the proper channel for asking: I would like to ask experts to read this blog post (https://blog.ogenki.io/post/dagger-intro/) that I'm going to communicate about on tuesday. Let me know if you see any obvious mistake please 🙏 . Have a great week-end

[wheat river]

Loved it. I'm not an expert, but I've been using it and developing modules for a while already. I think it describes it nicely.

Basically, Dagger is a tool that allows us to define tasks using our preferred language and make that code portable. In other words
The way I see it is that Dagger is setting the corner stone for a new kind of (DevOps?) Primitive: Reusable/Portable/Sandboxed Functions.

[daring vine]

Thank you Alex for your feedback, I'll probably reword that. Have a nice day!

[latent trellis]

@sharp zealot @plucky ermine we talked about the idea of returning "artifacts" from functions.

Let's say I have a lint step that returns a list of violations (in some format) and maybe some logs that are useful for debugging.

How would an artifact look like in this case?

I'm thinking maybe a struct like this?

type LintResult struct {
    Violations []Violation
    Logs string
}

I don't really know why, but returning Logs as string seems a bit weird.

Also, it requires a number of API calls, because I'd either need a file or stdout for the violations and stderr for logs.

Last, but not least: how would this show up on Cloud?

[sharp zealot]

I started doing exactly this. There are unknowns for sure.

• How to print in CLI: we merged a PR for dagger call to print the fields of the returned object. But it doesn't yet walk nested arrays & objects . That's the next step to properly print your result IMO
• How to show on Cloud: still TBD we are actively discussing this with @silent @thorn moat
• logs as string: I haven't needed this. I configure the lint tool to produce a structured report, and parse that for all the information I need.

For an example, see https://github.com/dagger/dagger/pull/7803

[quick wind]

@wintry prism Any chance you can update https://github.com/jpadams/daggerverse/tree/main/infisical to latest 🙏

[restive shore]

Love this change and it'll make gopls happy! But it's going to be a brutal refactor for module authors. But better now than later. https://github.com/dagger/dagger/pull/7831

[deft rain]

@thorn moat pointed out that you should be able to use . "dagger/<yourmodule>/internal/dagger so you can do it bit-by-bit if you like

i'll add that note into the release notes when i put them together, since i think that should make it a lot easier potentially

[restive shore]

true. but I'm going to rip out the band-aid 🙂

[sharp zealot]

First use of dagger call -i in a real-world scenario... 🤯 🤯 🤯 🤯 🤯 🤯 🤯 🤯 @chrome pilot

Load a module, a deep dependency of that module doesn't build, boom I am in the container ready to build that deep dependency, I just run go build and I see the error, can play with all the files... Would never in a million years been able to reproduce such a deep environment

That's an interactive container pre-built by dagger, run by dagger, building a dagger module which will then be loaded by dagger... and I can just go build and not worry about a thing🤯

This is especially helpful because of how bad our regular error messages are

The regular error

      ✘ Dirdiff.assertEqual(
          a: ✔ Directory.directory(path: "dev/dirdiff"): Directory! 1.5s
          b: ✔ Container.directory(path: "."): Directory! 0.0s
          paths: ["go.mod", "go.sum"]
        ): Void 1.9s
      ! call function "AssertEqual": context canceled
        ✔ Container.withExec(args: ["diff", "-r", "a/go.mod", "b/go.mod"]): Container! 0.1s
        ✔ Container.sync: ContainerID! 0.1s
        ✘ Container.withExec(args: ["diff", "-r", "a/go.sum", "b/go.sum"]): Container! 0.2s
          ✘ exec diff -r a/go.sum b/go.sum 0.2s
          ! process "diff -r a/go.sum b/go.sum" did not complete successfully: otel tcp proxy listen: context canceled: context canceled

🤷

The same command in interactive container

/mnt # diff -r -u a/go.mod b/go.mod 2>/dev/null
--- a/go.mod    2024-07-10 21:01:58.201330090 +0000
+++ b/go.mod    2024-07-10 21:21:18.378626013 +0000
@@ -27,14 +27,14 @@
     google.golang.org/grpc v1.64.0
 )

-require go.opentelemetry.io/proto/otlp v1.3.1
-
 require (
-    github.com/Masterminds/semver/v3 v3.2.1 // indirect
-    helm.sh/helm/v3 v3.15.2 // indirect
-    sigs.k8s.io/yaml v1.4.0 // indirect
+    go.opentelemetry.io/proto/otlp v1.3.1
+    helm.sh/helm/v3 v3.15.2
+    sigs.k8s.io/yaml v1.4.0
 )

+require github.com/Masterminds/semver/v3 v3.2.1 // indirect
+
 require (
     github.com/Microsoft/hcsshim v0.11.5 // indirect
     github.com/cenkalti/backoff/v4 v4.3.0 // indirect

Edit: I figured it out, the lint pipeline caught a missing go mod tidy

Do I understand what's going on? No... 😛 But I feel more informed in my confusion!

[thorn moat]

wonder why diff -r doesn't print its output in the original call above

[sharp zealot]

stdout

(my guess)

[sharp zealot]

@chrome pilot request to Santa: dagger call --breakpoint Type.function for injecting interactive breakpoints in my dependencies even if they don't fail 🙏 😇 😍

[sharp zealot]

@lime comet @dense canyon is https://github.com/franela/pocketci the source of truth for pocketci?

[dense canyon]

yes. the pocketci-agent branch has some ahead commits from main. Not sure if you pushed all your draft work there @lime comet

[sharp zealot]

Do you accept PRs? 😇

[dense canyon]

yep, sure

[sharp zealot]

should I use main or pocketci-agent?

[serene jetty]

Random question. I really love the idea behind Dagger, but I was wondering the role that Dagger plays when coding software. Do you plan down the road to figure out a way for IDEs to integrate with a Dagger-managed LSP?

[dense canyon]

thinking about what the value of this Dagger-managed LSP could be 🤔 . Verly likely I'm not seeing something here

[restive shore]

Same. Since dagger is based on native language SDKs, the native LSPs (like gopls, pyright) work out of the box for dagger.

[sharp zealot]

I understood it as containerized lsp (along with containerized dev env in general?) in that context dagger is relevant - IDEs currently just install random crap on my machine but I'd rather have it all 100% containerized

[serene jetty]

Here's how my thinking has gone:

• Dagger is amazing, you can run the same workflows that you'd run on CI locally (consistency)
• But some of those steps, are also steps that I run locally (e.g. run tests). Do I run them in the host, or through Dagger? If I run them in the host, I'd feel I've only gone half way in achieving Dagger's full-potential. I don't just want "but I can run the CI steps locally", but also "I use the same CI steps in development".
• But the above poses an interesting meta setup. Assume I'm using Dagger with an Elixir project. I'd end up with "Elixir Dagger pipeline (host)" > Dagger runtime > "Dagger-managed Elixir container running steps". It almost feels like the language that I use in the host shouldn't be Elixir. In fact, when I started playing with it today, I spent some time figuring out the language to use. I chose Elixir too becaus the Dagger script itself written in Elixir can auto-install the Dagger SDK. It'd be probably the same reason why I'd pick Deno over Node. Or maybe Ruby because I can assume that developers in my team, use an environment where I can safely assume Ruby is available.

Not sure if all the above makes sense,

[sharp zealot]

Using Dagger Functions you can move 100% of the dagger pipeline logic into dagger itself - as extensions of the Dagger API. Then everything is containerized and the only dependency for running your dagger pipelines is dagger

[serene jetty]

Gotcha! So it’s the runtime container itself the one that contains the runtime (e.g. Node) to evaluate the function?

It’s very interesting how it’s been solved. In that case I’m inclined to write them in JS.

[sharp zealot]

Yes, a Dagger SDK provides 2 components:

• A client library + bindings generator to call the Dagger API from your code
• A runtime to extend the Dagger API with your own functions and types (bundled in a module)

[plucky ermine]

I absolutely love that we can do this now when reviewing PRs for Dagger docs and it just works

dagger call -m dev --source=https://github.com/marcosnils/dagger#feat/kuberentes_integration docs server as-service up

This was the main use case I had in mind when I wrote the docusaurus module because I wanted to build other projects docs without having to install anything on my computer

I only wish source accepted the PR URL as-is and did the needful instead of me having to remember the proper incantation 👼

[plucky ermine]

@wintry prism I was able to dagger publish with no issues today 😍 https://daggerverse.dev/mod/github.com/levlaz/daggerverse/linkedin@9bbcd60663037c8b7c20897b060dc636a8abdb04

[serene jetty]

Dagger Architecture

Reading that @plucky ermine makes me want to move all the workflows to Dagger 🙂

[plucky ermine]

Lets do it 😄

[serene jetty]

Not sure if this is the right place to ask, but my team is going to explore developing a virtualization CLI upon Apple's Virtualization framework, and I was wondering if we can design it in such a way that Dagger can interoperate with it. We'd like to help break the strong dependency of Apple's ecosystem with proprietary YAML pipelines, and some of their automation runs in macOS environments. The Virtualization framework that Apple provides doesn't make it possible to have very lightweight images like it's the case with Docker, su that'll impact the design of Dagger pipelines, but I think developers will appreciate having a portable automation solution.

[sharp zealot]

I love the idea, it's a pretty common use case to have Dagger pipelines co-existing with native Mac automation. Anything that helps improve that integration is interesting to me - would love to learn more and help if we can

[sharp zealot]

@lime comet should I consider your pocketci-agent branch to be the "real main branch"?

[restive shore]

I don't see examples of this change either in the release notes or the blog. Am I missing something? https://github.com/dagger/dagger/pull/7831.

Reason I ask is ran into some trouble figuring out what the import is. My gopls imported dagger.io/dagger by default. Since I created this module some time back the module itself was named main in go.mod. So the import that worked for me was main/internal/dagger which is a bit awkward. I think this may need to be addressed in the docs. "How to figure out the import if the LSP does not auto import it"

Go alias change documentation?

The TUI verbosity level can go to ♾️ ! I'm not sure I'm ready for THAT level of detail 😄

[plucky ermine]

I am not sure if this is intentional but I kind of wish it was

[restive shore]

I have one more. Sorry for the issue spam but I got excited to test 0.12. Happy to open issues on the repo but wanted to understand if this is expected.

The terminal command seems to kick into "pre" terminal mode before the from image is pulled. So while the image is being pulled the TUI freezes on my screen (my image pull failed which is not related). I expected to have access to the TUI until at least the image for the corresponding container is pulled so I can watch that progress and know that it's being pulled. Pulling a large image can take time and it feels like everything is frozen while it does.

[thorn moat]

https://tenor.com/RfTn.gif

[thorn moat]

Just went to fix https://github.com/vito/swirly since it chokes on newer Go backtraces, and realized I had no idea how to build it anymore. Turns out it's nothing complicated, just jsx, but I took the opportunity to Daggerize it along the way using the TypeScript SDK so I never have to wonder again: https://github.com/vito/swirly/blob/master/dagger/src/index.ts

[restive shore]

I have another observation. The default --progress=plain view changed again with v0.12.0. It's much more verbose by default. To get back to 0.11.9 level (which I really liked since it was much cleaner) I have to pass in -q. It's not readily apparently how the outputs work wrt --progress=plain in the docs.

P.S: I found out by searching the codebase that DAGGER_QUIET=1 sets this at a global level. However, I couldn't find any references to it in the docs. I think it would be nice to have visibility in to the possible env vars to use for dagger CLI in the docs.

[lapis willow]

is calling modules from the direct SDK is on the roadmap?

[sharp zealot]

Yes.

[plucky ermine]

Here's the current issues to follow along https://github.com/dagger/dagger/issues/5993

[plucky ermine]

I am seeing something odd in 0.12 - it looks like calling WithDirectory("/tmp/foo", function.that.returns.Directory()) will mount the name of the directory inside of /tmp where before it would mount the contents - I don't see this explicitly called out in the changelog unless I am misreading something.

[deft rain]

Sorry I planned to look at this today and never did - do you have an example somewhere? If this is the case, I'm genuinely surprised any of our tests managed to actually pass

[wintry prism]

got a repro?

[plucky ermine]

This was my docusaursus module, I fixed it by updating this code

Original

// Build production docs
func (m *Docusaurus) Build() *dagger.Directory {
    return m.Base().
        WithExec([]string{"npm", "run", "build"}).
        // copying build to a temp directory because
        // cache volumes cannot be exported. This is totally
        // worth vs the time it takes to build on a cold cache
        WithMountedDirectory("/tmp/build", dag.Directory()).
        WithExec([]string{"cp", "-r", "build/.", "/tmp/build"}).
        Directory("/tmp/build")
}

It was called this way

// Build the docs website
func (d Docs) Site() *dagger.Directory {
    return dag.
        Docusaurus(
            d.Dagger.Source,
            dagger.DocusaurusOpts{Dir: "/src/docs", DisableCache: true},
        ).
        Build()
}

// Build the docs server
func (d Docs) Server() *dagger.Container {
    nginxConfig := dag.CurrentModule().Source().File("docs-nginx.conf")
    return dag.
        Container().
        From("nginx").
        WithoutEntrypoint().
        WithFile("/etc/nginx/conf.d/default.conf", nginxConfig).
        WithDefaultArgs([]string{"nginx", "-g", "daemon off;"}).
        WithDirectory("/var/www", d.Site()).
        WithExposedPort(8000)
}

In < 0.12 it would mount the contents of the output of Site() inside of /var/www/

Starting in 0.12 it would mount a directory called build inside of /var/www

I updated the code in the module to do this instead WithExec([]string{"cp", "-r", "build/.", "/tmp/build"}). which works, but I have no idea how it worked in the first place.

The other thing that changed was that I updated the return type from Directory to *dagger.Directory but I don't think this has anything to do with it (although I guess it might)

[deft rain]

Hmmm this is very weird and suspicious, I'm like 99% sure this shouldn't have been changed (cc @kind carbon, could this have accidentally snuck in from some of the breaking changes we all worked on mayyyybe?)

[sharp zealot]

Starting a thread about my experience upgrading modules to 0.12

[sharp zealot]

@thorn moat I'm trying to use your apko module to build our alpine images in addition to wolfi, but struggling with arch-related errors

Trying to reproduce the issue (had to reimplement a chunk of the logic in Go): https://dagger.cloud/dagger/traces/d563580a17f0169c7067f2b3748c6072?span=327fc1d643f3bed3#685d9ffb73af181a

Apko seems to correctly build for my desired arch (amd64):

2024/07/18 13:09:33 INFO Building images for 1 architectures: [amd64]
2024/07/18 13:09:34 INFO setting apk repositories: [https://dl-cdn.alpinelinux.org/alpine/edge/main]

But at the end it suddenly starts talking about x86_64:

2024/07/18 13:09:35 INFO built image layer tarball as /tmp/apko-temp-1145754839/apko-x86_64.tar.gz
2024/07/18 13:09:35 INFO OCI layer digest: sha256:4142854d9191c65a35d40314a55046b4b2651c39c37f6d39122b524d8531b0ec arch=x86_64
2024/07/18 13:09:35 INFO OCI layer diffID: sha256:844a55be58a564cbc2230bb887d750496c90b7833680d8ddc5e5eb43a17b2f74 arch=x86_64

Then sure enough, Container.import doesn't find an arm64 image where it should be:

recover: no manifest for platform linux/arm64 and tag

[thorn moat]

wait - are you building amd64 or arm64? First part says amd64, which afaik is same as x86_64 (prob gets translated somewhere)

[sharp zealot]

amd64

Gah.. Yes that typo/brainfart was the problem. I had a separate problem at container import level (didn't set the target platform in dagger.Container() which crossed wires in debugging)

Lots of amd64/x86_64 and amd64arm64/aarch64 inconsistencies in one serving (I did the brainfart again)

[sharp zealot]

Upgrading to 0.12.1, yay!

[restive shore]

In the TUI, when I zoom into something is it possible for that thing to span the height of the terminal like it does width? For context, I am running a mvn install and when I run it outside of dagger I get a full view in the terminal of progress. But in dagger TUI it's limited to a certain height (roughly 1/3rd of the screen). The text moves really fast, so any I'll take any extra screen real estate I can get. Not that I stare at the pointless log on the screen, but it's nice when I have to do it. Like when testing dagger 🙂

[radiant parrot]

Hey! 👋 I've found a tiny bug in the Container.Terminal PS1 configuration. I've proposed a PR to fix it: https://github.com/dagger/dagger/pull/7960 Is this the right place to ask for a review?

[plucky ermine]

fix: print current directory without par...

[sonic vessel]

Is verbosity in TUI is not the same as verbosity flag (-v ) ? I asked because when I use -v=2 the TUI show verbosity=3.

[thorn moat]

Is verbosity in TUI is not the same as

[restive shore]

TUI Size (viewport?)

[wintry prism]

Oh, I thought the core hack wasn't working anymore, but just tried again with v0.12.1 and it's working fine. So much fun to pair with the new output format when core types returned!
https://gist.github.com/jpadams/7610cefb2a9b2b70b1af5ca549814bae

core container from --address alpine/git:latest

_type: Container
defaultArgs:
    - --help
entrypoint:
    - git
mounts: []
platform: linux/arm64
user: ""
workdir: /git

https://github.com/dagger/dagger/issues/6947

[bright field]

Is it possible to specify the exclude during dagger init? dagger init --sdk=typescript in my project took 5m to run ModuleSource.resolveFromCaller

[sharp relic]

https://discord.com/channels/707636530424053791/1251260501426442290

A temporary solution is availtble but a better proposal awaits to be merged and released. Have a glrmpse over the thread td learn more 🙃

[bright field]

My main issue is the first time I run dagger init --sdk=typescript, it goes off for 6m of initialization. I'm assuming that is copying things like .direnv into the container, but I have not idea.

[sharp relic]

Hmm. I don't know if it does read the files. Have you tried to test this by init'ing in an empty directory (a la ´cd "$(mktemp -d)"´)?

[spiral whale]

Finally https://github.com/dagger/dagger/pull/7804 is live 🎉

[dense canyon]

@bright field any chance you can share Dagger Cloud trace about that initial dagger init to understand better where that time is going?

[restive shore]

Anyone running a runtime other than Docker in your personal setup? I am trying to use podman but running into some issues with buildkit compatibility for certain commands. Specifically the update-ca-certificates command. I'm on a Mac. Wondering if there's a better alternative.

[latent trellis]

After upgrading to 0.12.2 (from 0.12.0):

Error: response from query: input: ci.ci resolve: call function "Ci": process "/runtime" did not complete successfully: exit code: 2
Stdout:
invoke: input: helm.chart.package resolve: call function "Package": process "/runtime" did not complete successfully: exit code: 2
Stdout:
marshal: json: error calling MarshalJSON for type *main.Package: json: error calling MarshalJSON for type *dagger.File: input: container.from.withUser.withoutEnvVariable.withoutEnvVariable.withMountedSecret.withWorkdir.withMountedDirectory.withExec.file resolve: secret xxh3:92e7da10ead5f2c1: not found

[latent trellis]

Tried twice on GHA, same result. Reverting to 0.12.0 fixed it.

[plucky ermine]

Thanks for the report, can you share what the call was that led to this?

[latent trellis]

• https://github.com/openmeterio/openmeter/blob/main/ci/build.go#L246
• https://github.com/sagikazarmark/daggerverse/blob/main/helm/chart.go#L41

[rose hinge]

I update some modules with changes + upgrade to 0.12.2 but when I'm trying to publish I have the following error: get name: input: moduleSource.asModule resolve: failed to create module: select: module requires newer engine version: version v0.12.1 does not meet required version v0.12.2 , when the version of daggerverse is updated after a new release ? Or maybe I'm not correctly understanding the issue

[plucky ermine]

Thanks for the report!

@wintry prism do you know if we bumped Daggerverse?

Once again need to find a way to add this to our release checklist I think 🙂

[wintry prism]

@lime comet 👆

Thanks @rose hinge, agreed @plucky ermine. I'll look in to this now.

[plucky ermine]

After upgrading to 0.12.2 (from 0.12.0):

[latent trellis]

That would really be nice. There is nothing worse than feeling the excitement of a new release or a new feature and then getting denied by daggerverse.

Okay, I can think of a few worse things, but this is definitely in the top...ish. 🤣

[wintry prism]

@rose hinge @latent trellis should be good to go!

[wheat river]

Hello 👋 -
I'm wondering if the Daggerverse is already prepared to implement the examples/<sdk> pattern to self-document the allowed recipes or usages for my modules. By the way, I'm planning to use it only for documentation purposes, I'm still planning to keep the tests/ submodule on my modules to ensure their reliability.

[plucky ermine]

Yes, I belive this will work if its provided @wintry prism can you confirm?

[wintry prism]

Yes, indeed! You can take a look at this example for inspiration: https://github.com/marcosnils/daggerverse/tree/main/gptools

[strange arch]

Hello, it seems that passing a secret to a function using the cmd:command-to-run syntax doesn’t strip the final newline of the command output:

compare

dagger call -m github.com/yann-soubeyrand/dagger-test test --secret 'cmd:echo secret'

with

echo prefix.$(echo secret).suffix

Is it a known issue/expected behaviour or should I open an issue?

The source of the module is:

package main

import (
    "context"
    "dagger/test/internal/dagger"
)

type Test struct{}

func (*Test) Test(ctx context.Context, secret *dagger.Secret) (string, error) {
    plaintext, err := secret.Plaintext(ctx)

    if err != nil {
        return "", err
    }

    return dag.Container().
        From("docker.io/library/busybox").
        WithSecretVariable("SECRET", secret).
        WithEnvVariable("PLAINTEXT", plaintext).
        WithExec([]string{"sh", "-c", "echo prefix.$SECRET.suffix prefix.$PLAINTEXT.suffix"}).
        Stdout(ctx)
}

In addition, it makes Dagger unable to scrub secrets from its output. Explicitly striping the final newline shows the difference in output:

compare

dagger call -m github.com/yann-soubeyrand/dagger-test test --secret 'cmd:echo secret'

with

dagger call -m github.com/yann-soubeyrand/dagger-test test --secret 'cmd:echo -n secret'

[kind carbon]

Yes, it's known but I don't think we can strip that newline automatically. You need to do it through some other means. A while ago we did strip it because of a similar use case, but then we had to revert it because some other users had a problem with that triggering edge cases (e.g., reading SSH keys).

[kind carbon]

Thanks for the repro! That can be its own issue, re: scrubing. Do you want to create the issue in GitHub?

[strange arch]

Couldn’t we only strip secret for the cmd: source? In case of the env: and file: cases, the value should indeed be right from the start, but for the cmd: case, are there use cases where the output must not be stripped? Or maybe we could have an optional argument in the form cmd:command-to-run:s to ask for stripping or in the form of an annotation/comment on the function parameter in the code? Just putting some thoughts here, maybe it doesn’t make sense at all!

[strange arch]

Sure!
https://github.com/dagger/dagger/issues/8032

[kind carbon]

Couldn’t we only strip secret for the cmd: source? In case of the env: and file: cases, the value should indeed be right from the start, but for the cmd: case, are there use cases where the output must not be stripped?

I think this has the same issues. If you think about it, echo isn't representative of "fetching" a secret. Most commonly you'd be using some tool that fetches the secret like sops or op (1Password). In these cases, the secret value should be untouched. I for one use 1Password, and that's where I have my SSH key. If the newline needs to be there and cmd strips it it'll lead to the same issue.

Or maybe we could have an optional argument in the form cmd:command-to-run:s to ask for stripping or in the form of an annotation/comment on the function parameter in the code? Just putting some thoughts here, maybe it doesn’t make sense at all!

There may be merit to this, I suggest you open an issue for this specifically to gather feedback. However, why stop at cmd? Perhaps instead of adding it to the end (which can be difficult to know for sure it's not a part of the command), could be at the beginning, and for the other prefixes too:

• file-strip:/path/to/file
• cmd-strip:echo secret

Or just:

• cmd-s:echo secret

[strange arch]

You’re right, I had the gh auth token use case in mind which doesn’t work due to the trailing newline, but I didn’t think about the retrieval of SSH keys from a password manager.

Of course the argument to ask for stripping could be available for all the mechanisms, I just took cmd as an example. I proposed the syntax with an s separated by a colon at the end because I thought of the volume syntax of Docker (src:dest:ro), but you’re right that it might not be the best in our case.

[sharp zealot]

@kind carbon I think stripping by default might be the correct behavior actually. I just tried with op read (1password cCLI) and it adds a newline to the secret by default. They have an optional flag to strip. We should check with a few more common secret managers, and if they have the same default behavior as 1p, we should adapt ours

At the very least the stripping-by-default behavior deserves serious discussion in the issue, shouldn't be dismissed right away

[kind carbon]

This has been discussed, not dismissing. We used to strip but it created edge cases so we had to revert because when in question, best default to not change it. See Revert secret whitespace stripping #7271. Doesn't mean we can't bring it back, but needs to be more thought out and researched, at least what the default should be if we do add an option like cmd-s:.

Issue where this was discussed: https://github.com/dagger/dagger/issues/7220

[sharp zealot]

I see, tricky! Didn't realize stripping final newline broke ssh private key files, that's very surprising

[strange arch]

Here’s an issue to discuss it again: https://github.com/dagger/dagger/issues/8034.

[honest hearth]

My anecdota experience with this: Secrets as ssh private key (or pem in general) and service account jsons (e.g. google) are extremely sensitive to this kind of thing. Just having those two as test cases will cover a lot of the difficult problems around this

i guess a service account json is kind of superset hard mode anyway, since often it has a private key embedded inside the json

[sharp zealot]

can't wait for special-cases dag.Context().SSH().Keys(), dag.Context().GoogleCloud().Auth() etc

dag.Context().1password().Read()

[honest hearth]

would love this, because a month or two ago I was actually trying to do something very similar to what we are talking about here, which is to fetch a service account json from 1password dynamically and serialize it into a secret to be loaded into dagger. The amount of shell hurdles and tears shed trying to achieve this was... not something I really like to recall

(this had little to do with dagger, by the way, and more to do with the fact that storing, fetching, and moving secrets into shell from 1p that are super complex like a service account json is just not handled super well by shell constructs) so having a golden path for this would be a big boon

the problem was, perhaps unsurprisingly, how shells handle newlines and quotes is just.... painful

[lime comet]

This would definitely be a dream. At the moment, what I settled on for my pipelines and local tests is to use the 1Password CLI inside an envrc file in the folder of a given project/module. This combined with direnv makes it so that I can enter a project, do a dagger call and the env variables will already be there for me to send them

Its not great because it makes every single secret available for all dagger calls in that project, no per-call isolation. But its better than remembering how to fetch a secret every time i make the call

[sharp zealot]

Yeah that feature will need a very solid trust system, specifically to avoid or rather manage that security risk. Modules cannot be allowed this level of access without explicit permission in a way that minimizes the risk of user error, and doesn't break usability. Hard but worth it!

Also the only way to correctly implement "multi modal" auth such as platform-specific auth helpers (think keychain integration for docker)

[proud verge]

New here, so not sure if this is the right place for this question, so feel free to direct me elsewhere.

My company has been using a Dagger-like tool for a while now, but it's YAML-based DSL and doesn't have a large support community, so I'm evaluating Dagger (I've always wanted to write pipelines in my GPL, this feels like a dream come true).

One issue I'm running into is how best to utilize Dagger with a CI/CD platform (same problem with existing tool). The issue is that there are features of a CI/CD platform that seem like must-haves (e.g. step-based metrics, step retries, etc.). So the direction I'm receiving from more senior engineers is to call Dagger for each step. My pushback is that this takes away one of Dagger's best features in terms of its strong typing system.

How are others handling this? Do you have a Dagger function for CI that runs from PR to merge? If so, how are handling things like metrics and retries? Or do you call individual Dagger functions in CI?

[quick wind]

Haven't used the dagger.GitRepository/dagger.GitRef types before - am I missing a simple way to interrogate git refs for dagger.Directory? If I want to do something like accept a directory from the user (via parameter as usual), determine the current branch name, then use the directory as normal, is that something like parameter type becomes dagger.GitRepository, and some combination of branch() and tree() gets me the directory?

[deft rain]

Unfortunately, there's not yet a way to load a Git type from a Directory.

i don't think there's a tracking issue for this yet - but i've also wanted this for a while

[quick wind]

Would be a convenient switch yep

[wheat river]

After upgrading some of my modules to version 0.12, I've noticed that when Terminal() is being called, it's not returning a *Container instead of dagger.Terminal as before. What's the rationale behind this change?

[kind carbon]

You can have multiple Terminal() in a Container pipeline, which also means Terminal() can be in the middle of a Container pipeline. When you exit that shell, it'll continue to run the rest of the pipeline.

You can see it in action here: https://www.youtube.com/watch?v=gUw-9kuvwDM&t=120s

[wheat river]

Is it currently supported using structs as function parameters? - it seems it's not supported. To express what I'm thinking, it's close to this:

type WithAWSKeysOpts struct {
    AccessKeyID     string
    SecretAccessKey string
    SessionToken    string
}

func (m *ModuleTemplate) WithAWSKeys(options WithAWSKeysOpts) *ModuleTemplate {
    m.Ctr = m.Ctr.WithEnvVariable("AWS_ACCESS_KEY_ID", options.AccessKeyID).
        WithEnvVariable("AWS_SECRET_ACCESS_KEY", options.SecretAccessKey).
        WithEnvVariable("AWS_SESSION_TOKEN", options.SessionToken)

    return m
}

What's the reason why it's not supported? -

[deft rain]

So we used to have this 😆 But we then removed it - see https://github.com/dagger/dagger/pull/5907 for a very long discussion

However... yeah. The thing that's changed since then is that we now have // +pragmas as a way of passing metadata. At the time we were using Opts in the name to determine if it was a magical opts struct, but we could now do that with our pragmas - which would make it a lot more usable

[west magnet]

Hello 👋 I just joined the community after peeping at the docs for the past few weeks. I am wondering if dagger would be suitable for my use case.

I am a kubernetes administrator and I would like to replace my shell scripts for deploying and destroying kubernetes clusters. The shell scripts are beginning to become too complex for bash and I want to introduce some modularity. One avenue I am exploring is writing it in Go using either client libraries or subprocess calls. The other avenue I'd like to explore is Dagger.

The scripts do various tasks like calling the AWS CLI, kubectl, helm template/install etc. What do you think? Would Dagger be a suitable option here?

[vagrant sage]

For those of you using Zsh or Oh My Zsh in your work with Dagger or DevOps in general - what plugins do you find most useful?

[honest hearth]

For those of you using Zsh or Oh My Zsh

[opal shard]

I am trying to invoke a reusable module function from Python but I do not know how to approach it.
I created a public gitlab repo, and I can invoke the dagger CLI "dagger functions -m dagger-shared-module-functions" and "dagger call -m dagger-shared-module-functions my-function"
The dagger .json is setup for this, but I want to invoke these functions not using the CLI but from my python code when executing the "dagger run python3 main.py"

In the main.py I have this code snippet see the file

What am I doing wrong or what is missing to invoke for example the "my_function" on the module, the objects() and interfaces() are always return empty []

python

[kind carbon]

This is not yet supported. You can track this in GH: Call Dagger Functions from an external client #5993. I've actually been meaning to get into that since last week. It's easy enough to do in Python, just need to toggle a few knobs in the engine, but need to finish off a few things before I can focus on it.

[opal shard]

Ok I will wait for your implementation, thank you for the quick response, i you could ping me that would be great I am currently using "engineVersion": "v0.11.9"

[latent trellis]

Is it possible to create circular dependencies between services (eg. multi-node Kafka cluster using KRaft where the controllers need to talk to each other)?

[thorn moat]

not currently possible, would probably need a new services API since the current one is fundamentally a DAG. previously: https://discord.com/channels/707636530424053791/1239817495309713428

[limber canyon]

I'm trying to use functional options in a module, something like this:

type options struct {
    foo string
}

type Option func(*options)

func WithFoo(s string) Option {
    return func(opts *options) {
        opts.foo = s
    }
}

func (m *Module) MyService(ctx context.Context, opt ...Option) *dagger.Service {
    ...
}

but I get a codegen error:

Error: generate code: template: module.go.tmpl:79:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: missing method XXX_GraphQLType from DaggerObject interface, which must be embedded in interfaces used in Functions and Objects

is it impossible to do what I want or am I missing something ?

[deft rain]

Sadly, this isn't possible (yet, mayyybe it could be? not sure)
To work out what args are allowed on a function, we scan through each parameter - having function opts like this isn't really something we detect or look through, I'm also not sure if it neatly translates into graphql-args, so that other sdks can call it

[sharp zealot]

That seems like a uniquely Go-specific pattern, a portable equivalent would be to use enums, or optional arguments, a struct argument, or chaining pattern with a sequence of withFoo() calls to set options in the state

[limber canyon]

understood, thanks.

[sharp zealot]

As discussed with @thorn moat@kind carbon @deft rain yesterday: https://github.com/dagger/dagger/issues/8123

[sharp zealot]

Today I learned that context directory exacerbates the lack of self-calls cc @rocky harbor @deft rain @kind carbon @thorn moat . You quickly end up copy-pasting ignore pragmas all over the place

actually self-calls might not solve it

[kind carbon]

Yeah, it's a problem in Go because it's a comment, but the other SDKs can reuse with variables. Additionally, when we add support for .gitignore by default, it should avoid the need for a bunch of patterns in the pragmas.

[sharp zealot]

.gitignore won't help me, because I have several views on the same dir

But I would love a +ignoreFile

@kind carbon @upbeat herald I'm getting a Go error mismatch on this trace...https://dagger.cloud/cubzh/traces/304bf452887b3fe883965f9de2b98df2

Is that normal? I just created the module, it's nice and fresh. And the CI config is fresh from the docs also. So not sure how Dagger could generate go.mod that it cannot run itself?

[sharp zealot]

@dense canyon I think you explored this in depth at some point. Does dag.Git() support sparse checkout?

[dense canyon]

no it doesn't. That'd require an upstream buildkit change I assume as it currently does --depth=1 checkouts. cc @deft rain

[deft rain]

yup, currently it's all --depth=1

we don't need to do it buildkit since we have a forked source op for git

[sharp zealot]

OK, I can do it in userland then. At least to experiment.

[dense canyon]

right, I forgot we forked the git part in Dagger ❤️

having sparse checkouts in the Git API would be awesome though for monorepos

[sharp zealot]

Yes. There's also LFS support. We can always merge the userland implementation in the core API later

What would be really nice (and can't be done fully userland I think) is to hook up the new +ignore pragmas to upstream git, so that the engine honors ignores by making a sparse checkout on the fly. Do you think that's doable @deft rain @rocky harbor ?

I'm working on a real-world CI config ( 👋 @vivid hatch @visual blaze ) and there is a lot of redundancy between sparse checkout config (in CI config) and context directory ignore (in Dagger code)

[deft rain]

my initial guess is maybe - this is different from the blob directory case, where this isn't possible - but how hard it is depends on where that directory currently gets unlazied - since we're effectively wanting to change some part of the evaluation "last minute"

i think it's possible but i have a suspicion it might not be quick and dry

[sharp zealot]

I didn't understand any of that

what is the blob directory case?

[deft rain]

when you just pass a directory in, we can't do this optimization and only upload those files

[sharp zealot]

Is this a known issue? failed to compute cache key: failed to get layer download url for digest sha256:c4c966285248640e73e2e7a12b2e5d7bbb37cd20454edff9032b2bfa9e6ca427: unexpected status code: 500:

--> https://github.com/shykes/cubzh/actions/runs/10304200959/job/28522049408?pr=1#step:3:251

[deft rain]

that's because if we allowed this, the caching would get entirely messed up - it would require an entire rewrite of how we manage file upload/download and calculate hashes (potentially we could do this one day, i'd definitely like to)

[sharp zealot]

I see. "blob" in this context is related to local upload

[deft rain]

is this using magicache?

hm, yeah, that's where the error comes from

i have seen this before, but not recently

[dense canyon]

cc @lime comet any chance to find the API logs for this?

[dense canyon]

a 500 here is weird given that it's only requesting the layer URL

[sharp zealot]

Not sure if it's relevant, but I'm running this on my fork of the upstream repo, while using their token. ie. the PR itself is from my fork to my fork

[dense canyon]

I can repro locally. Seems like a specific issue with their cache config

are they still using self hosted runners in AWS @sharp zealot ?

[sharp zealot]

You need to use the same token to repro, right?

[sharp zealot]

Yes

[dense canyon]

yes, that's what I used

[sharp zealot]

If you can repro locally, then it's not related to the CI runner configuration right?

[dense canyon]

their magicache config has their desitnation to our bucket in AWS. I'm checking if running this from your fork is causing the issue as you can't use private AWS endpoints

I think this is what's likely to be happening

[dense canyon]

can still be. Check my comment above

[sharp zealot]

I see. What's weird is that it's not all CI jobs failing. Only 1/3 in that PR

[dense canyon]

are the others using the token?

[sharp zealot]

Yes

[dense canyon]

seems related to the cache config Solomon. Can you try using the token from your personal Cloud Org?

just to make sure it passes that way

having said that, the pipeline shouldn't fail if the configs missmatch, we should disable magicache and log a warn in the engine logs. Opening issue

[lime comet]

It seems that the issue is what @dense canyon said here. We are trying to access an AWS bucket from an origin not in AWS. The behaviour is correct but we should not fail. We'll open an internal issue to change magicache and handle this gracefully

[sharp zealot]

Ah I see, it's because the runner on my fork is the regular free GHA runner?

[dense canyon]

yes

[sharp zealot]

https://daggerverse.dev/mod/github.com/shykes/dagger2gha

[sharp zealot]

How do I use the new Examples feature in Daggerverse? I tried creating an examples/go module, but my examples aren't getting picked up. Is there documentation for this somewhere?

[latent trellis]

Examples feature?

[sharp zealot]

Yeah, Daggerverse will now look for examples modules at a standard location, introspect them, and render them in the module page

I just haven't managed to get it to work yet 🙂

[latent trellis]

https://tenor.com/bgbOA.gif

Let me know if you do, so I can use your examples as examples for my examples

[sharp zealot]

Ah! Here's an example of examples 🙂 https://github.com/marcosnils/daggerverse/blob/main/k3s/examples/go/main.go

But it doesn't seem to be rendered in daggerverse... https://daggerverse.dev/mod/github.com/marcosnils/daggerverse/k3s@f4851a9c308410d58c5199dfd1b65e4f3ccae813#K3S.server

[latent trellis]

That's what I was about to say

[limber canyon]

I think I'm getting confused with the Go SDK. I wanted to use the https://pkg.go.dev/dagger.io/dagger@v0.12.4#Client.Host method, so in my function I tried to use dag.Host() because dag is a *dagger.Client but actually not the right one, it's an internal, generated type (dagger/ops-devenv/internal/dagger)
What's the relation between the upstream SDK and the generated code ? Shouldn't they have the same capabilities and methods ?

[sharp zealot]

Yes they do have the same capabilities and methods, but there are a few exceptions and discrepancies, which we would like to fix

cc @kind carbon @upbeat herald

• Host() is a special case because by design, Dagger Functions are sandboxed and can't access the real host.

[kind carbon]

The other one that's excluded from modules is dag.Engine.

We should update the descriptions in the API mentioning that it's not available for modules.

[limber canyon]

makes sense, thank you

[plucky ermine]

Daggerverse Examples

[limber canyon]

is there a way to obtain the "default" ssh agent socket so that I can use it with https://pkg.go.dev/dagger.io/dagger@v0.12.4#GitOpts ? There's https://pkg.go.dev/dagger.io/dagger@v0.12.4#Client.LoadSocketFromID but I don't know what the ID is supposed to be.
I'm trying to automatically clone a repository without the user having to provide a repository as a dagger.Directory

[sharp zealot]

Are you developing a Module, or an external client running on the host?

Assuming a module: since your functions will be sandboxed, they can't access the ssh agent socket on your host directly, but you can pass it as an argument. Have your function receive an argument of type dagger.UnixSocket, and pass it from the CLI

[limber canyon]

it's a module yes.

to give more context, the module runs a cassandra service and needs to run fixtures. These fixtures are stored in another Git repository so I was trying to figure out a way to hardcode this repository in the module so that the user doesn't need to explicitly pass the repository from the CLI. This works: dagger call cassandra --default-fixtures="[e-mail]:BatchLabs/foo.git#master:myfixtures" but it's a bit of a pain.

With this command dagger finds the socket: ✔ Host.unixSocket(path: "/private/tmp/com.apple.launchd.zlLKHFJcWV/Listeners"): Socket! 0.0s so I was thinking maybe I could also do this automatically from inside the module but yeah I didn't think of the sandbox

[sharp zealot]

Yeah that Host.unixSocket call is made by the CLI itself, as a convenience from your [e-mail]:BatchLabs/foo.git#master:myfixtures argument. I think that you're already doing it in the least cumbersome way... for now at least.

We just merged "context directory" which is a big step towards solving this family of problems

Basically we're introducing ways for functions to sometimes access resources outside the sandbox, in a carefully controled and safe way

Right now (as of upcoming 0.12.5) it's only available for directories (you can define a default value for your directory arguments, which are a path relative to your module's context, typically the surrounding git repo)

So that will take care of the super repetitive dagger call build --backend-source=../backend --frontend-source=../frontend. Now you can just dagger call build and the defaults will reach out of the sandbox

We plan on doing the same for other types, including your SSH agent socket... But it's tricky because there are security implications. What if you run a random untrusted module, and it silently gets access to your ssh creds. So we need a proper trust model

[limber canyon]

understood, thanks for the insights. In the meantime I'll probably wrap the call to the CLI with a Makefile or something.

[wheat river]

👋 !
A couple of questions about the Daggeverse 'behaviour'.

• What I'm missing here: https://daggerverse.dev/mod/github.com/Excoriate/daggerverse/module-template@5d41af03e43c3ed85fb942adcae751edf0f2e65a? - the examples/go isn't visible when the module is published. I've tried a couple of times, without success.
• How the dagger verse handles the 'deprecation' of a certain module. Let's say the module was directly removed; is it removed from the Daggerverse? How much does it take?

[latent trellis]

The beginning of the function name has to match the module name:

https://github.com/Excoriate/daggerverse/blob/5d41af03e43c3ed85fb942adcae751edf0f2e65a/module-template/examples/go/main.go#L55

https://github.com/sagikazarmark/daggerverse/blob/04381efdb27e3c3873e6528d63edf6ec24178d02/helm/examples/go/main.go#L23

[latent trellis]

Someone asked about updating dependencies used in Dagger code a while ago. I just came up with this: https://github.com/openmeterio/openmeter/pull/1333

[wheat river]

Thanks @latent trellis , indeed. It works — however, I've noticed that some of the functions aren't rendered correctly. I tried with MyDaggerModuleMyExampleFunction and **MyDaggerModule_**MyExampleFunction and both breaks. It partially renders the example, but depending on how the example function was documented (in my case, I'm using detailed Go docs sort of comments), it breaks the rendering. Example: https://github.com/Excoriate/daggerverse/blob/9f822d5041caddfb1bd8df3b72d6e8aeaef0cfef/module-template/main.go
I suspect that the Go Docs are somehow breaking the rendering process.

[latent trellis]

AFAIK it's still an experimental feature, so there might be some rough edges. cc @dense canyon

One thing that's a bit weird with the enum support: enum names are not prefixed/suffixed with the module/enum name. I guess that's not necessarily a problem on its own, but given all other types (structs, opts) are prefixed (at least in Go), I found that weird. Also, it's easy to create collision between different modules/names.

Also, trouble in enum paradise again 😛

invoke: input: trivy.container.file resolve: failed to convert arg "format": *core.ModuleEnumType.ConvertToSDKInput cannot handle type dagql.DynamicOptional

type ReportFormat string

const (
    Table      ReportFormat = "table"
    JSON       ReportFormat = "json"
    Template   ReportFormat = "template"
    SARIF      ReportFormat = "sarif"
    CycloneDX  ReportFormat = "cyclonedx"
    SPDX       ReportFormat = "spdx"
    SPDXJSON   ReportFormat = "spdx_json"
    GitHub     ReportFormat = "github"
    CosignVuln ReportFormat = "cosign_vuln"
)

func (m *Report) File(
    ctx context.Context,

    // Trivy report format.
    //
    // +optional
    format ReportFormat,
) *dagger.File {
    reportPath := "/work/report"

    args := m.Args

    if format != "" {
        reportPath += "." + string(format)

        args = append(args, "--format", string(format))
    }

    args = append(args, "--output", reportPath)

    return m.Container.
        WithExec(m.Args).
        File(reportPath)
}

[wheat river]

It seems like that. Anyhow, I'm happy to submit an issue if it's confirmed it's a bug.

[sharp zealot]

https://tenor.com/view/trombone-pusheen-musician-instrument-gif-11434220432919976776

[dense canyon]

👀 checking

I can see the examples got parsed correctly but they're not being displayed in the FE for some reason. Thx for sharing this 🙌

[deft rain]

😢

i can repro, and i want to use it

so, expect a fix soon ❤️

[latent trellis]

not gonna help me much, since I can't upgrade again 😢

but thanks

[deft rain]

ah, sorry 😦 what's the other thing blocking the upgrade?

[latent trellis]

Error: response from query: input: ci.ci resolve: call function "Ci": process "/runtime" did not complete successfully: exit code: 2

Stdout:
invoke: input: helm.chart.package resolve: call function "Package": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *main.Package: json: error calling MarshalJSON for type *dagger.File: input: container.from.withUser.withoutEnvVariable.withoutEnvVariable.withMountedSecret.withWorkdir.withMountedDirectory.withExec.file resolve: secret xxh3:108a41a8d03181bb: not found

#commander-sagikazar message

[deft rain]

ahhh yes ofc - will also try looking into this ❤️

[latent trellis]

Thanks!

[latent trellis]

Anyone saw this error before?

Error: response from query: Post "http://:mem/query": command [docker exec -i dagger-engine-6a33d13bdddceaf8 buildctl dial-stdio] has exited with exit status 137, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

[dense canyon]

fixed @wheat river https://daggerverse.dev/mod/github.com/Excoriate/daggerverse/module-template@5f90e87895dfd795b2137fd04b2959ae5b179fc7

[wheat river]

Thanks a lot! (muchas gracias! @dense canyon !

[dense canyon]

de nada! 🧉

[latent trellis]

I managed to build a pipeline that eats up all 10 GB of RAM I gave Docker. 😄

[sharp zealot]

I'm getting Full trace at https://dagger.cloud/ (rotate dagger.cloud token for full url) in my CI logs, what does it mean?

[latent trellis]

AFAIK there was a change in how tokens are created. There is now a prefix or something.

[sharp zealot]

ah. I love that you're the one providing support to me 😛

thanks!

[latent trellis]

anytime!

[dense canyon]

yep. this is exactly the case @sharp zealot

[wheat river]

Hey guys, is there any plan to extend dag.HTTP to support more advanced configuration, like headers, specific authentication methods, etc.?

[wheat river]

Hey @dense canyon , thanks for the fix yesterday ❤️ , the auto-render (https://daggerverse.dev/mod/github.com/Excoriate/daggerverse/module-template@3b8536c74008de137f515d1bbc64880e69e2d5c5#ModuleTemplate.withHttpcurl) of the examples when releasing modules looks great!

[dense canyon]

my pleasure. Glad you liked it!

[deft rain]

🎉 https://github.com/dagger/dagger/pull/8148
looks like we just had an accidental early return that wasn't applying optional right 😄

[sharp zealot]

I just realized that with the way Go SDK handles optional arguments, changing an argument's optionality will break Go clients, even though the graphql schema is backwards compatible

[dense canyon]

I guess it makes sense? Since it'd be the same as if you were coding any other Go app. I'd actually expect a new function to be added with the optionality changes and the old one being marked as to be deprecated

[sharp zealot]

But it doesn't break other SDKs. Only Go (when making a mandatory argument optional)

[dense canyon]

Still makes sense I guess.. the other languages have native support for optional function arguments 🤷‍♂️

[sharp zealot]

I don't know what "makes sense" means. In practice it means everyone has to assume they will break all clients when making a mandatory argument optional.

Maybe it's the right tradeoff because the DX would be worse in Go otherwise. But it's still a tradeoff

[dense canyon]

Makes sense as: "I'd expect the same output as if I'd have to change a function argument optionallity in any app written in that language regardless of Dagger"

[sharp zealot]

Yeah but that's a weaker platform feature than "I expect the guarantee that it won't break anyone, because the Dagger API is based on GraphQL, which offers that guarantee, and it's reasonable to expect official SDKs will find a way to carry that guarantee". We didn't find a (good enough) way in Go, so we didn't make that guarantee.

So it makes sense that Dagger has flaws. But it's still a flaw.

[latent trellis]

I agree, it makes sense as in Go doesn’t have that feature, so to me that’s expected.

I’d even argue that any additional API magic to circumvent Go’s limitation would be worse (looking at you aws.String)

[sharp zealot]

I'm not arguing for doing any magic - we've explored alternatives thoroughly, and landed on the least bad DX. But, we should also acknowledge that it makes our cross-language DX just a little bit worse. Put yourself in the shoes of a Python dev, or Typescript dev, not familiar with Go. It's reasonable for them to expect that making a mandatory flag optional won't break their clients: because it doesn't break Python; it doesn't break Typescript, and it doesn't break the underlying GraphQL. But they would be wrong. Understanding why, requires learning Go-specific knowledge, which has now leaked into Dagger-wide knowledge.

It's a subtle example of lowest-common-denominator erosion. The least capable language dictates what all languages can do. This is a relatively benign example of it. But it's a good learning opportunity. We have to learn to fear this kind of erosion. It's like rust on a ship, a little bit is OK, but if you let it get out of control, things can get bad.

The backbone of our cross-language capabilities is GraphQL. It has to be our measuring stick for what to aim for. It's OK to deviate as long as we have our eyes open. This is a (small, acceptable) deviation from the GraphQL baseline.

As alternative SDKs start to proliferate (which I am excited for), eventually we will need to develop a benchmark to grade all SDKs. These benchmarks will require criteria, based on guidelines to SDK developers - what SDKs should and should not do. We'll have to decide if the guidelines say "don't break clients when mandatory arguments become optional". And if we do, then the Go SDK will have a "red" check where Python and Typescript get a green.

[latent trellis]

But, we should also acknowledge that it makes our cross-language DX just a little bit worse.

That's fair. I guess I don't really rely on that much since I'm working mostly with Go, but people using multiple languages may indeed have a hard time figuring out what's going wrong in that case.

[dense canyon]

💯 , it's energizing to process the way you think about it and to imagine how this could be translated to the product so our users have the best experience. 🙏

[wheat river]

I'm a bit biased by the Do not break the API contract point of view — however, it's a bit tricky, especially dealing with multilanguage support (and SDK[s] for that matter). Sometimes language benefits constrain the design and the desired Developer Experience. It's okay in my opinion to break it an 'a bit' if the versioning and releasing (and communication, and documentation for that matter 😃 ) is properly handled. In my experience with Dagger, I've seen that it's amazingly handled ❤️ .

[deft rain]

obviously agreed with @sharp zealot above, it's all about compromise, and trying to not break anything unnecessarily - i like the thing about trying to keep rust off a ship

but also, potentially - if we wanted a similar experience in go, we could just make all args named with our big opts struct 🤔 instead of automatically only putting optionals in there

huge breaking change, but would solve this as a problem (maybe a third-party go sdk might appear that did this one day 🎉)

[dense canyon]

but also, potentially - if we wanted a similar experience in go, we could just make all args named with our big opts struct 🤔 instead of automatically only putting optionals in there

/me cries in Gopher 😄

[sharp zealot]

DX feedback. While designing my gha generator module, I found that I have to choose between 1) a good CLI experience, and 2) a good code experience. I can't have both. This is mostly because the CLI is too limited in what input types it supports.

• CLI can't pass structured types, structs etc
• CLI also can't stitch multiple calls together

If I had one of those 👆 , I could make it work

[sharp zealot]

@thorn moat is there a way to disable colors with --progress=plain?

[thorn moat]

NO_COLOR=1 (https://no-color.org/)

[sharp zealot]

https://tenor.com/view/hell-yeah-bitcoin-ethereum-eth-btc-gif-27338496

Context: https://github.com/shykes/gha/actions/runs/10396283376

Note the pretty markdown report 🙂 With ugly control characters at the bottom 😦

[thorn moat]

lol nice

[sharp zealot]

I'm telling you once you start generating your GHA files there's no going back. My generated config is so much better already, and the pace of improvement is crazy

[sharp zealot]

no-color diff: https://github.com/shykes/gha/commit/f4570b76f269a4f74fb07f43c1cfe488cba0cecb

note that the diff is in a proper shell script which you can run locally

Yay it worked! https://github.com/shykes/gha/actions/runs/10396376561/attempts/1#summary-28790255357

[sonic vessel]

Awesome!! 🎉

[high storm]

This may be the wrong location to ask but is there a plan to be able to integrate GitHub shared actions into these generated pipelines?

[sharp zealot]

Well to be clear, this is my personal experiment at this point, so there is no official plan 🙂

At the moment I don't rely on shared actions, except for "checkout". What's your use case?

[restive shore]

Bundle CLI in the Engine image so that both parts of Dagger (CLI+Engine) ship as a single artefact by @strong ingot in #8147

I don't understand the benefit of this. How is this useful to a Dagger consumer? We still have to install the dagger CLI to match the engine right? What am I missing?

[high storm]

Mainly on release focused actions like https://github.com/marketplace/actions/release-please-action and https://github.com/github/branch-deploy which can be a bit tricky to manage without the actions

Which these can use dagger for example using the release-please CLI or building out the issue ops commands myself but was just curious on the thought of adding them as possible intermediate pipeline steps to speed up adoption @sharp zealot

[sharp zealot]

Well there are 2 separate things here:

1. The current best practices in Daggerizing an existing Github Actions project
2. Possible future improvements via my experimental generator module

The current best practices involve:

• Incremental adoption: it's fine to "daggerize" some of your GHA pipelines while leaving others alone. Sometimes you will even see a big pipeline get partially daggerized
• Copy-pasting snippets for fully daggerized pipelines. Once a given GHA pipeline is fully daggerized, the remaining YAML config is fairly generic. We provide a generic snippet in our docs, which you can then adjust

In that context, what you're describing is perfectly fine, and very common.

The improvement I'm exploring is specific to fully daggerized pipelines. It's about replacing the manual copy-paste of a snippet, with a fully automated generation instead.

What you're describing is a partially daggerized or not-yet-daggerized pipeline, and is out of scope for my module

I want to make sure my module supports mixing generated GHA config files with manually edited ones.

[sharp zealot]

This is for production deployments, for example when deployment Dagger on kubernetes. For example healthchecks are easier if you can have kubernetes execute a dagger command directly in the engine pod

[restive shore]

Ah! Actually, it just occured to me. We run Jenkins in EKS and we use docker-in-docker to run all our workloads within pods. With this change I won't have to download the CLI within the pipeline. Which is pretty slick!

[high storm]

Thank you this makes a lot of sense and helps me too understand the use case!

[sharp zealot]

That said, it's not impossible that the generator tooling could be extended beyond the original scope, and reused for generating even non-daggerized pipelines 🙂 It's technically feasible, but is it actally useful? Unclear at the moment

I know @deft rain is interested in such a generalization in the future

[deft rain]

yes 🙂

i have a neat idea for this, it needs some cli/engine work, but i might fork your module to show something 😄 (assuming time, etc 😢)

[wheat river]

Hey guys 👋 -
I've been facing (randomly, just occurred 3 times today) this error. Anyone else got the same? v0.12.4

! call constructor: process "/runtime" did not complete successfully: exit code: 2
┃ marshal: json: error calling MarshalJSON for type *main.Mymodule: json: error calling MarshalJSON for type *dag
┃ r.Container: input: container.from resolve: failed to resolve image docker.io/library/alpine:latest: failed to
┃ solve source metadata for docker.io/library/alpine:latest: failed to authorize: failed to fetch oauth token: un
┃ pected status from GET request to https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&servi
┃ =registry.docker.io: 502 Bad Gateway
  ✘ Container.from(address: "alpine:latest"): Container! 10.7s
  ! failed to resolve image docker.io/library/alpine:latest: failed to resolve source metadata for docker.io/libr
ary/alpine:latest: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https:
//auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io: 502 Bad Gateway
    ✘ resolving docker.io/library/alpine:latest 10.7s
    ! failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://auth.docker
.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io: 502 Bad Gateway
      ✘ HTTP GET 10.6s

[deft rain]

hm, 502 generally a server-side issue 🤔

https://www.dockerstatus.com/

doesn't seem to be reporting any auth issues

[wheat river]

It happened 3 times only out of plenty of dagger <> executions. It's worth mentioning that I haven't seen it other days, just today. Retrying just works.

[restive shore]

I tested this and it works flawlessly! I don't have to install the dagger CLI anymore in my pipelines. I did have to install the docker-cli (apk add docker-cli) as I need it to run docker login. We use a sidecar dind image and have the DOCKER_HOST set to that in the dagger engine. It's engine-ception because now everything is running in dagger within the dagger engine.

[sharp zealot]

@restive shore nice 🙂

Heads up, we're going to build the entire prod architecture on top of this.

For example instead of CI runners alongside dagger engine, reference architecture will be CI runners on dagger

this allows us to get rid of the pesky _EXPERIMENTAL_DAGGER_RUNNER_HOST socket, which is 1) undocumented and 2) unsafe

[restive shore]

Interesting! I may need to see more details around that as I don't think I get the picture. Is that an attempt to get away from CI orchestrators? The _EXPERIMENTAL_DAGGER_RUNNER_HOST is a necessary evil for us right now as I don't see a way around it because we need a custom engine container within the Enterprise.

[sharp zealot]

Yes the custom engine container will eventually run the CI runner as well

So you'll get a combined "Dagger-powered CI runner" unit which you can deploy and scale at will

That CI runner and Dagger engine are tightly coupled anyway.

[restive shore]

ah! I see. Well that is cool indeed! Long ways to go for us to adopt something like that because we are very much married to our CI orchestrators. Primarily because of audit and compliance reasons. Lots of supporting infra has been built around (jenkins) it like log/build retention, security, support etc. I can see us using it for the foreseeable future (with GHA looming in the distance). I could be wrong.

[sharp zealot]

How do you run dagger engines alongside those blessed CI runners today?

[restive shore]

Until today, within a fairly vanilla Ubuntu container and a sidecar dind. Today I switched to running within the Dagger container itself. We are using Cloudbees core which is hosted on EKS. Each agent (runner) is a EKS pod with one to many containers.

[sharp zealot]

how does the dagger CLI connect from the CI runner to the dagger engine?

[restive shore]

_EXPERIMENTAL_DAGGER_RUNNER_HOST with my custom dagger engine.

until today the CLI was installed ad-hoc in my Ubuntu container. But today I started using the dagger engine image as my container. So the CLI comes with it. It still pulls another dagger engine (my custom one) when running. That's why I called it dagger-ception

unless there is a better way to use the engine inherent inside the dagger-engine container?

[sharp zealot]

I think I need a diagram 😁 does ..._RUNNER_HOST point to a unix socket? A tcp address? Do all CI runners connect to the same dagger engine, or does each CI runner get a dedicated engine?

[restive shore]

Haha don't blame you.

Each runner gets a dedicated engine.

RUNNER_HOST connects to a docker-image

I'm about to head out but I can give more details later

[sharp zealot]

tcp or unix socket?

https://tenor.com/view/game-gaming-puzzle-box-penguin-gif-1678862369754362426

[restive shore]

Neither. Literal docker image. Not at the PC to get the exact value but it basically pulls a new engine image from our private registry

[sharp zealot]

ah!!!

[restive shore]

Now I'm curious. There has to be a better way to just connect to the engine inside the container right? Without pulling a new one?

Is the CLI inside the dagger engine aware of the engine?

[sharp zealot]

Yes the best way is to have the engine run the CI runner 🙂 Then the dagger CLI can use introspection to talk to the same engine running it

Like a unix process talking to the kernel to spawn more processes

As long as the CI runner is executed with Container.withExec(experimentalPrivilegedNesting: true)

then any dagger client running inside that container will magically find that engine

[restive shore]

Interesting. Could that same magic can be applied directly to the CLI within the engine to understand its running inside the engine container and just use the local engine instead of pulling a new one?

[sharp zealot]

yes that's exactly how it works 🙂 all SDKs rely on dagger session under the hood, so the CLI is the source of truth for this. All dagger commands will magically discover the nested session in this way

[restive shore]

Oh interesting! Let me try that

[sharp zealot]

remember to use that nesting flag in withExec (we'll make it the default eventually)

[rapid heart]

is there an example of this?
I just started reading into Dagger. Love the idea, and I have daggerized an application.
I have Jenkins setup so that when a new pipeline run, it spawns a POD, with one/many containers, as defined by the spec, to execute the pipeline.

Initial step for dagger with CI -- as I understand it:

With dagger, I believe I would need only two containers in the POD, one a DIND where the docker daemon is running, the other a docker-client and both containers will share a volume with the docker-socket. It is very likely that the docker client exists in the DIND image, but i prefer to have a separate client container. In the docker-client container I will install the dagger-cli. When I execute dagger call -m .... commands it will create a dagger-engine container and execute the commands there. Once all the commands are executed, the pod (jenkins agent) will be terminated and then garbage collected. This is not great because I will not be makins use of the dagger cache.

Given the above, i do not understand this statement:
"Yes the best way is to have the engine run the CI runner "

how can i get the dagger engine run th eCI runner?

[restive shore]

The CI runner Solomon is referring to does not exist yet. It's on their roadmap.

[restive shore]

I don't think this will work in it's current state. How will I run commands that may not have a withExec like dagger core or publish. I would have to nest all my modules under a parent "CI Runner" module that has the withExec with nesting.

I tried simply running a dagger core version command from the dagger CLI within the engine image but it tries to pull the external dagger engine image. It's not a huge deal right now as it only takes a few seconds (can take upto a minute to init) to pull the engine but it would be cool if the CLI inside the dagger-engine was "engine-aware".

[sharp zealot]

sorry I wasn't clear. You only need that flag when executing the Jenkins runner. Then everything else is 100% transparent

[sharp zealot]

Yes I agree with that, we will fix that cc @strong ingot

[restive shore]

Right.. This is future state when runner support is available. Not something I can do in my setup today

[strong ingot]

I tried simply running a `dagger core

[echo spindle]

I guess I am not 100 percent following. So today we run dagger in gitlab runners that are hosted on kubernetes. The dagger engines are setup based on the blog post about how you guys run(c5ds with nvme disks). We get scale from k8s and own the inf end to end (gitlab and k8s etc). So the gitlab runner pulls and image (right now it is an image with the dagger cli, python, python sdk installed though we are close to getting modules working thanks to recent efforts go proxy etc). So we have a daemonset of engines and then runners that run on those same nodes via affinity and taints/tolerations. So you are saying that instead of a daemonset the dagger engine would be bundled with the image that gitlab runner runs? And the ds goes away? Maybe I am following...

[sharp zealot]

So you are saying that instead of a daemonset the dagger engine would be bundled with the image that gitlab runner runs? And the ds goes away? Maybe I am following...

Instead of running 1) the gitlab runner in a kubernetes deployment, and 2) the dagger engine in a separate kub daemonset, instead you would run 1) the dagger engine image, with 2) the gitlab runner run by the dagger engine

One caveat is that I am not super familiar with this part:

the gitlab runner pulls an image

The relationship between the runner and that image may be something to take into account in what I'm describing. (Github Actions doesn't have a direct equivalent I don't think)

[echo spindle]

So how do I get all my k8s scaling? Today that is drivine by the runner (memory requests, cpu reqs, in the gitlab-ci.yml def) definition on k8s? Maybe that still works...

[sharp zealot]

Exactly same as before.

It's the same CI runner as before, just "dagger powered"

basically the dagger engine is inserted as a shim, specifically so that you can apply the same kubernetes strategy without special case (eg. daemonset)

[echo spindle]

So still confused. Sorry. Today there is a process that runs on k8s (call it the gitlab runner orchestrator). It pings gitlab and says "What work do you have for me?" Gitlab says "I have this task." The orchestrator says "Great, I'll start a pod." How does dagger shim this?

[sharp zealot]

But in the case of Gitlab, there seems to be 2 different possible insertion places for the shim: 1) between kub and gitlab runner, 2) between gitlab runner and the image you mentioned.

[sharp zealot]

I think that 👆 is the answer.

Ah the shim is for the runner not the orchestrator

Let me rephrase with that terminology confusion taken into account

(actually I have to run to an event but will return later today with a clearer explanation - sorry for the confusion)

[echo spindle]

No problem. I am very curious. Thanks Solomon!

[sharp zealot]

I'm assuming you're using the DOcker executor? https://docs.gitlab.com/runner/executors/docker.html

[echo spindle]

No.

K8s executor.

[sharp zealot]

If you're able to share an example config file, that would be super useful

(even sanitized is fine)

[echo spindle]

Sure. First we are using this. And we use these env variables to drive memory requests and cpus. And here is a sample gitlab_runner ci, it isn't very interesting:

variables:
  _EXPERIMENTAL_DAGGER_RUNNER_HOST: "unix:///var/run/dagger/buildkitd.sock"
  ARGOCD_SERVER: argocd-server
  KUBERNETES_CPU_REQUEST: "3"
  KUBERNETES_CPU_LIMIT: "5"
  KUBERNETES_MEMORY_REQUEST: "2Gi"
  KUBERNETES_MEMORY_LIMIT: "4Gi"

before_script:
  - git clone https://${CI_SERVER_HOST}/dagger/common.git /dagger

build-job:
  tags:
  - dagger
  image: local-registry.com/dagger-python:current
  stage: build
  script:
    - git clone  https://{CI_SERVER_HOST}/argocd/app-parent.git /parent
    - DO_NOT_TRACK=1 dagger run python /dagger/docker-java-build.py $CI_PROJECT_NAME

So the way it works is that the k8s executor is tagged with dagger. It says "Ha, I have work, let me start a pod." It then starts a pod with the image specified and also with the cpu and memory requests. If the machine the ds is on doesn't have enough memory (doing other jobs) it scales out a new machine to accomdate the 3 cpus and 2Gis of memory, New machine comes up, daemonset starts, and the pod is scheduled and runs. That means we can define our cpu/memory usage on a per job basis, etc

[sharp zealot]

future Gitlab CI architecture

[west magnet]

Is there a nix package for the CLI?

[wheat river]

It's useful, the one I use is https://github.com/Excoriate/daggerverse/blob/main/flake.nix#L57-L57 - Additionally, another juggernaut posted something related here: #general message

[deft rain]

we also have https://github.com/dagger/nix/, not sure if that's something you might be interested in?

there's also a dedicated #nix channel you might be interested in 👀

[plucky ermine]

I seem to have forgotten the incantation to pass a PR as an argument to Directory, can someone point me in the right direction? 😇

[sharp zealot]

• For a module: github.com/FOO/BAR@pull/NUMBER/merge
• For an argument: https://github.com/FOO/BAR#pull/NUMBER/merge

[plucky ermine]

You are the best, Im going to add this to the cookbook! ✍️

[sharp zealot]

BTW we haven't talked about the integrations pages a lot, but IMO this deserves a paragraph in the Github integration page

(not github actions)

[plucky ermine]

Oh, is this is a GH specific thing? I assumed it worked for others as well

[sharp zealot]

yes gh specific. Other SCMs (eg gitlab) might have an equivalent but to be checked on a case by case basus

[wintry prism]

Using dagger v0.12.5 cli (nevermind how I got it 😉 )

dagger core container from --address registry.dagger.io/engine:v0.12.5 with-exec --args dagger,version stdout

[latent trellis]

How do you implement pipeline steps that may fail (and ultimately should fail the pipeline), but always yields some report. Typical example: test or lint results.

From what I can tell, if the test/lint command fails, it's game over for that particular container. There is no "continue on error".

I can wrap the command in a shell and always return 0 exit code, but that feels like a hack and require additional magic to actually get the failure (eg. write the exit code to a file?).

(Ultimately, what I'm trying to do is produce a markdown report of all the different test and lint runs. There is a lot of noise in the logs and it's not easy to spot failures, so I'm trying to make it easier by highlighting failed tests and lint violations on the GitHub Actions summary page)

[restive shore]

I've been using 0.12.5 since last week. What am I missing? The dagger core version command came in really handy for initializing the engine before starting a pipeline.

[wintry prism]

dagger core --help

FUNCTIONS
  blob                Retrieves a content-addressed blob.
  container           Creates a scratch container.
  current-type-defs   The TypeDef representations of the objects currently being served in
                      the session.
  dagger-engine       The Dagger engine container configuration and state
  default-platform    The default platform of the engine.
  directory           Creates an empty directory.
  git                 Queries a Git repository.
  host                Queries the host environment.
  http                Returns a file containing an http remote url content.
  module              Create a new module.
  module-dependency   Create a new module dependency configuration from a module source and
                      name
  module-source       Create a new module source instance from a source ref string.
  version             Get the current Dagger Engine version.

Indeed dagger core version is in there, but I was just delighted by using dagger to pull a Dagger Engine container (which as of v0.12.5 now has the Dagger CLI inside) and invoking dagger version inside it 🤓

[restive shore]

haha gotcha! I've been waiting for core for a while to replace my hacky "run a dagger call -m module functions" to initialize the engine at the top of the pipeline.

[plucky ermine]

Added PR for new Github integration guide 🙂 Thanks again

https://github.com/dagger/dagger/pull/8194

[latent trellis]

I'm assuming that secret names are supposed to be unique.

So this is what I'm doing for generated secrets:

    if jwtSecret == nil {
        secret, err := generateRandomString(20)
        if err != nil {
            return nil, err
        }

        h := sha1.New()

        _, err = h.Write([]byte(secret))
        if err != nil {
            return nil, err
        }

        name := fmt.Sprintf("svix-generated-jwt-secret-%x", h.Sum(nil))

        jwtSecret = dag.SetSecret(name, secret)
    }

It's kinda annoying to be honest. I know we talked about this a while ago, but I wanted to bring it up again.

[sharp zealot]

Sorry about that Mark... I added your comment to the issue (https://github.com/dagger/dagger/issues/7358)

[spiral whale]

Hey, I saw private git support (https://github.com/dagger/dagger/pull/7708) merged but couldn't figure out how to use it for private it daggerverse repo. Is it supported at the moment or is this only accessing private git repos not daggerverse

[wheat river]

👋 Is it supported (ref: https://github.com/dagger/dagger/issues/8202), returning multiple containers from a Dagger function?

[plucky ermine]

feat(core): add private git support via ...

[wintry prism]

Yes, it is supported in the SDKs, but it seems we don't have a good way to handle it from the CLI yet. For example, you can run:

package main

import (
    "dagger/reproduce-bug/internal/dagger"
)

type ReproduceBug struct{}

func (m *ReproduceBug) Containers() []*dagger.Container {
    return []*dagger.Container{
        dag.Container(dagger.ContainerOpts{Platform: "linux/arm64"}).From("alpine:latest"),
        dag.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).From("alpine:latest"),
    }
}

func (m *ReproduceBug) First() *dagger.Container {
    return m.Containers()[0]
}
func (m *ReproduceBug) Second() *dagger.Container {
    return m.Containers()[1]
}

@kind carbon for when you return 👆

[wheat river]

Maybe it's worth to continue in that issue. It makes sense that it's supported at the SDK level (from the GO perspective). What do you mean handle it from the CLI, it's ex pected to handle gracefully when two containers are returned from the CLI? - What'd be the use case (return two containers in a dagger call -m function-that-return-two-containers)? — I think if it's not expected from the functional aspect of a return of multiple containers from a function, maybe the CLI should just handle the error properly with a better error message. 🤔

[sharp zealot]

The CLI could simply print the array, with metadata about each container. Or it could chain other function calls against each container, in a fan-out pattern. That might actually work today.. `

What Jeremy means is that the Dagger API supports returning an array of containers just fine, it's just one API client (the CLI) that doesn't know how to handle the result properly, in some cases.

[willow carbon]

a dagger module for portage ebuilds would be cool

[leaden pagoda]

Regarding the recent GHA demo. I wonder if there's something even bigger here as an idea for dagger and 3rd party CI Platforms. Most platforms have some way of manual inputs, and outputs into artifacts along with some specific CI Platform features like Releases or recognised 'environments'. Would it be possible for some kind of introspection to take place on functions that could be attributed/annotated with a CI attr. If those functions return a directory, the proceeding yaml generated for a target CI platform would ensure the dag directory artifacts are extracted/uploaded. If the function has several input parameters could they be mapped to custom CI platform inputs like dropdown box/basic input form inputs that users fill in before those values are passed on to the dagger function input arguments. Essentially the idea is very much solomans GHA module, but an idea of a kind of framework hooking into specific CI features by understanding what a dagger ci function is doing, what its returning, what arguments are passed to it. Probably this has already had discussions, its wayy more complicated i imagine but it sounds great. Write functions and it 'knows' what you really need setup based on the code somehow. Regarding uploading of artifacts, not sure if this could be handled within a dagger function maybe using a github API, i know I was using marks gha release function in a dagger function, meaning it wasn't part of the github workflow yaml and that was fine.

[sharp zealot]

Yeah I've been wondering how far we can push the capabilities in this area. Introspecting the Dagger module to expose a "smarter" Github Actions workflow could be pretty cool. I think there might be a local maximum, where the effort to "transpile" to Github Actions just isn't worth it, and it's easier to just rip off the bandaid and Daggerize.

I think the smartest way to approach this is to find a concrete use case, and see how to implement an awesome solution.

For example, uploading the output as a GHA artifact is easy enough - no need to introspect there, it's just a flag. But, you can also just do that in code. So it becomes a convenience.

Does this error ring a bell? github.com/shykes/gha/internal/dagger: no matching versions for query "latest"

I'm getting the issue in CI, but can't reproduce locally

[wintry prism]

Golang thing, right? Different versions of Dagger local vs runner thus different go versions?

[sharp zealot]

yeah it was a shared go.mod..

https://tenor.com/view/the-simpsons-angry-grandpa-gif-14233249

[sonic vessel]

How to access a service on interactive terminal (option -i)? I have some problems with the service setup in my test pipeline and want to check in terminal after test failure but it seems it cannot access the service via the debugger.

[sharp zealot]

It seems service bindings get wound down before giving you access to the debug terminal.

But, you can make an explicit call to Container.terminal() in your code. That will open an interactive terminal, with all service bindings working

[sonic vessel]

Yeah, just manage to make it work by using Container.terminal . 😹

[sharp zealot]

Example in the CLI:

In first termina:

dagger core container from --address=nginx as-service up --ports 80:80

In second terminal:

dagger core container from --address=alpine with-service-binding --alias www --service=tcp://localhost:80 terminal

(really wish I could pass a container address as a service argument to CLI)

[sonic vessel]

Thanks for the suggestion. 🙏

[lime comet]

I've been iterating on a few ideas for pocketci that I would like to open up and get some feedback on. In case you don't know what pocketci is, here is a quick community call demo you can see: https://www.youtube.com/watch?v=xcWcR4rZy44.

We've been trying to see how far we can take the Daggerization. There is a good talk you should check out from @strong ingot titled The Square Hole: https://www.youtube.com/watch?v=s_Fxpm1pp3I. In this talk he explains how we can make our favorite tool the "right fit" for pretty much any problem, simply because its our favorite tool and we are familiar with it. That doesn't necessarily mean that its the right tool for the job, but it felt good for you because you know it well. Its no secret that Dagger for me is that tool, and I've been trying to see how much of the CI specific YAML I can move into my Dagger code. If you daggerized a pipeline, then your pipeline is somewhat slim. You have a section were you define a vendor specific way of triggering your pipeline (e.g GitHub's on: section) and then a list of steps, which if fully daggerized, should involve one or more dagger call commands. The question is: could we move the event triggering logic to Dagger as well? Would that actually be a better experience than using a vendor-defined config language? Could this also allow even more flexibility when it comes to moving away from a CI platform onto another one?

I'm playing with these ideas in the open, most of it happens in these repo via PRs, issues and discussions: https://github.com/franela/pocketci. If you want to try it out and give feedback, the place to start is the README. There is a section called Guide that explains how to get started.

[latent trellis]

I don't know if it's a recent change, but optional dagger.Platform arguments result in the following error (when the argument is empty):

resolve: failed to set call inputs: failed to convert arg "platform": "" is an invalid OS component of "": OSAndVersion specifier component must match "^([A-Za-z0-9_-]+)(?:\\(([A-Za-z0-9_.-]*)\\))?$": invalid argument

99% of my dagger projects are stuck on 0.12.0 and I don't see it there, so my guess is this was introduced since, but I don't know where. (It's happening on 0.12.6)

[wintry prism]

@deft rain 👆

[deft rain]

will investigate early next week, need to head off for the weekend right now sorry!

[wintry prism]

We’ll dig in 🙌

[deft rain]

i think there's something in the way we do conversions/etc, i think there was something similar with optional enums

see https://github.com/dagger/dagger/pull/8148

[restive shore]

So glad the cotext dir is shipped! Although I can't yet use it properly without private Git (PAT) support :). However, I wanted to give some feedback on another thing that I keep having to repeat for every single command - user and password. Especially in a corporate environment. I feel like I'm going to end up having to pass those in to every command.

Do you want a maven build or test? Need user/pass to auth to Artifactory. Do you want to run a sonar scan? user/pass again to auth. Do you want to build and publish a docker image? Need user/pass because I want to search my artifact registry before promoting it etc. etc. I hope as part of context dir or an extension to existing API we can figure out a way to default some of these.

[sharp zealot]

That is next on the "contextual access" list 🙂

Could open an issue with a list of the credentials you need to access in that way, and how you pass them today? Ideally specific env var names, file names, commands executed etc. one question mark for auth contextual access is how standardized and strict we can make it. The more open-ended, the harder it will be to secure

[restive shore]

Are you asking me to open an issue with a list? If so, yes, happy to! Or if you have something already I can add to it.

[wintry prism]

Yes please! 🙏

[restive shore]

I have to head out but created a quick issue. I'll add further use cases when I encounter them too. https://github.com/dagger/dagger/issues/8286

[sharp zealot]

thank you! yes that is what I meant 🙏

[frosty lion]

Context dir is a great feature, I really like not having to pass the source argument!

Has anyone tried to export the output of tools like golangci-lint run --fix? I suppose I could try to export --path=. a *dagger.Directory but I can't tell if that's an accepted pattern or a terrible idea, possibly unsafe, etc.

[astral crag]

hi, once have a look at these projects:
https://direnv.net/ &
https://dotenvx.com/docs/platforms/docker
dotenvx encrypts & decrypts using .env.keys file (to be ignored in .gitignore & .dockerignore ).

[deft rain]

I don't know if it's a recent change,

[wintry prism]

@kind carbon can we use something like this with the Python SDK?

class Infisical:
    infisical_client: InfisicalSDKClient = dataclasses.field(init=False)
    api_url: Annotated[
        dataclasses.InitVar[str],
        Doc("Your self-hosted Infisical site URL. Default: https://app.infisical.com."),
    ] = "https://app.infisical.com"

    def __post_init__(self, api_url: str):
        self.infisical_client = InfisicalSDKClient(host=api_url)

The __post_init__ in particular

[hybrid heart]

Hi @kind carbon @wintry prism

This is Akhi from Infisical. We are implementing the infisical dagger module and using the chaining pattern. Repo. We are getting the below error if we chain it with three functions:

__post_init__ missing required api_url error

[wintry prism]

Hi @Helder Correia @Jeremy Adams

[latent trellis]

Idea: scaffold projects using Dagger npx style.

Using Terminal, anyone can build a module that asks for a number of inputs and generates a Directory, generates templates, etc.

Demo for custom TUI: dagger call -m github.com/sagikazarmark/demo-dagger-tui terminal

[ember aspen]

can i use dagger as a drop in replacement for github actions / workflows?

[wintry prism]

can i use dagger as a drop in

[sharp zealot]

This comes up a lot, so I finally wrote the issue for it: dagger doc 🙏 https://github.com/dagger/dagger/issues/8334

[leaden pagoda]

I did something stupid on purpose to find out what the error would be, i made a module, then installed the module into itself as a dependency and tried to run it. It let me install itself and only fails on running the module. This is obviously silly why would someone do this, but just wondering if anyone thinks its overkill to somehow validate a 'dagger install' isn't somehow referring to itself.

hmm, now i wonder what happens if install it with another name

[leaden pagoda]

Can a public dagger module, published on the daggerverse, reference a private ssh dagger module and should it be allowed?

[deft rain]

it likely won't end up published to the daggerverse

(not tested) but the module should fail to be analyzed, since the daggerverse scraper doesn't have access to the private module

reasonably confident that that's the right behavior though

[leaden pagoda]

should a module not defined as private, show a warning if referencing a private module? "hey, your dagger module isn't declared as a private module, are you sure you want to continue?" User error could be publishing something including the private repo in a public place, some of that is 100% down to the individual to make sure they aren't exposing anything they dont want to though.

i think its minor if anything, was only thinking from a DX perspective of public/private modules and such things

yeah, i guess if what you say is true, it might be publically available (down to the user) and it cant be pushed on the daggerverse, since it would fail. So it's not like dagger would be showing something it couldnt analyze correctly.

[subtle plinth]

I bet hooking this up to projen would be pretty sweet

[sharp zealot]

I have an open-ended question. Given a list of modules, how can I dynamically inspect them for types that match a given interface, and call the interfaces? Importantly, the module list is an argument to my function - so it's only known at runtime. I can't just dagger install them.

[sharp zealot]

If someone's looking for a quick contribution, the Dagger repo's CI pipelines have an inline call to Trivy, should use a reusable Trivy module instead

[wintry prism]

Hmmm...mine is referenced in some spots. I'd be honored

[wintry prism]

@deft rain you made some of those changes recently, so let us know what features we need there (existing or on wishlist) so we can pick/adapt/create the right module (silent ping)
I'll take a peek too and see if there's a match already.

[sharp zealot]

@wintry prism what I've been doing, is:

1. Spin out what's already in the CI module, into github.com/dagger/dagger/modules/<FOO>, then calling that. For the least intrusive possible first step

2. As a follow-up PR, refactor the newly spun out module, to incoporate the goodness from existing modules in daggerverse

Over time, the implementation in dagger/dagger looks more and more like the pre-existing dagger modules

Eventually, github.com/dagger/dagger/modules gets hardened and improved by real-world usage, and we promote it to stdlib

[wintry prism]

nice. makes great first issue material!!!

[sharp zealot]

@warped canyon thought you might like this 🙂 Coming soon in a pipeline near you

// A sidecar service with zeroconf binding
type Sidecar interface {
    Service() *dagger.Service
    Bind(context.Context, *dagger.Container) (*dagger.Container, error)
}

[sharp zealot]

Does anyone have experience using Dagger interfaces? I'm trying to refactor a module by inserting a Go interface, then spinning out parts of the implementation into a sub-module (using the interface). But I'm finding it very difficult to mix Dagger interfaces and "regular" Go interfaces

It seems that either my interface can either be usable as a regular Go interface, or as a Dagger interface, but not both

./dagger.gen.go:407:11: undefined: sidecarImpl
./main.go:333:33: cannot use dev.Engine() (value of type *Engine) as Sidecar value in argument to NewGoSDK: *Engine does not implement Sidecar (missing method XXX_GraphQLID)

If I remove the DaggerObject from my interface (to try and use it as a regular Go interface), I still get this:

./dagger.gen.go:407:11: undefined: sidecarImpl

(my interface is called Sidecar)

@thorn moat do you have any experience with interfaces by any chance?

(removing the interface for now)

[warped canyon]

Nice! Do you think some interfaces like this might find themselves in stdlib or even core?

[sharp zealot]

yea definitely

I think leveraging interfaces to build higher-level abstractions is the next big unlock for the platform

we were just discussing that with @thorn moat yesterday

and by the way, our core types should all be interfaces too 🙂

[deft rain]

yes - did some work a while ago in https://github.com/dagger/dagger/pull/7710

will rebase this and merge it in

then if we do future work on this we make sure to capture those changes too

[leaden pagoda]

Module in Python + Go/Python SDK 'Consumers'

[sharp zealot]

I kind of want a Container.WithError and Directory.WithError... Anyone else encountered that?

When you have a beautiful chain of 20 function calls, no error checking end-to-end... except the one in the middle, then you're forced to break your chain, plumb error checking all the way up... So much beauty ruined.

But instead I could do:

func Beautiful(ctx context.Context) *dagger.Container {
  return dag.Container().
    Foo().
    Bar().
    Baz().
    With(func(c *dagger.Container) *dagger.Container {
      contents, err := c.
        WithExec([]string{"check", "the", "thing"}).
        File("the-thing-output.txt").
        Contents(ctx)
      if err != nil {
        return c.WithError(err)
      }
      if contents != "expected-thing" {
        return c.WithErrorf(fmt.Errorf("The thing wasn't the expected shape! Failed to check the thing"))
      }
      return c
    }).
    Bla().
    Bla().
    ChefsKiss()
}

Sometimes I actually just panic() in that chain 😛 But not great with intra-module calls

[warped canyon]

makes sense to me!

[sonic vessel]

I wonder that I cannot reference the object from another module like this:

type A struct {
    Source *dagger.Directory
    Elixir *dagger.Elixir
}

The Elixir is from another module. When running dagger functions or dagger call, it returns error:

failed to initialize module: failed to add object to module "a": failed to validate type def: object "A" field "Elixir" cannot reference external type from dependency module "elixir"

Found similar issue if I convert Elixir field into method as well.

The reason that I do this way is my monorepo categorize services, lib per language (elixir, js, go, etc.). So I want the main module to delegate all the works to sub module rather than having a big monolith module. But it seems it cannot. 😭

[latent trellis]

External module fields

[plucky ermine]

I upgraded engine to 0.13 and now a previously working pipeline fails due to a dependent module

✘ ModuleSource.asModule(engineVersion: "latest"): Module! 2.2s
! failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "plausible" to get functions: call constructor: process "uv pip check" did not complete successfully: exit code: 1

Does anyone have any ideas?

Offending module appears to be this one, but it worked just fine in 0.12.7

https://daggerverse.dev/mod/github.com/levlaz/daggerverse/plausible@04376aba3bbdf5c3614f1b04b9e6ae1888b5c80f

[echo spindle]

I have some use cases where it would be nice to pass arguments to an sdk. Was gonna write a ticket but thought I would check if there was strong opinions against ..

[sharp zealot]

Lots of interest in this topic 🙂

[plucky ermine]

I wonder what the best way would be

[proud verge]

Using the Go SDK for Dagger.... is there a way to make function arguments optional or to set defaults?

[sharp zealot]

@proud verge yes, it's in the docs though a little hidden:

• Optional arguments: https://docs.dagger.io/manuals/developer/functions#optional-arguments
• Default values: https://docs.dagger.io/manuals/developer/functions#default-values

[proud verge]

Thank you very much!

[sharp zealot]

@plucky ermine is there a regression in your docusaurus module with 0.13? I'm getting a weird error related to typescript SDK when trying to use it in my docs PR

https://dagger.cloud/dagger/traces/7fdb8c73579d5f500c15fb4c3cef2b3c?span=3328c60366985aca

cc @upbeat herald @kind carbon

Looks like a SDK co-existence issue?

[plucky ermine]

I have not tested it in 0.13 but ill take a look today

[sharp zealot]

Here's an example of how lack of self-calls prevents me from fully taking advantage of context dir.

func (m *MyModule) LintPython(
  // +optional
  // +defaultPath="/"
  // +ignore=["*", "!**.py"]
  source *dagger.Directory,
) error { ... }

func (m *MyModule) LintGo(
  // +optional
  // +defaultPath="/"
  // +ignore=["*", "!**.go", "!go.sum", "**/go.sum"]
  source *dagger.Directory,
) error { ... }

This 👆 is a really neat pattern. I don't need to carry state around that much. I just get what I need from the context (or directly from the caller).

But what if I want to aggregate?

Ideally I would do this:

func (m *MyModule) Lint() error {
  if err := m.LintPython(); err != nil {
    return err
  }
  if err := m.LintGo(); err != nil {
    return err
  }
  return nil
}

Very cool. And in fact this works! But only when calling from another module. If I want to do this in the same module however... It looks like this 👇

func (m *MyModule) Lint(
  // +optional
  // +defaultPath="/"
  // +ignore=["*", "!**.py"]
  pythonSource *dagger.Directory,
  // +optional
  // +defaultPath="/"
  // +ignore=["*", "!**.go", "!go.sum", "**/go.sum"]
  goSource *dagger.Directory,
) error {
  if err := m.LintPython(pythonSource); err != nil {
    return err
  }
  if err := m.LintGo(goSource); err != nil {
    return err
  }
  return nil
}

😭 Not as nice. I have to copy-paste those ignore patterns everywhere, and just write more code for no reason.

Anyway. One more reason that self-calls will be very nice!

[upbeat herald]

Intestesting, I need to check but it's possible because I'm more strict with which file from the module are copied, might lead to that issue

[wheat river]

What about a daggerignore file?

[subtle plinth]

I'm pretty sure there used to be a .daggerignore, back in ye 'ole times

[subtle plinth]

I've been exploring a new pattern for using Dagger within a mono-repository to provide global and shared configuration across many services.

It's working really well!

import { object, func } from "@dagger.io/dagger";

    /**
 * Global Rawkode Academy Config
 */
@object()
export class Config {
    /**
     * Cloudflare Account ID
     */
    @func()
    cloudflareAccountID(): string {
        return "0aeb879de8e3cdde5fb3d413025222ce";
    }
}

You can then dagger install ../../../ to grab the root config, or specific config from anywhere in your repository.

[robust gate]

ive been hitting this for a few days now, both at work and on my home PC, im not sure why this is happening. either when i was running a custom engine but right now im not using a custom engine.

13:28:02 WRN failed to get stable ID, defaulting to random value error="failed to rename stable ID temp file: rename C:\Users\mageep\AppData\Local\dagger\stable_client_id2326
┃ 7246 C:\Users\mageep\AppData\Local\dagger\stable_client_id: The process cannot access the file because it is being used by another process."

I can wait long enough and hope it resolves itself or im just stuck and kill it and try again.

wait, maybe this is due to a env variable which i never reset...

[robust gate]

Im thinking this might occur for others from time to time, but theres no good message or feedback. Would it be possible for the client to somehow realise no engine was found with the the variable set?

[wintry prism]

https://github.com/dagger/dagger/issues/8448

[wintry prism]

@kind carbon when trying to publish https://github.com/Infisical/infisical-dagger (Python SDK, module engine-version v0.12.5) to Daggerverse (engine-version v0.13.0) I get this error:

[subtle plinth]

This seems interesting

https://runme.dev/blog/runme-for-dagger

[wintry prism]

cc @edgy cape

[sharp zealot]

I am curious to hear everyone's thoughts on this! https://github.com/dagger/dagger/issues/8452

[plucky ermine]

@Lev Lazinskiy is there a regression in

[latent trellis]

I noticed something strange: when I return a struct in Go with additional functions implemented on it, instead of just returning the struct name, Dagger now seems to call the following function specifically:

func (r *Run) Summary(ctx context.Context) (string, error) {
    // ...
}

I don't know if it's the function name or the signature, but if I remove it, nothing gets called, as I would expect.

Is this a new "feature"?

[sharp zealot]

Yeah we're trying to "auto-print" the contents of the object in the CLI

There's an ongoing debate on how to select what to print

right now we scan for functions which return a printable type (scalar) + have no required arguments

[edgy cape]

Notebook-Native CI/CD: Runme for Dagger ...

[subtle plinth]

OK. I've taken this further and now have a module that provides arbitrary config and SOPs secrets decrypting from anywhere in the monorepo and it's working really well

The sops decrypting was a little difficult because of the GraphQL type system not allowing record types, but a list of Secrets works fine

[subtle plinth]

https://github.com/RawkodeAcademy/RawkodeAcademy/blob/main/daggerverse/sops/src/index.ts

[leaden pagoda]

question. a benefit of the sdk is to be able to use the ecosystem of that language sdk. if a pipeline can use dagger functions to get the job done and no or little use of dagger containers, files, directories etc are needed, then that'd also perfectly acceptable and a good use of dagger? I.e some node based tool. or 2: should logic be wrapped and embedded into dagger.Containers where possible? I'm assuming my first point makes more sense, but would like someone from dagger to share thoughts on that.

basic example woukd he install a node module into typescript sdk instead of using a dag.container to pull that tool and use it in a with exec. simply just use the node module code in a dag function.

[latent trellis]

I guess that's possible yes. But I'm not sure how much of that would be cached.

[subtle plinth]

If you don't use modules and create your own client, it would probably work quite well; as it'd be pretty easy to fetch files/dirs out of those containers and put them on your host.

[leaden pagoda]

so we need to be careful when we say language sdk gives you a whole ecosystem of your own language libraries to use. because how you solutionise will 100% impact dagger behavior

also sometimes caching is not important if you want portability. so it could be decided from weighing up pros and cons. would you agree?

[kind carbon]

That's definitely a use case we want to support, the only caveat right now that would make it much better would be to cache function calls by default. Right now functions themselves aren't cached, only the pipelines you run inside them. If you do more in the function without dag pipelines, that code won't be cached. I think we're very close to be able to move forward with implementing that feature though. Basic idea will be to cache by default but allow module authors to define a TTL for its cache so it's not all or nothing, can be "for how long". Without this control we can't know if the function can't be cached (some shouldn't), so we can't cache any of them!

[leaden pagoda]

amazing. being able to say don't cache totally makes sense if a dagger function is mostly pure code. give that control to the function author. Nice to hear.

[reef cloak]

Hi all, I am looking for some pointers on how someone might use dagger to deploy a docker image into elastic container service. In particular, how you create/update the service, monitor service deployment and deal with things like rollbacks and canary deployments. From what i've read so far it seems like dagger really doesn't help with this out of the box? I could certainly hand write some modules but i am not sure what value dagger provides over just having a custom program do the deployment.

[leaden pagoda]

Is that custom program that will be doing the deployment made up of bash/yaml scripts and proprietary steps/stages of a CI Platform?

[reef cloak]

it's a custom go program i wrote. It deals with the annoying bits around doing updates to ECS task definition and then monitoring that the service successfully deployed etc

[leaden pagoda]

how is that program triggered and where does it run? how is it passed credentials etc.

[reef cloak]

gitlab pipeline triggers a job. Credentials are from an IAM role attached to the ec2 instance where the go program is running and credentials are thus automatically injected in through the AWS SDK

[leaden pagoda]

if that job is messy with lots of scripts and would be hard to debug/move/migrate to other CI Platforms, dagger would be really beneficial in those situations.

Dagger out of the box provides containerisation and the ability to run pipelines on the dagger engine, its a foundation and then building blocks go on top

you mentioned its a custom go project, that could be daggerised so that the build, tests, publish of your go image can all be executed and run in dagger, no need for a Dockerfile etc. And then you could run that on GitLab in 1 stage that's portable to other CI Platforms, since the solution is implemented in dagger, not CI Yaml/Scripts

[bleak nest]

In case an example helps, I wrote a module that does something similar for Google Cloud Run: https://github.com/vvaswani/daggerverse/tree/main/google-cloud-run

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/8473

[dense canyon]

/me waiting for LLMs to be able to answer this reliably 🙏

[sharp zealot]

I ran into problems dealing with container platform... Created a placeholder issue to track it. https://github.com/dagger/dagger/issues/8482

If you have specific examples of Dagger API not managing platform well, please share them in the comments!

[latent trellis]

Related https://github.com/dagger/dagger/issues/6675

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/8484

[crude gazelle]

Is there currently a way to view the inline documentation of the whole module?

The documentation shows how to write it, but not where it can be seen.

The PHP SDK supports function and argument descriptions already, but I want to add support for module descriptions as well. I just need somewhere to check that I've done it correctly

[spiral whale]

Some stupid question,

I have a monorepo with dagger module in ci directory with dagger.json on the root of the project. What is the side effect loading project directory using dag.CurrentModule().Source().Directory("..") instead of using params like

// Project source directory
//
// +defaultPath="/"
source *dagger.Directory,

Both of them working and only working slightly different for DaG. Just curiosity.

[deft rain]

for one, you get nicer defaults, and don't need to handle them all yourselves

another one is that the include/exclude filters applied by your sdk and by your dagger.json aren't applied for the defaultPath

so you can load files that aren't just part of your "module"

[wintry prism]

It can be viewed in the Daggerverse listing for the module, but several folks have asked for a dagger doc command or similar.
It would be great to have your needs reflected in the issue: https://github.com/dagger/dagger/issues/8334

[spiral whale]

loading directories

[wheat forge]

hi everyone, i have just cloned dagger repository in my local machine and encountering some issue like UndeclearedImportedName, BrokenImport, MissingFieldorMethod and i don't know how can i fix it. I’ve tried updating dependencies and cleaning the build cache, but the issues persist. If anyone has insights or suggestions on how to resolve this, I would really appreciate it. Thankyou

[sharp zealot]

hi everyone, i have just cloned dagger

[sharp zealot]

@thorn moat we're in a live demo with @plucky ermine , and struggling with a service healthcheck timeout situation

[sharp zealot]

How do service bindings influence caching? For example:

func initDB(db *dagger.Service, sqlFile *dagger.File) *dagger.Service {
  dag.Container().
    From("client-image").
    WithServiceBinding("db", db).
    WithFile("data.sql", sqlFile).
    WithExec([]string{"sql", "blablabla", "data.sql"}).
    Sync(ctx)
  return db
}

This feels "wrong" but I don't know how to implement this "initialize db" pattern

[spiral whale]

I'm getting this error time to time while trying to load source directory. The repo I'm testing quite a big monorepo

failed to set call inputs: failed to load contextual arg "source": failed to load contextual directory "/": failed to import local module src: rpc error: code = Internal desc = received 4294967289-bytes data exceeding the limit 131128 bytes

[deft rain]

🤔 huh what

how big is the source directory? and out of curiosity, can you share what your include/exclude patterns for it are?

[spiral whale]

contextual directory data exceeding error on CI

[proven tusk]

It just became clear to me that to update an SDK we need to run dagger develop. I was looking for something like dagger upgrade since I am used to brew and they use brew upgrade.

[plucky ermine]

I am trying to use the defaultpath argument inside of a constructor, is this not supported or am I doing it incorrectly?

Code:

@object()
class Medplum {
  source: Directory

  constructor(@argument({ defaultPath: "." }) source) {
    this.source = source
  }

  /**
   * Return base container image
   */
  @func()
  base(): Container {
    return dag.container().withDirectory("/src", this.source)
  }
}

Error:

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "medplum" to get functions: call constructor: process "tsx --no-deprecation --tsconfig /src/.dagger/tsconfig.json /src/.dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

Stderr:
/src/.dagger/sdk/introspector/scanner/abtractions/typeToTypedef.ts:108
      throw new Error(`Unsupported type ${strType}`)

🥹 - I wish there was a combined changelog - as someone who often writes in every SDK I keep missing breaking changes because I only pay attention to the main dagger/dagger changelog

I am going to change my own behavior to pay attention to everything but it would be nice to be able to see everything (esp breaking) all in one place

[latent trellis]

Reported a rather weird bug: https://github.com/dagger/dagger/issues/8515

A quick search didn't lead to any similar issues.

[latent trellis]

Is .dagger the new convention for a contextual module?

[sharp zealot]

Yes

We haven't actually shipped the default change, but that's coming

[latent trellis]

👍

[latent trellis]

I noticed daggerverse is behind on some of my modules, even though I run the same publish.

For example: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/helm

The latest is actually 0.13.0

Hah, I figured out why you settled on .dagger.

It's high up in the alphabet 😛

Free marketing

[latent trellis]

Looks like it hasn't been working for a while: I found modules with updates from June, but I know I released at least two versions since

[restive shore]

but you won't see it when you list in terminal unless you do ls -a. So there is a downside too.

[latent trellis]

It's on GitHub

[sharp zealot]

@latent trellis that's now why, I promise 😛

(but it doesn't hurt)

[plucky ermine]

This is new I think, love the upgrade info though!

A new release of dagger is available: v0.13.1 → v0.13.2
To upgrade, see https://docs.dagger.io/install

[sharp zealot]

yeah it comes with a little surprise too 😅

[plucky ermine]

Not sure if I like the sound of that 😄

[plucky ermine]

I am still seeing builds failing wit this error 😦

Error: Encountered an unknown error while requesting data via graphql

Does anyone have any idea on what is happening here? This is Typescript SDK on 0.13.2

cc @wintry prism did you get anywhere in your own investigation?

[sharp zealot]

The surprise was that it keeps nagging you even when you've upgraded. We're shipping 0.13.3 to fix that

[plucky ermine]

Yeah I just saw that haha

[subtle plinth]

Next rebrand: A1-DAGS

[wheat river]

Can @sharp zealot or anyone expand on this? — .dagger as a new convention for contextual modules. I'm planning to update my code-gen tool and my module template (which accelerate a lot my development of new modules), so I'd like to be prepared so to speak 🙂

[sharp zealot]

This shouldn't affect any tooling at all, full context: https://github.com/dagger/dagger/issues/8123

[subtle plinth]

I've ran into a problem with this approach. If I have a "central" config module for my monorepo which exposes 10-20-50-whatever config and secrets; if ANY of these change, all downstream services bust their cache.

Is there a trick I'm missing that would allow such a service to exist where it's outputs are the inputs to downstream services rather than inputs?

I'm thinking I may need to build my own sops and store the secrets as hardcoded values within the module, so there's no dependency on the disk and each property is exposed by its own function ... which will work, but isn't ideal

It could be that the solution is we look at Provider modules that can hook in before the real DAG work starts, to provide config, secrets, and what nots?

Thinking aloud

^^ This would be a good way to handle env propagation too, without needing to contextualze them like directories too.

[plucky ermine]

@sharp zealot I recall the other day you were doing some amazing stuff by calling our dagger/dagger module to get a container with latest dagger inside of it that you can then use to call dagger and so on.

am I hallucinating this?

If not, do you have a one liner handy? 👼

ah I found it I think

dagger -m https://github.com/dagger/dagger call dev terminal

I also discovered this by running dagger -m https://github.com/dagger/dagger functions and RTFM'd

Dagger for the win! 😁

[sharp zealot]

yup

dagger + context dir ftw

[plucky ermine]

I really feel like I am in devops heaven right now.

I am trying to test out this fix to see how it works in a repo where I was seeing some intermittent graphQL errors, and I get get into an environment to debug this with this single command

dagger -m https://github.com/dagger/dagger call dev with-mounted-directory --path "medplum" --source "https://github.com/levlaz/medplum#daggerize" terminal

Its truly magical 😍

Specifically, its a bit painful to have multiple versions of dagger locally, so this makes all of that pain go away!

[restive shore]

What is the best way to exclude/include directories for non-contextual modules? Views?

[dense canyon]

yes, until we can find a better design for contextDirectory and remote modules 😬

[restive shore]

Is there a secret document for views? 🙂 I couldn't find it in docs

[dense canyon]

No there's not beause we're still under the debate if we should keep then or not. Ideally we'd like contextDirectory to also work for remote modules and deprecate views altogether 🙏

[restive shore]

aw. I am in a bit of a bind then lol. Since I don't have the ability to use private git modules (with PAT) I am having to clone my "daggerverse" on every build onto the local workspace. But when I pass directory into my module functions, I don't want the engine to load my daggerverse also.

[dense canyon]

how about passing the directory with those exclusions? i.e:

    dag.Directory().WithDirectory(".", srcDir, dagger.DirectoryWithDirectoryOpts{
        // Include and excludes here
    })

[restive shore]

That would still load the whole thing into the engine before filtering right?

[dense canyon]

yes, that's correct

views is your only way to filter before sending to the engine

[wheat river]

I think there'll be a I/O overhead since it's equivalent to a COPY operation if I'm not mistaken. I usually prefer mounting instead of copying.
Nevertheless, talking about context directories, I have a doubt about the ignore patterns (perhaps @dense canyon *cuando tengas un momento? * 🧉)

1. Can ignore patterns act upon a directory argument that doesn't have a default path set, so it can filter the directory content that it gets?
2. I was not clear just by reading the documentation; when the default path is . is it equivalent to the current directory of the Dagger module which's also equivalent to the directory where the dagger.json resides?

[dense canyon]

well.. now that I think about, if the references are not being unlazied that might not be entirely true. Let me do a quick test Nipuna

nope, it doesn't do it @restive shore.

i.e:

// Returns a container that echoes whatever string argument is provided
func (m *Lala) Test(ctx context.Context, d *dagger.Directory) (string, error) {
    return dag.Container().From("alpine").WithDirectory("/dir", d, dagger.ContainerWithDirectoryOpts{
        Exclude: []string{"some_large_file"},
    }).WithExec([]string{"ls", "-la", "/dir"}).Stdout(ctx)

}

since some_large_file was excluded before the dag was resolved, the engine didn't need to upload it

[restive shore]

Interesting! Thank you for testing it! Let me try that

[subtle plinth]

😮

That's awesome

[dense canyon]

🙏

[dense canyon]

hmm wait. still verifying.. might have misinterpreted the output 🙈

[dense canyon]

no.. it's still being updated, sorry. Fake news. 😬 .

[dense canyon]

@restive shore can't you use contextDirectory to filter your daggerverse modules?

ie: if you clone your daggerverse modules to a folder call daggerverse. This would be the equivalent:

func (m *Lala) Test(
    ctx context.Context,
    // +defaultPath="/"
    // +ignore=["some_large_file"]
    d *dagger.Directory) (string, error) {
    return dag.Container().From("alpine").WithDirectory("/dir", d).WithExec([]string{"ls", "-la", "/dir"}).Stdout(ctx)

}

and then replace some_large_file with your daggerverse folder

[restive shore]

I may be able to. Let me try that (in meetings right now 😢 )

[subtle plinth]

You'd impressed me as well. Now I'm disappointed

[restive shore]

hmm I can't get this to work. Does the +ignore still work if +defaultPath is overridden?

[dense canyon]

yes, if a directory is passed via the CLI, the pragma gets ignored. You need to use views in that case 🙏

[restive shore]

Do you have an example usage of views?

[dense canyon]

sure

[dense canyon]

{
  "name": "lala",
  "sdk": "go",
  "source": ".dagger",
  "engineVersion": "v0.13.3",
  "views": [
    {
      "name": "default",
      "patterns": [
        "!some_large_file"
      ]
    }
  ]
}

[thorn moat]

this'll be fixed in the next release thanks to @upbeat herald! https://github.com/dagger/dagger/pull/8436

[restive shore]

Oh nice! For now, views seem to have worked

[sharp zealot]

If I override the default path, but my explicit value is still a host directory, will the ignores be applied pre-upload or post-upload?

[shadow mauve]

You guys are no fun I just use dagger so that instead of littering yaml all over the place I can litter golang (which litters yaml) all over the place so that my team will learn golang by osmosis... 😅

[sharp zealot]

Haha, well we're only changing the default, you are free to dagger init --sdk=go --source=. and the Go SDK will add its files directly in the current directory. Then it's up to you to manage co-existence. Totally doable. We just don't want to do it by default.

You can also use the right layout for each module, if you have several modules in the same repo.

For example in the dagger repo:

• github.com/dagger/dagger is a Dagger module with separated source (.dagger) to avoid polluting the existing contents of the repo (https://github.com/dagger/dagger/blob/main/dagger.json#L58)
• github.com/dagger/dagger/version is our logic for assembling version strings. It's implemented as a Dagger module, there is no other code to co-exist with. So the module source is right there in the module root. (https://github.com/dagger/dagger/blob/main/version/dagger.json#L10)

[sharp zealot]

A neat real-use case of dagger for developing dagger:

• The problem: dagger call -m github.com/dagger/dagger --help now fails with an error about ".git: no such file or directory". This is a new issue caused by 1) change to our own dagger module, combined with 2) dagger -m stripping .git information.

• I think I remember that this has been fixed in main. But how to make sure?

• I find a recent commit that doesn't have the problem, and run a dev environment from that version: dagger call -m github.com/dagger/dagger@bda4ea1b7104e0bf2c4ec1a34cbc8ecc6c856fe8 dev terminal

• In that terminal, I try the same command: dagger call -m github.com/dagger/dagger --help. It works! No more error. I just confirmed that the error is indeed fixed in main.

[dense canyon]

This is the perfect example of the intersection between CI and Dev environments and how they complement each other. Planning to hype this pattern up in my Kubecon US talk in November

[shadow mauve]

I was thinking about doing something similar actually and writing a submodule or something to deal with versioning. I do something roughly analog (different string layouts for tags, but similar umbrella/component versioning scheme) and right now I need a better interface because I have a few hacky conditions for certain types of builds. The version module in dagger is much more thorough in dealing with various build conditions and that's something I definitely want...

[restive shore]

I am seeing linter issues with some modules. Are the replace directives required? It's possible that this linter - https://github.com/ldez/gomoddirectives is being too opinionated. But I wanted to get other's thoughts on whether this should be ignored or if it's something that can be remediated by Dagger.

[deft rain]

Yeah this is your linter being wayyyy too aggressive

Sadly we do need these directives

This is because we wanna pin to a specific OTEL log API version, since they keep changing it - automatic upgrades make it break in really difficult ways

[restive shore]

Totally understood and I agree. I'll tell the linter to not complain about these. Thanks for confirming!

[restive shore]

Re: Enums. From what I'm seeing, they always have to be optional because an empty default value doesn't work (is that accurate?). This makes it a bit cumbersome when using it in other modules and I think the documentation needs to be clearer on these points with an example of how it's used from other modules. It took me a fair bit of trial and error to get it right.

[wheat river]

I'm not sure if it's the same case as this issue (https://discord.com/channels/707636530424053791/1286816240043819123), but I'm facing randoms 137 (SIGKILL) but for an unknown reason. It's started to occur when I'm upgrading my (dagger) module template to 0.13.3.

Error: response from query: Post "http://dagger/query": command [docker exec -i dagger-engine-v0.13.3 buildctl dial-stdio] has exited with exit status 137, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=
error: Recipe `test` failed on line 136 with exit code 1

I've double-checked my container resources, and I don't have problems with limited resources in my OCI (Docker) configuration. Something else is killing the engine's process somehow. Ref: https://github.com/Excoriate/daggerverse/blob/a9b2a3f95a46a34da59e49286f345f618d2b48e7/module-template/tests/main.go#L63 —  Does it sounds familiar@plucky ermine ?

[plucky ermine]

Yeah it does, are you able to grab and share the engine logs next time this happens?

https://docs.dagger.io/manuals/developer/debugging/#access-the-dagger-engine-logs

[dense canyon]

Not necessarily since you could set a default without optional

[ashen spindle]

Quick Q: If I have a dagger directory which contains code checked out from a git repository, is there a way to turn that into a Dagger git object and then call git commands on it?

[restive shore]

I didn't know this was possible! Let me test that out.

[restive shore]

This didn't make it non-optional.

[restive shore]

I don't think this is possible today. The amount of commands you can run with the dag.Git() object is limited anyway (no write ops). What I do is use a small image with git to run git operations. Something like https://hub.docker.com/r/alpine/git works well.

[ashen spindle]

My intended use was git describe --tags to generate an container image tag; currently using a git container but would be nicer to do it using Dagger primitives

[kind carbon]

Enum default value

[dense canyon]

I see. I didn't understand that you were expecting it to be non-optional. If you need it to be non-optional, the only way to achieve this is by not setting a default value as setting a default value implies that the argument will be necessarily optional

[latent trellis]

Has anyone experimented with generating SLSA provenance in Dagger?

[wheat river]

An architectural thought: recently, I've been rewriting a module, and I'm a bit confused about the context directories and how they combine with the +optional pragmas when setting a default value for a *dagger.Directory parameter. Let's say I have an argument in a Dagger function:

    // source is the source directory that includes the source code.
    source *dagger.Directory,

In this particular use case, it's okay to make it optional; if so, I can handle it in the function's body, and if it's nil, then I can set it to a certain value. However, with context directories, I wonder whether a +defaultPath= with a certain value combined with +optional would be a better approach.

What have been your recent experiences designing your function interfaces with either +optional pragmas or the recent context directories features?

[sharp zealot]

In general, an argument with a default value implies that it's optional. For technical reasons, it's safer to keep the +optional explicitly even if you have a +default or +defaultPath. But conceptually, it is redundant.

I like to add a default path to my directory arguments whenever it makes sense. Usually for "contextual modules" that are attached to a host project on the same repo. For standalone modules, usually it makes less sense.

I hope this helps!

[deft rain]

not yet, but i really want to - ideally i'd love to produce it from the dag automagically, but i suppose there are also other approaches?

happy to chat about it more in detail sometime for sure 😄

[wheat river]

Thanks @sharp zealot , that makes sense. As a follow-up question, I've found that when a value is passed to a *dagger.Directory argument; which as a +defaultPath="." pragma set, and also an +ignore=".somecachefolder" I'd expect that even if the value is passed, the filters applies and that folder is ignored. As per my recent tests, it's not taken into consideration when the value is passed. Was that intended when Context Directories were released?

[sharp zealot]

I think that's fixed in the latest release. What version are you running?

(it was a known issue in past versions)

[wheat river]

0.13.3 — seems to be the last one.

[restive shore]

This? https://github.com/dagger/dagger/pull/8436. I don't think it's released yet.

[spiral whale]

Hey is there any specific reason why we can only use env: prefix for secrets? dagger call foo --token env:GITLAB_TOKEN --id $CI_MERGE_REQUEST_IID this syntax feels weird. I was expecting to pass value to string var with env: prefix as well

[sharp zealot]

heads up we are releasing 0.13.4 today and I think it will include your fix

[latent trellis]

0.13.4:

[plucky ermine]

Hey! Can you let us know which command triggered this?

[latent trellis]

dagger develop after upgrading to 0.13.4

here: https://github.com/openmeterio/openmeter

[sharp zealot]

I'm thinking that we should perhaps merge this channel into #general .

Rationale:

• This channel started as the Project Zenith (modules) testers channel

• Then it remained as the place for programming Dagger (writing modules) as opposed to simply "using Dagger" without writing code.

• In practice that distinction has turned out to be artificial. You can't really use Dagger without understanding its API and programming model.

• Meanwhile we have too many discord channels

TLDR: how do you feel about archiving this channel and just talking in #general instead?

[deft rain]

investigating

[warped canyon]

There's a lot of overlap today for sure. I'm not against it

[latent trellis]

general feels like a "hello, I'm new to dagger" or "hey, here is my awesome dagger stuff" channel

this one is more like a hardcore dagger (internals) channel

I suspect there would be a lot of noise. But I rarely use general at all. 🙂

[plucky ermine]

What if we rename general developers-developers-developers-developers? 😄

I am for merging, with Discord less is more

[latent trellis]

My ignore directive is getting loooooooong:

    // +defaultPath="/"
    // +ignore=[".devenv", ".direnv", ".github", ".vscode", "api/client/node/dist", "api/client/node/node_modules", "api/client/web/dist", "api/client/web/node_modules", "api/spec/node_modules", "api/spec/output", "tmp", "go.work", "go.work.sum"]

[sharp zealot]

But we already have #intros .

personally I find myself often hesitating on which channel to post

[latent trellis]

We are about to publish a Dagger+Helm post. I would certainly post it in general, not here. News about Dagger is also something I'd expect to be there. But it's nice to have a more "focused" place for development related chat.

my 2 cents, I can live with merging them

[deft rain]

i can repro this now, but run out of time for today - will keep this going tomorrow