daggernauts

[restive shore]

I am trying out constructors and I can't get this to work. What am I doing wrong?

package main

import "fmt"

type Java struct {
    jdkVersion   string
    mavenVersion string
}

func New(
    // JDK version to download
    // +optional
    jdkVersion Optional[string],

    // Maven version to download
    // +optional
    mavenVersion Optional[string],
) *Java {
    return &Java{
        jdkVersion:   jdkVersion.GetOr("17"),
        mavenVersion: mavenVersion.GetOr("3.9.6"),
    }
}

func (j *Java) Container() *Container {
    fmt.Println("maven version", j.mavenVersion)
    fmt.Println("jdk version", j.jdkVersion)

    return dag.Container().From(fmt.Sprintf("maven:%s-amazoncorretto-%s", j.mavenVersion, j.jdkVersion))
}

Both mavenVersion and jdkVersion are empty.

dagger engine 0.9.7

[unreal moat]

looking through https://docs.dagger.io/zenith/ I'm having a hard time finding an example of how to import and use modules inside a pipeline...is this supported?

[plucky ermine]

The main question I would have is when you say pipeline, do you mean a traditional "classic" dagger pipeline?

If so, unfortunately that is not supported right now but we are working on it.

The "zenith way" of doing things right now is to have a "main module" that defines your pipeline, this then can call other modules.

https://github.com/dagger/dagger/issues/5993

[plucky ermine]

I have done some stuff with exec.Command() that works ok, but its not ideal, depending on your use case that might be an approach you could take in the meantime. A very simple example here:

package main

import (
    "log"
    "os"
    "os/exec"
)

func main() {
    cmd := exec.Command("dagger", "call", "-m", "github.com/shykes/daggerverse/hello", "message")
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    err := cmd.Run()

    if err != nil {
        log.Fatal(err)
    }
}

[unreal moat]

The "zenith way" of doing things right now is to have a "main module" that defines your pipeline, this then can call other modules.
is this available in the typescript sdk?

[strong ingot]

@upbeat herald?

[kind carbon]

Yes, works in all SDKs.

[dense canyon]

👋 is there anything I particularly need to setup locally when working with python modules for my code editor to pick the generated packages? I initialized a module with python and when I open main.py in my neovim editor which has pyright set up I get dagger could not be resolved. This doesn't happen when I use Dagger via the SDK and virtual environments. Anything I need to configure here @kind carbon ?

[unreal moat]

I also had this question, re: Typescript. Should I expect intellisense to work on the dag namespace after installing a module?

[unreal moat]

ok. the Advanced Programming section is missing from the Typescript SDK documentation.

[sharp zealot]

I'm trying to grok the consequences of having merged interface support. Does it mean I can for example dagger up my own custom type, as long as it implements a Service interface? And if so, how do I find out what the interface is, and whether I implement it correctly?

[rocky harbor]

load package ".": package name is empty

[wheat river]

Hey guys! more or less following this discussion (#daggernauts message) — Is there any way to currently use the Dagger 'old fashion' way of creating pipelines importing 'zenith' modules? And, extending a bit further my question, what's the recommended approach when there's a main 'zenith' module —let's call it, a parent pipeline— and several child 'zenith' modules that live in different repositories? I've seen that I can do a bit of exec.Command magic there, but it does not feel "pleasant" I'd say.

[latent trellis]

Do you have an ETA for updating the Dagger version under Daggerverse?

Modules using the new Name function on the File type do not show up at the moment: https://daggerverse.dev/crawl/github.com/sagikazarmark/daggerverse/archiver@71d46691258f1b807806068680a596603294fa44

[strong ingot]

@void widget @thorn moat @wintry prism are actively working in that area and are in the best position to answer.

[wintry prism]

Thanks for the nudge @latent trellis, I'm looking into it now.

[sharp zealot]

@strong ingot perhaps we should make daggerverse update (or at least daggerverse tests) a gate to engine release?

[sharp zealot]

@rocky harbor sorry for the wall of text in https://github.com/dagger/dagger/issues/6291 . It did help me a lot to clarify my thoughts. Basically my position is "no awkward middle ground": we pick one side and stick to it. Draw bright clear lines so that users aren't confused.

[rocky harbor]

No worries, it'd be incredibly hypocritical of me to not appreciate a good wall of text 😄 Getting back ASAP, need to think about a few parts of it

[upbeat herald]

@rocky harbor Hey 😄
Do you have a PR or some resources about the reference Go module?
I'm starting the Node one inside the dagger repository (as a dependency of the typescript dev module) and I wonder if you wrote some standard stuff or anything 🚀

[bleak nest]

There's an example here that you might find useful https://github.com/dagger/dagger/pull/6262

[strong ingot]

That is a good idea. I'm thinking more along the lines of: one of the first to run the newly released version and make sure that everything behaves as expected. WDYT @thorn moat ?

[thorn moat]

That makes sense to me. Would be nice if we could automate the engine bump in Daggerverse, but we need integration tests for publishing modules before I could trust that to auto-deploy

[strong ingot]

Yes! Also: #infrastructure message

[wintry prism]

We're working on fleshing out https://github.com/dagger/dagger/issues/6470

[thorn moat]

@kind carbon @rocky harbor re: dagger call / chaining - are we planning to retire the automatic File.contents call when you do dagger call build where build returns a *File? Seems this is likely to cause issues because returning e.g. a compiled binary is really common, and if you call that in CI you'll get a file size X exceeds limit Y error when really I wouldn't have wanted it to spit out the file contents in the first place. cc @latent trellis

[warped canyon]

IMO each type should have a default action if none is specified from the CLI

[thorn moat]

yeah, agree; I just don't think spitting out the contents is a sane default

showing name + size could make sense? same for Container.stdout really, a summary of the container's OCI config instead could be cool

[rocky harbor]

Yeah we are changing the defaults to be more sane: https://github.com/dagger/dagger/issues/6359

[thorn moat]

sweet, ty 🙏

[latent trellis]

https://github.com/dagger/dagger/issues/6256

and this 🙏

[warped canyon]

I like it

[latent trellis]

I'm doing things like dagger call build all when I have additional functions on a build struct. dagger call build would be nicer.

[latent trellis]

@rocky harbor I can't seem to make codegen work for interfaces.

Repro:

1. dagger mod init --name iface --sdk go
2. dagger mod install github.com/sagikazarmark/daggerverse/archivist
3. Code:

package main

import (
    "context"
)

// Archiver archives a directory of files and returns a single archive.
type Archiver interface {
    DaggerObject

    Archive(ctx context.Context, name string, source *Directory) *File
}

type Iface struct{}

func (m *Iface) Test() *File {
    dir := dag.Directory()

    return dag.Archivist().Tar().AsArchiver().Archive(context.Background(), "test", dir)
}

dag.Archivist() is a module, Tar() returns a struct. I tried all variations with/without the context parameter and a second error return value, but there is no AsArchiver method. I'm on Dagger 0.9.7.

Any ideas?

[rocky harbor]

@Erik Sipsma I can't seem to make

[latent trellis]

I see some intermittent failures ever since I upgraded to 0.9.7:

Might be related: https://github.com/vito/progrock/issues/12

[latent trellis]

Something weird is going on with Daggerverse:

I have a helm module here: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/helm

If I search for Helm, it doesn't show up: https://daggerverse.dev/search?q=helm

Types are also weird.

[ocean escarp]

I've also seen these type problems, and it seems that the latest changes to enums are not yet supported by Daggerverse.

[latent trellis]

Are there any known issues around calling Sync or File on containers with bound services?

https://github.com/sagikazarmark/daggerverse/pull/42

I get lookoup errors and failing commands (with weird exit codes) for no apparent reason:

  ┃ host alias: lookup ceg6723ji0dva on 10.87.0.1:53: no such host                                                                                                                                                             
  ┃ lookup ceg6723ji0dva.j64at12lurf3i.dagger.local on 10.87.0.1:53: no such host                                                                                                                                              
  ┃ lookup ceg6723ji0dva.kqmk1t6ik09d6.dagger.local on 10.87.0.1:53: no such host                                                                                                                                              
  ┃ lookup ceg6723ji0dva.9s2cfb2o7h0bi.dagger.local on 10.87.0.1:53: no such host 

I've been debugging this for hours and it's driving me crazy.

[wintry prism]

Concurrent calls to WriteStatus and Clos...

helm :: Daggerverse

I've also seen these type problems, and

[wintry prism]

Chainable API for working with Helm char...

[plucky ermine]

In honor of OpenTofu going GA today https://daggerverse.dev/mod/github.com/levlaz/daggerverse/opentofu

🙂

[plucky ermine]

Not that it helps you at this moment,

[latent trellis]

I'll probably replace using golang.org/x/sync/errgroup with github.com/sourcegraph/conc

It has more options for context and error handling (cancel on first error or not, wait for all errors or not, etc)

errgroup always cancels the context on the first error and only returns the first error.

[thorn moat]

haha I just adopted conc recently for the same reason. I find any time I use errgroup + especially errgroup.WithContext i'm surprised by its behavior 🤷‍♂️

[latent trellis]

I actually didn't notice it at first. 😄 I guess it makes sense to have a default behavior (even though I don't agree with the choice), but not being able to change its behavior is kinda annoying.

[plucky ermine]

Does anyone have an example of using the new Constructor method that was introduced in 0.9.4 in a module?

[wintry prism]

Does anyone have an example of using the

[olive field]

Hi, looking some daggerverse modules code. The bad thing is relatead on module reusability. Supposing we have a module to reuse, i need to look the module codebase and i have to distill some constraints on my own like distro choice etc. I suggest to introduce some abstraction like in functional programming to enforce some type checking. For example ```
type Alpine struct{}
type Debian struct{}

type Dialect interface {
Alpine | Debian
}

type Container[D Dialect] struct {
container *dagger.Container
dialect D
}

A module must be able to install required tools even if i need to propagate/pass a base container as argument. I agree to mantain an untyped raw dsl but for modules i would like more checks.

[latent trellis]

Hi, looking some daggerverse modules

Which API do you like better?

chartDir := dag.Host().Directory("charts/some-chart")

dag.Helm().Lint(ctx, chartDir)

// returns a *File
pkg := dag.Helm().Package()

// closer to the Helm CLI API
dag.Helm().Login("ghcr.io", "username", "password").Push(ctx, "ghcr.io/org/helm-charts", pkg)

OR

// returns a custom type: Chart
chart := dag.Helm().Chart(dag.Host().Directory("charts/some-chart"))

chart.Lint()

// returns a custom type: Package
pkg := chart.Package()

// resembles the container publish API
pkg.WithRegistryAuth("ghcr.io", "username", "password").Publish(ctx, "ghcr.io/org/helm-charts")

cc @wintry prism @warped canyon

(The second one is blocked on #1199329512203235349 )

[warped canyon]

Which API do you like better?

[last fox]

Is there already a module for reading in an existing CI workflow like GitHub Actions?

[sharp zealot]

There's project GALE that is basically a port of the GHA runtime to Dagger. It has a lot of the primitives running, although it's hard to tell with those types of projects how close you really are to 100%

See #github

[last fox]

Yeah I’m familiar with Gale. But I think I’m thinking of something slightly different but i need to articulate it a bit better.

[sharp zealot]

Ah, I see.

[unreal moat]

is this a known issue when upgrading version from 0.9.5 -> 0.9.7? ✘ load call ERROR [6.97s] ├ [6.97s] loading module ┃ Error: failed to get loaded module ID: input: resolve: host: directory: asModule: failed to call sdk module codegen: call function "Codege ┃ : process "/runtime" did not complete successfully: exit code: 2 ┃ input: resolve: host: directory: asModule: failed to call sdk module codegen: call function "Codegen": process "/runtime" did not complete ┃ uccessfully: exit code: 2 ┃ ┃ Stdout: ┃ json: error calling MarshalJSON for type *main.GeneratedCode: input: resolve: container: from: withMountedCache: withoutEntrypoint: withDi ┃ ctory: withFile: withDirectory: withMountedDirectory: withWorkdir: withNewFile: withExec: withDirectory: withExec: withExec: withExec: dir ┃ tory: process "sh -c if [ -f package.json ]; then npm install --package-lock-only ./sdk --dev && tsx /opt/runtime/bin/__tsconfig.updato ┃ ts; else cp -r /opt/runtime/template/*.json .; fi" did not complete successfully: exit code: 1 ┃ input: resolve: container: from: withMountedCache: withoutEntrypoint: withDirectory: withFile: withDirectory: withMountedDirectory: withWo ┃ dir: withNewFile: withExec: withDirectory: withExec: withExec: withExec: directory: process "sh -c if [ -f package.json ]; then npm insta ┃ --package-lock-only ./sdk --dev && tsx /opt/runtime/bin/__tsconfig.updator.ts; else cp -r /opt/runtime/template/*.json .; fi" did not c ┃ plete successfully: exit code: 1

(with typescript sdk)

[warped canyon]

bumping up this issue with Services + Sync for visibility: #1199329512203235349 message

[unreal moat]

apologies if I keep asking questions in the wrong place, currently evaluating zenith to make a case for a POC at my company. I'm noticing it's mostly dagger devs in here so lmk if it's better to post in other channels.

I'm noticing that I can't catch container exec errors in a module function the same way it works in 'legacy' pipelines.

Running this: ```import { connect, Client, Container } from "@dagger.io/dagger";

connect(async (client: Client) => {
try {
await testIt(client);
} catch (e) {
// Otherwise, show the full stack trace for debugging.
console.error("an error ocurred");

// Abort script with non-zero exit code.
const output = (await failureFn(client)).stdout();

}
});

async function testIt(client: Client) {
await client
.container()
.from("alpine")
.withEnvVariable("CACHEBUSTER", Date.now().toString())
// .withNewFile("doTest", { contents: SCRIPT, permissions: 0o750 })
.withExec(["sh", "-c", "exit 1"])
.sync();
}

async function failureFn(client: Client): Promise<Container> {
return client
.container()
.from("alpine")
.withEnvVariable("CACHEBUSTER", Date.now().toString())
.withExec(["sh", "-c", "echo 'hiya buddy'"])
.sync();
}```

works as expected, results in ❯ dagger run node --loader ts-node/esm ./src/test.mts ┣─╮ │ ▽ init │ █ [1.18s] connect │ ┣ [0.13s] starting engine │ ┣ [0.25s] starting session │ ┃ OK! │ ┻ █ [1.95s] node --loader ts-node/esm ./src/test.mts ┃ (node:76189) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time ┃ (Use `node --trace-warnings ...` to show where the warning was created) ┃ an error ocurred ┣─╮ │ ▼ from │ ┣─╮ │ │ ▽ from alpine │ │ █ [0.73s] resolve image config for docker.io/library/alpine:latest ┣─┼─┼─╮ │ │ │ ▼ sync │ │ █ │ [0.01s] pull docker.io/library/alpine:latest │ │ ┣ │ [0.01s] resolve docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 │ │ ┣─┼─╮ pull docker.io/library/alpine:latest │ ┻ ┻ │ │ │ █◀┤ [0.06s] exec sh -c echo 'hiya buddy' │ ┃ │ hiya buddy █ │ │ [0.13s] ERROR sync │ █◀╯ [0.11s] ERROR exec sh -c exit 1 ┻ ┻ • Engine: e25b554b244d (version v0.9.7) ⧗ 3.15s ✔ 18 ✘ 2

but running the equivalent in a module ```import { dag, Container, object, func } from "@dagger.io/dagger";

@object
// eslint-disable-next-line @typescript-eslint/no-unused-vars
class Test {
/**

• example usage: "dagger call test"

*/
@func
async test() {
try {
await this.testIt();
} catch (e) {
// Otherwise, show the full stack trace for debugging.
console.error("an error ocurred");

  // Abort script with non-zero exit code.
  const output = (await this.failureFn()).stdout();
}

}

async failureFn(): Promise<Container> {
return dag
.container()
.from("alpine")
.withEnvVariable("CACHEBUSTER", Date.now().toString())
.withExec(["sh", "-c", "echo 'hiya buddy'"])
.sync();
}
async testIt() {
await dag
.container()
.from("alpine")
.withEnvVariable("CACHEBUSTER", Date.now().toString())
.withExec(["sh", "-c", "exit 1"])
.sync();
}
}

results in ❯ dagger call test ✔ dagger call test [2.53s] ┃ <nil> ✘ sync ERROR [0.10s] ▶ sync ✘ exec sh -c exit 1 ERROR [0.09s] • Engine: e25b554b244d (version v0.9.7) ⧗ 12.83s ✔ 180 ∅ 33 ✘ 2

it also looks like the including the try catch is calling the module function recursively

[plucky ermine]

Its mostly dagger devs in mostly every channel 😎, thats a feature not a bug.

keep the questions coming! 🙂

I'd like to repro this myself and poke around a bit, is this code open somewhere?

[unreal moat]

no but I can make a gist

[unreal moat]

here you go https://github.com/bgroupe/dagger-zenith-test

[plucky ermine]

GitHub - bgroupe/dagger-zenith-test

[plucky ermine]

I am having a tough time understanding and explaining the difference between these two diagrams.

When I inspect running commands inside of my engine running classic dagger or modules it all kind of looks the same (a bunch of stuff executing around _shim)

Is there a succinct one-liner that helps explain the difference?

[sharp zealot]

two different views of basically the same thing. Zenith adds an extra layer of concepts (functions & modules) on top of the same core

the first diagram emphasizes the containerized aspect - "all your steps are containers"

the second emphasizes the reusable modules feature

we're still searching for the best way to explain Dagger in diagrams

personally I think we need several diagrams, each illustrating a core feature or concept

[plucky ermine]

Maybe I asked the wrong question then haha.

What is a simple way to describe how zenith execution is different.

For example how client.Host() is very different in classic dagger vs zenith.

[sharp zealot]

it's on my list somewhere 😁

The main difference is that in zenith, everything is a function call. All features old and new are reframed in terms of function calls

for example a core API query like { container { from { withExec { stdout } } } } is a chain of 4 function calls.

• The functions are builtin (not loaded from an external module) but they are still functions.
• Calling functions from external modules works the same way.
• The term for a chain of function calls is "pipeline".
• All functions run in containers under the hood. This concept of containerized functions are the next big primitive that the devops world can standardize on. We aspire for Dagger to be that standard.

[plucky ermine]

Thanks Solomon, all of that makes sense and is clear to me.

I am asking more about the execution environment I guess.

In classic dagger the code executes from your computer, so you can do things like os.Getenv (as shown here) https://docs.dagger.io/723462/use-secrets/#use-secrets-from-the-host-environment

In zenith I cant use os.Getenv because I guess the code is loaded somewhere else and then executed right?

[sharp zealot]

Yes see the last point on functions being containerized. You're right it's the biggest change in the programming model for transitioning to Zenith

Because your code no longer has ambient access to the client's host system

This is an area where the UX is still moving. We are looking for ways to match the convenience of ambient host access without compromising the core benefits of containerized functions: safety by default, portability, reproducibility.

[plucky ermine]

I think my brain is very close to unlocking whatever was blocking it 😄

So is the reason you no longer have ambient access to the host environment because we closed a gate that was previously opened, or is it because we moved the execution enviroment to another layer of abstraction?

[sharp zealot]

in practice this means moving existing dagger client code from a custom tool to a Dagger Function is not automated, but is still pretty easy.

[sharp zealot]

closed a gate

[plucky ermine]

Thank you

Its all clear now!

that helps explain this for me

When I inspect running commands inside of my engine running classic dagger or modules it all kind of looks the same (a bunch of stuff executing around _shim)

[sharp zealot]

like mac apps moving to app store which enforces a stronger sandbox for security and quality

or the difference between ios sandbox and much laxer android sandbox

[plucky ermine]

Hm, I just noticed that dagger export is no longer in the zenith docs (also checked its alias download) was this intentional? https://docs.dagger.io/zenith/reference/979596/cli/#commands

[sharp zealot]

Are you running a dev version, or release?

[plucky ermine]

I am talking about the docs on our public site

[sharp zealot]

Ah, then assuming zenith docs are deployed from main, then it might be related to the CLI changes that were recently merged by @rocky harbor .

They implement a design decision we planned a few weeks ago, to collapse all the different "verbs" (dagger shell, dagger up, dagger export) into just dagger call, and expose the functionality as core functions that you can chain in your pipeline.

For example:

• dagger shell foo becomes dagger call foo shell
• dagger export foo becomes dagger call foo export

etc

@rocky harbor @kind carbon can confirm

[rocky harbor]

FWICT export actually wasn't there in the first place: https://github.com/dagger/dagger/blob/e1a3764f9fcd676374a504e37eb7334e31e2c639/docs/versioned_docs/version-zenith/reference/979596-cli.mdx#L4. We're planning on updating all the docs once the changes are fully settled before the next release

[mossy hazel]

The return of the inline-python module (best module ever)

How to know your public IP with a very simple shell command 😄

dagger call -m github.com/samalba/inline-python-mod@main \
    with-package --name "requests" \
    code --code "import sys,requests as r; sys.stdout.write(r.get('https://api.ipify.org?format=json').json()['ip'])"

[mossy hazel]

(I promise, one day I'll make a useful one)

[sharp zealot]

Now with support for not always getting the same IP forever 😁

that alone is worth a 2.0 release

[kind carbon]

Python module dev environment

[sharp zealot]

@kind carbon 👋 coincidentally I'm also running into Python issues...

$ pip3 install -e ./sdk
[...]
ERROR: Package 'dagger-io' requires a different Python: 3.9.6 not in '>=3.10'

My bigger problem is that I'm failing to install a python dependency in my Python module...

$ pip3 install ollama
[...]
$ 

$ dagger functions
[...] 
┃ │ /sdk/src/dagger/mod/cli.py:44 in get_module                                  │        
┃ │                                                                              │        
┃ │   41 │   """Get the environment instance from the main module."""            │        
┃ │   42 │   # TODO: Allow configuring which package/module to use.              │        
┃ │   43 │   try:                                                                │        
┃ │ ❱ 44 │   │   py_module = import_module(module_name)                          │        
┃ │   45 │   except ModuleNotFoundError as e:                                    │        
┃ │   46 │   │   msg = (                                                         │        
┃ │   47 │   │   │   f'The "{module_name}" module could not be found. '          │        
[...]

[sharp zealot]

I got it to work, thank you @kind carbon !

Very basic version of a working ollama module 🙂 https://daggerverse.dev/mod/github.com/shykes/daggerverse/ollama

dagger call -m github.com/shykes/daggerverse/ollama llama2 --prompt 'hello AI, are you there?'

Warning: this will run the ollama server in a container, with no gpu access. So it's pretty slow

[wheat river]

Hey guys, I've done a couple of 'classic' dagger pipelines and now I'm exploring heavily the new 'zenith' approach. I believe that having 'containerized functions' is a game changer and it'd help a lot in solving multiple challenges in platform automation, DX and DevOps in general.
Well... to the point. I was checking this documentation https://github.com/dagger/dagger/blob/e1a3764f9fcd676374a504e37eb7334e31e2c639/docs/versioned_docs/version-zenith/developer/go/191108-advanced-programming.mdx#L127 and I'm not fully clear how we can handle errors in a zenith module; especially using constructors. If any of the arguments got an invalid value —passed from the caller module— what's the best approach to handle an error, or even return an error in the Go way?

[kind carbon]

Zenith errors in Go

[bleak nest]

Yes, afaik this didn't exist when I last updated the CLI reference. As you said, once the commands are frozen, I'll revise the CLI page.

[sharp zealot]

would it be worth investing ibuprofen a CLI reference doc generator? Maybe one exists already?

[bleak nest]

Yes, Helder suggested this as well some time ago. I didn't have time to look for one yet though. TBH we're only feeling it now because the CLI is in flux, once it settles we won't need to update the CLI doc so often because the CLI won't change so frequently (at least based on what I've seen over the last year or so)

[kind carbon]

Cobra has a util that can generate docs (md) for commands.

[sharp zealot]

That's true, we definitely are in a period of intense change. But the baseline will probably more frequent changes, since we are shifting to a much more CLI-centric UX. So there will naturally be more changes I believe.

[kind carbon]

@bleak nest, how would this integrate into docusaurus? We can easily add a frontmatter to each doc.

[sleek nest]

Welcome @ashen ember! @wintry prism, I've been talking with Miguel about building a module for https://chainloop.dev/ because I think there is a great integration story there, but they'd love to learn more about how to get started. @ashen ember These docs would be the best place to start. You can pick the QuickStart based on your preferred language . https://docs.dagger.io/zenith/developer

@wintry prism and @warped canyon can help you out with whatever you need!

@wintry prism , @ashen ember was looking for some best practice modules too, so some guidance on that would be great!

[plucky ermine]

I've mostly been writing in go but I have one module in python.

I just added it to a go module and inspected the generated code.

I know this was happening in the background but seeing the codegen actually work and show me my python module translated to go is really mindblowing 😀

[ashen ember]

👋🏼 Thanks @sleek nest

Hi @wintry prism and @warped canyon

We've just started skimming through the docs and brainstorming internally a first cut of the module functionality.

Once we know more or/and we get into any hiccups I'll let you know 🙂

[restive shore]

When loading a source directory from local disk (by setting root in dagger.json ) is it possible to list excludes and includes? Currently it loads the entire directory and it's slow.

On a related note, is there a reference for all dagger.json valid keys somewhere?

[plucky ermine]

When creating a Service is there a dagger way to set params like --ulimit memlock=-1 ? -- I cant seem to find any info about this anywhere.

[thorn moat]

haven't seen that before, how is it used? could it be done by a CLI wrapper?

[plucky ermine]

This is an argument to docker run specifically for this DB but I am sure there are other cases like this. https://www.dragonflydb.io/docs/getting-started/docker

[plucky ermine]

Hm, this may not be necessary because when I run ulimit inside a container it tells me its already unlimited.

[thorn moat]

yeah this may be a case where buildkit's default strays from docker's

[ocean escarp]

I think you should put the table borders in the README in daggerverse and adjust the padding

[hexed flume]

Yes we are currently polishing the UI. Thanks for your feedback 🙏

[wintry prism]

Hi all! 👋 Some of us are hanging out in #911305510882513037 for the Project Zenith Community Call. Join us to chat about Dagger functions, moudules, Daggerverse, etc 🙂

[hoary geyser]

Yes, this is the problem space that the C4 model addresses
https://en.wikipedia.org/wiki/C4_model
where you have a hierarchy of four types of sets

• Context
• Container
• Component
• Code
  (The term Container in C4 is not related to Docker containers.)

Sadly, the example docs tend to have the worst default layouts I have ever seen, luckily, the settings are easily modified.

diagramming strategies

[wheat river]

Hey guys, quick question -
In the 'old school' Dagger 😄 way, having more complex logic that requires exchanging data with an external API (E.g.: GitHub's API). The custom logic + the specific Dagger's logic (meaning calling the engine, creating a container, executing commands, etc.) is developed in the same 'context' let's say. In the Zenith way, what's the recommended approach? — I was thinking of having, for instance, a Dockerfile with an app/component that performs more complex logic (E.g.: calling an external API) and uses it from the Zenith module; hence in such a way, the Zenith module will remain more 'clean' while this app/component executes the domain logic. Or, perhaps avoiding such 'domain' specific logic within a module and leaving that to the 'caller' module instead (but sounds like it's going to be the same dilemma).

[sharp zealot]

Both are fine, you should do whatever is the most convenient to you and your module's users. For example if there's a lib that does exactly what you need, you can just import that lib in your Dagger Function's code. If it's a CLI tool, then you would wrap that in a container. If there's an existing Dockerfile and you want to share that logic with non-Dagger tooling, then wrap that. etc

It's perfectly fine to have a Dagger Function that encapsulates domain logic. You can use it as a generic encapsulation method for your entire domain actually, one function at a time 🙂

[wheat river]

Hey guys, I'm curious about a standard behaviour in Dagger for managing the outputs of commands (specifically stdout) — why does it return only the latest command's standard output?
I have a module that executes several commands passed, say for example

dagger call -m . --src=$(pwd)/../ run \
--cmds="cat .golangci.yml, ls -ltrah iac-terragrunt"

In the code, the commands are parsed and —following this example— they create two WithExec commands.

(Just a snippet, but works in explaining the case)

    for _, cmd := range cmdsToExec {
        tg.Ctr = tg.Ctr.WithExec(cmd)
    }

If I set the --focus=false flag it's going to show me the first and second command, but if I'm not setting it, it only shows the latest one. Why? And what if I'd like to return all the stdout(s) of all the commands executed? Is there any workaround?

[kind carbon]

Output from multiple commands

[upbeat herald]

Question of curiosity: is it possible with dagger to expose to the host multiples services in 1 dagger command?
Because now we use dagger call <my-service> up --native, but what if we return an array of service?
I'm curious 😮

[spiral whale]

 ✘ exec ghx ERROR [0.04s]
  ┃ host alias: lookup pqmamj10kfr52 on 10.87.0.1:53: no such host
  ┃ lookup pqmamj10kfr52.7vdcsprimo8cu.dagger.local on 10.87.0.1:53: no su
  ┃ ch host

did we change anything related networking or dagger-in-dagger setup with 0.9.7? I'm getting this error while I'm trying to update gale from v0.9.5 to v0.9.7

[subtle plinth]

Before I go scrolling through history and issues, does the following work in dagger.json yet?

"dependencies": [
    "../../dagger/supabase"
]

[sharp zealot]

Yes it works, but it requires configuring the root of your module. That method of configuring the root is about to change (break) in the next week. When that happens, it will be trivial to unbreak. Either way the dependencies field won't change. cc @rocky harbor

[subtle plinth]

By that you mean the source or root of my current module has to include the entire repo?

[sharp zealot]

• Today, for this import to work, you need to set root of your module to ../.. (To be confirmed by @rocky harbor ?)

• After the fix, for the same import to work, you need to create ../../dagger.json if it doesn't exist, and set "root-for": ["./your/module"]

In both cases, "dependencies" remains the same in your module.

[subtle plinth]

Ah, different use-case I think

I'll create a thread

[sharp zealot]

OK, I'm addressing strictly your need to import a "sibling" module in the same repo

[subtle plinth]

Maybe it is the same thing then 🤷🏼‍♂️

[plucky ermine]

Question of curiosity: is it possible

[sharp zealot]

Hey @rocky harbor @bleak nest I wanted to flag that we have some ongoing iteration on 2 parallel tracks, that have some dependency on each other: CLI UX and Docs. Will start at thread to coordinate that

[wintry prism]

Can you not call a function that returns a Secret from the cli?

I've tried this from a couple of SDKs

dagger call bar --token TOK plaintext

[sharp zealot]

Good question. Does it work? 🙂

[wintry prism]

no, but I might be doing something wrong, so getting a quick gut check. Mine was calling a specific implementation that wasn't working. Might work fine with something returning a Secret properly.

[restive shore]

Getting a nil pointer dereference in a new module I'm creating. I searched here and it looks like that's a symptom for multiple causes. I can't figure out why this is failing.

[restive shore]

How do I bust cache of a module function? Adding the time.Now() env var doesn't seem to work.

[rocky harbor]

They (currently) are invalidated every session (i.e. every time you invoke one CLI command). So if you dagger call fn --arg foo multiple times, then fn will execute each time.

Subtle-but-important detail: within the context of a session, they will be cached if invoked multiple times with the same input. So if you invoke a module function A from the CLI and A itself invokes another module function B multiple times with the same input, then function B will only run once. If we didn't do this you'd get crazy exponential runtime complexity.

So the answer to your question depends on exactly how you're calling the module + what behavior you're currently seeing 🙂

[restive shore]

Hey all, one more. I am trying to get a sidecar docker daemon running. however, no matter what I try I can't get my container to connect to the docker daemon. It can't find the host

error during connect: Get "http://6lrxqnijxx4lg:2375/v1.24/images/json": dial tcp: lookup 6lrxqnijxx4lg on 10.*.*.*:53: no such host

I am basically following the same pattern as here - https://github.com/kpenfound/dagger-modules/blob/main/dockerd/main.go. I see the sidecar service come up in the logs but when I shell into the container I can't do a simple docker images without getting the above error.

• • is a mask.

[rocky harbor]

Looks like it's probably the same bug as https://github.com/dagger/dagger/issues/6493

[wheat river]

Hey guys, how can I read environment variables, from a caller zenith module (through code)?

[sharp zealot]

Starting a nitpick thread here, since there are a lot of PRs in flight and I don't want to hold anything up - but don't want to drop my notes either 🙂

[kind carbon]

@rocky harbor, what's the new way to get the root dir in a module? I expected dag.CurrentModule().Source() to be ModuleSource but it's Directory.

[rocky harbor]

A module can't access the root dir anymore, that's what we decided on here: https://github.com/dagger/dagger/issues/6291#issuecomment-1909240049

It can only access its source dir. If it needs something from the root dir then it has to accept it as an argument. The only need for root dir now is the use case around needing go.mod/package.json/etc. from a parent directory, but that's only needed by the SDK, so we aren't exposing it to the end module

[sharp zealot]

This closes a big escape hatch / implicit API. I'm very glad we're doing that

[restive shore]

A module can't access the root dir anymore

[sharp zealot]

In the new world of "always dagger install", how much computation could we shift from call to install? For example, could we build the module at install rather than first call? This would make the crucial first call feel much snappier. Sure install would take longer, but people are used to that. It would go a long way to make Dagger feel fast on the first try - that first impression matters a lot.

Thoughts?

[rocky harbor]

dagger install currently results in all deps to be fully built and codegen to run for the module you are installing to, which is the the bulk of work of "building" a module, so I think this is mostly the case today already

[sharp zealot]

Then we just made the CLI experience feel snappier as a side effect 🙂 Love it

[sharp zealot]

And by "we" I mean "you" 😛

[rocky harbor]

Well to be clear I think this has been the behavior more or less for a while, so it’s the product of the 10000 layers of iteration we’ve all done the last few months, so it’s “we” 😁

[sharp zealot]

I think we have accumulated more layers of iteration than a Buildkit DAG could handle

https://tenor.com/view/high-mileage-car-dash-board-gif-25309540

[restive shore]

Does this also mean dagger call does not build deps or gen anymore? If so that would be huge for CLI experience, but a new thing (dagger install) for us existing Zenith users to get used to. I am cool with that tradeoff 🙂

[restive shore]

Is there a (or planned) guide for existing module devs to migrate to the new way of doing things?

[sharp zealot]

Yes, planned

Note that the current way of doing this will still be supported

[restive shore]

I'm doing Zenith to be on the cutting edge so current way goes out the window as soon as the new shiny thing comes along 🙂

[sharp zealot]

Ha ha 🙂 Well it will certainly become a more niche use case - not everyone actually needs to write a custom client that calls functions from the host machine. But sometimes that's actually the best way (for example if a tool wants to add support for calling Dagger Functions within their own UI)

The way we'll explain the transition:

• Everything is a function. When you query the Dagger API, you're calling functions and chaining them into dynamic pipelines.
• Until now, there was only one type of function (core) and only one way of calling functions (from an external client)
• With the Zenith release, there are 2 types of functions (core and external) and 3 ways of calling them (from the CLI; from another function; or from an external client)

So it's a 1x1 grid that becomes a 3x2 grid.

[plucky ermine]

Is there any easy way to tell in daggerverse which language a module was written in? this would be helpful for being able to easily find sample modules to build upon.

[restive shore]

Glad to see the external client use case is on the roadmap for Zenith release

[thorn moat]

dagger ate my JSON

[sharp zealot]

There isn't, but I agree it would be useful.

[thorn moat]

I think we have that data already (specifically which SDK) but we've been hesitant to show it prominently in the UI, since we don't want people to get the idea that they need to choose a module in their particular language

[sharp zealot]

Yeah it was pretty low on my list too. And I agree we have to be careful not to create that impression. But we got user feedback today, from someone who was looking for best practices to help them write their first module. And only saw modules in Go, leading them to a similar conclusion anyway. So ideally we'd find the perfect balance between those two constrains. I have no idea how 🙂

[sharp zealot]

@rocky harbor in my review of https://github.com/dagger/dagger/pull/6262/files# (zenith quickstart PR) should I assume dagger install && dagger run -m <shortname> is available, or is it better to leave that part alone for now?

There are a bunch of "dagger call straight from the URL" which I think will still work, but we might want to show off the new flow (and it's an easy patch in the docs)

[rocky harbor]

Those didn't make it for v0.9.8, so I think it's best to ignore at the moment and then update the docs once those changes are in the next release

[sharp zealot]

I believe dagger call -m FULL_MODULE_ADDRESS will continue to work same as today, with build-on-demand. So dagger install && dagger call -m SHORTNAME will be a backwards-compatible improvement. Correct @rocky harbor ?

Sorry Nipuna I forgot to answer earlier

[sharp zealot]

Roger that

[rocky harbor]

Yes correct on all accounts

[sharp zealot]

What's the story for documenting function arguments? Is that supported today?

[rocky harbor]

Yep, varies from SDK of course but here's Go for example: https://docs.dagger.io/zenith/developer/go/525021/quickstart#step-3-use-input-parameters-and-return-types
dagger call hello-world --help shows:

sipsma@dagger_dev:~/repo/github.com/sipsma/throwaway/potato$ dagger call hello-world --help
✔ dagger call hello-world [0.00s]
┃ Usage:
┃   dagger call hello-world [flags]
┃
┃ EXPERIMENTAL:
┃   dagger call hello-world is currently under development and may change in the future.
┃
┃ Flags:
┃       --count int   the number of potatoes to process
┃       --mashed      whether the potatoes are mashed        

[restive shore]

Last Zenith call there was a discussion about how to document the module itself. I don't think there was a consensus?

[sharp zealot]

Yes there is consensus. It's top-level code comments for now. And if needed later, an optional field in dagger.json as an override. No README

[restive shore]

Top-level code comments meaning? For example for Go would it be the module struct doc?

[sharp zealot]

Yes I believe so. Either that or the comments at the package level (very top of the file) which is how go doc documents a package I believe.

Documenting the module struct makes perfect sense to me

For sure we want to avoid relying on READMEs, because in our specific situation (interfaces still moving, multi-language examples + CLI examples) manually authored snippets just won't cut it. If we want high-quality module docs, we need to generate it ourselves.

Go packages don't have this problem, because you only need to document how to use your Go package in another Go program. So manual snippets in a README are fine

[restive shore]

Plus it goes in the godoc

[rocky harbor]

<@&785926252749651978> <@&1152040561344262174> As a general FYI, dagger v0.9.8 includes breaking changes for any modules that were using root in their dagger.json or were using the host API in their modules (e.g. dag.Host().Directory()). If you're impacted, I added instructions on updating to the description here: https://github.com/dagger/dagger/pull/6386

[restive shore]

I knew this but still messed it up because I didn't specify the sdk. Is there a way to fix that via CLI? Or do I manually edit dagger.json?

I somehow assumed dagger would derive the SDK :/

Or prevent me from doing that operation without an SDK

[sharp zealot]

What command caused this @restive shore ?

[restive shore]

dagger mod init --name=mymodule

[rocky harbor]

Ah yeah, support for reconfiguring via CLI it is coming in the release next week, but didn't make it in this one. So yeah either rm and re-dagger mod init or can hand edit. Sorry, that's annoying.

[restive shore]

From the previous root dir

no worries! life of an early adopter

[rocky harbor]

Oh interesting, mod init was intended to enforce that if only --name or --sdk was set, it errors out... I switched at the last minute to a cobra setting that's supposed to enforce it but maybe something got missed there. Will fix and test

[sharp zealot]

Going over the quickstart, I want to propose a calling convention:

dagger -m MOD call FUNC, instead of the usual dagger call -m MOD FUNC.

A few reasons:

• Often the module and function name are similar or identical. This convention makes it feel less redundant.
• I could see -m becoming a global flag. It already applies to query, it could easily apply to install, and even init - at least for modules on the local filesystem
• With the improvements to dagger run that we started discussing, it paves the way to dagger -m MYAPP run build

[restive shore]

If I was using the root to indicate I have modules that I want to "install" from the top level. Do I need to create a dagger.json at that root location anymore? I don't necessarily care for the content of that "root"

[rocky harbor]

You don't need to manually create it or anything, just be sure to do dagger mod init -m ... from the root directory for any modules that have local dependencies on each other. After that you can just dagger mod install and all the updates to dagger.json are done automatically

[restive shore]

dagger mod install in which directory?

[sharp zealot]

what does -m do after init?

[rocky harbor]

It's flexible at that point and should work intuitively. So in that example in the PR description linked above, I ran

dagger mod install -m testcontainers docker

from the root directory.

But it also works equivalently if you did e.g.

cd testcontainers
dagger mod install ../docker

[restive shore]

I am asking because it doesn't seem to work for me. I did dagger mod init for 3 of my modules. My modules A depends on C and B, I went into module A folder and did dagger mod install -m ../c ../b didn't work for me

[rocky harbor]

It just specifies what module you are operating on, with the default being the module in the current directory. So dagger mod install -m testcontainers ./docker is saying "install docker to the module testcontainers"

[restive shore]

ooooooooh it's a dest to target thing

[sharp zealot]

Makes sense 👍Another example where I prefer the dagger -m MOD install order, makes it more readable IMO

we may be able to retire --workdir soon no? -m kind of replaces it

[rocky harbor]

Oh yeah totally, at the very minimum it should not be a top-level flag on dagger and only be used by some of the older subcommands like listen, the hidden SDK util session command, etc. But maybe even those no longer need it.

Added a note to https://github.com/dagger/dagger/pull/6533#issuecomment-1922363722

[kind carbon]

Yeah, that's how I always use it 🙂

So used to it that it jumped out to me in the docs.

[sharp zealot]

Ok cool. My superficial quickstart review turned into a more in-depth one. I'll fix every instance of the call convention I can find.

[sharp zealot]

Going through the whole zenith quickstart, and going over every small detail, is laborious, but reminds me how powerful this feature is, and how far we've come. I'm excited to share it with the world 🙂

[restive shore]

When I run dagger call mod function shell How do I set the entrypoint? dagger shell used to have a --entrypoint flag.

I tried setting with-entrypoint --args before I invoke shell but that didn't seem to have any effect.

[sharp zealot]

There should be an argument to shell (which is getting renamed to terminal fyi). There is also with-default-shell

[restive shore]

ah I see --args. And tested it works. What's the reason behind changing --entrypoint to --args?

[kind carbon]

entrypoint was a global flag in dagger shell (CLI). It competed with other flags and could be put anywhere. args is an argument to the Container.shell field in the API and is consistent with other fields that set a command, like withExec, withEntrypoint, withDefaultArgs, and withDefaultShell.

[restive shore]

I see, should shell be withShell then? Or withTerminal?

I guess that wouldn't make sense as i don't want shell to return a Container . It's a special function. In that sense, it's not like withExec, withEntrypoint or withDefaultArgs.

[kind carbon]

Yes, exactly 🙂 It's more like Container.asService. How do you feel about dagger call my-ctr as-terminal? 🙂

[restive shore]

better IMO. consistent at least.

[rocky harbor]

Chatting about Zenith in dev-audio for anyone who wants to join 🙂

[sharp zealot]

@rocky harbor sorry I can't join. My topic: small breaking CLI changes and how quickly we can ship them. eg container.terminal

[rocky harbor]

We discussed that 🙂 We all seemed in agreement about changing --args to --cmd. terminal is slightly more controversial due to verbosity but I think we agree we should change it from shell.

how quickly we can ship them
This is fortunately all extremely easy to implement (a nice relief from everything else recently), so I'll send out a separate PR to update shell in a few and include the other name possibilities that were mentioned. I'll leave https://github.com/dagger/dagger/pull/6533 to just be focused on the commands currently under dagger mod so we don't glob too much together in one PR.

[rocky harbor]

https://github.com/dagger/dagger/pull/6582

[sharp zealot]

tty? 🙂

[rocky harbor]

That's in the PR as a suggestion 😄

[sharp zealot]

I love that it's short, don't love that it's cryptic unix lore for many

[rocky harbor]

Yes that's what I mentioned there too 🙂 It's also of course possible to support both, but will pollute the core API unless we bite the bullet on being able to annotate a single graphql field with short aliases

[thorn moat]

strawman: what about just exec? it's sort of like an imperative form of withExec, and would match docker exec et al

[rocky harbor]

Was just fixing this and adding a test but I actually am seeing the behavior that was expected, running in an empty dir:

$ dagger mod init --name=abc
Error: if any flags in the group [sdk name] are set they must all be set; missing [sdk]

Was dagger mod init --name=mymodule the exact command you ran? Just curious if there's somehow an obscure corner case where this could happen somehow

[thorn moat]

Anyone have 🌶️ takes on short-flags in dagger call? On one hand, it would make common emergent patterns like exec -it and up -n more palatable. On the other hand, I don't think module authors want to worry about this level of detail, and short flag names can be pretty cryptic. I tend to write longform names in scripts, but my fingers do appreciate shorthand in the shell.

[rocky harbor]

Was just fixing this and adding a test

[sharp zealot]

I just noticed the zenith commands are not hidden in main. Was that intentional?

[rocky harbor]

Yeah context here: https://github.com/dagger/dagger/pull/6509

[sharp zealot]

Yeah having those commands in the main stable docs is a problem. Maybe a small enough problem to ignore it for a couple more weeks.

The very first command in the CLI reference is dagger call, there's a notice "under development" but it's easy to overlook. If a lot of people go to this page, it will cause confusion. We can decide that it's not a lot of people, over a small enough window, that it's worth it

But if I had caught this earlier I would have said "wait until zenith is GA before merging this"

Especially since 0.9.8 didn't catch all the breaking CLI changes. So there's probably a 0.9.9 as early as next tuesday that will break (the experimental parts of) this doc

[restive shore]

Setting optional for Directory/File

[kind carbon]

It's in the Zenith docs, not the main docs. It was there before, just generated now instead of manually crafted.

Wait, @bleak nest, I didn't realize the reference in main has dagger call et, all. It should be only on the zenith version:

• Main: https://docs.dagger.io/cli/979595/reference
• Zenith: https://docs.dagger.io/zenith/reference/979596/cli/
  Notice they're different. The PR above only changed the zenith version.

[rocky harbor]

Oh yeah sorry I thought "main" was referring to the branch "main"; that is surprising

@sharp zealot @kind carbon I've updated+rebased https://github.com/dagger/dagger/pull/6533 to just enough updates that we can fan out the rest of the polish to parallel tracks (listed in description)

[sharp zealot]

UI note: in the quickstart, the output of dagger call commands doesn't show Engine: <engine version>. I assume that's because Vikram manually removes them. But it would be nice if we removed them also, at least by default (could be added by --debug)

[mossy hazel]

trying to find the best practices after the new restriction on escaping the module root dir. In particular when implementing the CI code of your modules, it's seems that the only way to load modules from local path is to have the main module at the root of the repo... I ended up with this dir structure: https://github.com/samalba/dagger-modules/tree/main
I think it's too restrictive for implementing your own CI (not everyone will want those generated files at the root).

the previous way of loading modules from ../ made those things more flexible. It'd be nicer to at least authorize relative path from the same git repo?

[sharp zealot]

Just submitted a 2nd pass of quickstart edits FYI, in case anyone else is working on that and wants to review

[thorn moat]

Current state of the new TUI - feedback welcome! 🧵

• dagger call ... up: https://asciinema.org/a/8omoN7sxZrfnz0m1lWsayeRnG
• dagger call ... stdout: https://asciinema.org/a/f3Oj2Dkr9DF0XlAT844kwbgVj

[bleak nest]

I'll open a PR to remove this from the stable docs, it's a quick fix.

JFYI this question came up earlier (when we released 0.9.4) and my understanding was it was ok to have the new commands in main with a warning note. For reference: question #1181205023053582408 message, answer #1181205023053582408 message and follow up #1181205023053582408 message

[bleak nest]

btw Helder, can we use the CLI reference generator to also generate the stable docs CLI ref page, without the experimental commands?

[plucky ermine]

This may be a problem on my end, but I am seeing some strange behavior in 0.9.8

I am serving up a service and trying to bind it to port 1313 using a command that worked before, however when it starts up dagger seems to bind it to some other port that appears to be randomly assigned.

this command:

dagger call --focus=false -m ci --port 1313:1313 serve --dir . up

results in this tunnel:

┃ 65505/TCP: tunnel 0.0.0.0:65505 -> km5ul81gpadea.8d975us9eml4u.dagger.local:1313  

[thorn moat]

You have to pass --native for it to forward the ports to the same ports on the host. This will be the new default in the next release, with --random for the current behavior (https://github.com/dagger/dagger/pull/6585)

[plucky ermine]

Thank you very much!

[wheat river]

Hey guys. What's the recommended approach in modules for socket forwarding, particularly in the context of SSH/auth for GIT private repos (https://github.com/Excoriate/daggerverse/blob/6afe1de075be4196d68ef697a9d8f29c97a0f906/iac-terragrunt/main.go#L102)
In the classic way, having client.Host().UnixSocket(sshAgentPath) works well, what's the supported approach in zenith? In my current use case, my module is going to be used mainly from the CLI.

[sharp zealot]

Hi Alex. You're encountering what we call the "ambient system access" problem: containerized functions can't access the caller's environment variables, network interfaces or filesystem; but sometimes they need it.

The dilemma is whether to choose the convenience of ambient access or the safety and scalability of a strong sandbox? We intend to solve this problem in a way that gets us both.

We are approaching the problem in layers, like Git with its plumbing and porcelain:

1. The plumbing: dagger call. Functions can receive secrets, files, directories, containers etc as arguments. We make sure the CLI can pass useful values via command-line flags to dagger call. Note: I don't know if we support this for unix sockets, but if we don't:
   we should and we will.

2. The porcelain. 90% of calls have at least one argument which needs a default value, but can't have it because of sandboxing. So everyone has to type it out, and it's annoying. We're going to design an optional "porcelain" user interface that solves that usability issue for that 90% of simpler calls, without getting in the way of the other 10%.

Here's one issue where we discuss all this: https://github.com/dagger/dagger/issues/6112

[wheat river]

Thanks @sharp zealot , that's clear. The first point that you're mentioning sounds great. I don't think it's supported now, at least as part of the latest version that I'm using (dagger v0.9.8 (registry.dagger.io/engine) darwin/arm64) .

I'm aware about that GH issue, it's indeed a 'big' one considering that it entails (important design decisions are dependent on the outcomes of that thread). This —at least in a few of the use cases that I'm tackling with Dagger 'modules'— will force me to use Dagger 'classic'. The SSH authentication is quite a common scenario that need forwarding the unix sockets from the host in order to leverage SSH authentication properly, unless there's a workaround in Zenith

[bleak nest]

@sharp zealot I added your changes in plus a few others. Could you review and approve? I would like to merge today and then we can continue to iterate in a follow-up PR if needed.

[ashen ember]

Hi!

I might be missing something but is it possible to do this kind of recipe with the new dagger? https://docs.dagger.io/cookbook#transfer-and-write-to-host-directory-from-container

Basically there is some state generated by my dagger containerized action that I want back in the host so it can then be put back during some other calls.

In classic Dagger (not sure that's how you call it, if not apologies 🙂 , something like this works

contents, err := client.Container().
    From(cliImage).
    WithDirectory("/tmp", client.Host().Directory(".")).
    WithSecretVariable("CHAINLOOP_ROBOT_ACCOUNT", token).
    WithExec([]string{"attestation", "init"}).
    Directory("/tmp").
    Export(ctx, ".")
if err != nil {
    log.Println(err)
    return
}

since that run creates a file in /tmp/something-somwthing-state.json that it's writen back into . in my host.

But I don't seem to be able to make it work with the new Dagger. Am I missing something? Thanks in advance 🙂

[honest hearth]

Just some stream of consciousness and gut feedback going through the docs for the first time as someone who has before only tangentially touched Zenith:

https://docs.dagger.io/zenith/quickstart/883939/containers

dagger call -m github.com/shykes/daggerverse/wolfi base with-package --name cowsay container

It's not immediately obvious to me chaining of commands here, like it wasn't very clear at first glance that return of base is going into with-package which is then going into container. I figured it out after reading it a couple of times though. Perhaps asking the end user to run dagger functions first with the module might make it more obvious and be more natural to the flow of someone discovering a CLI for the first time. In other words, if you're a dagger person, what's the workflow you might use if just given a URL for a module and nothing else to discover things about that module? Is it just to start running dagger functions -m mymodule and walking down the tree? Whatever it might be, this guide should probably follow that same process.

I also found it a bit verbose when I had to walk the functions down to discover what child functions existed. In other words if I run

dagger functions -m github.com/shykes/daggerverse/wolfi       
✔ dagger functions [0.00s]
┃ Name   Description           
┃ base   Initialize a Wolfi base configuration      
• Engine: 71f9f4d21fc2 (version v0.9.8)

Then I have to walk down and run

dagger functions -m github.com/shykes/daggerverse/wolfi base

to see what is in it. That's perhaps what you might expect to see in a CLI but it just felt a bit verbose. Maybe a dagger explain or similar that unrolls the chain from a full command and explains what's going on (or a dry-run flag, which is usually similar) could help?

[honest hearth]

This may actually be a terrible idea, but is there a way to save yourself some typing and/or copy pasting when typing out dagger -m mymodule? Like an alias in your home dagger.json or something? I absolutely love the obviousness of either referencing a local path or a remote git in the -m command, because it just points at source code, and what's more obvious than that, but it does feel like these module urls have the potential to be reaaaaaly long sometimes

[sharp zealot]

Thinking through the implications of // +optional in the Go SDK...

How will it impact intra-module function calls? Calling one module's function from another function in the same function.

• Obviously using Opt() would break, but not having to do that anymore is a plus
• Default values won't be applied. Is that a problem?
• Optional status won't be respected. What does a call look like with lots of optional arguments? How do I pass "don't set this argument"?

[latent trellis]

Thinking through the implications of `

[unreal moat]

can someone maybe please help me understand what is expected in terms of exporting file and directory types in zenith. When I call this function ```
@func
async addHostFile(directoryArg: Directory): Promise<File> {
const filename = ${Date.now().toString()}-foo.json;
const file = dag
.container()
.from("alpine:latest")
.withMountedDirectory("/mnt", directoryArg)
.withWorkdir("/mnt")
.withExec(["echo", ${crypto.randomBytes(7).toString("hex")}, ">>", filename])
.file(/mnt/${filename});

file.export("/tmp/foo", { allowParentDirPath: true });
return file;

}```
with this invocation dagger call addHostFile --directory-arg /tmp/foo

I get the following output

I get the same result if I remove the file.export call and run dagger call addHostFile --directory-arg /tmp/foo export --path /tmp/foo from the command line

[restive shore]

This might help - https://discord.com/channels/707636530424053791/1204087468203450440

[kind carbon]

witExec doesn't execute in a shell. You can wrap in sh though:

      .withExec(["sh", "-c", `echo ${crypto.randomBytes(7).toString("hex")} >> ${filename}`])

Or use this option:

      .withExec(["echo", `${crypto.randomBytes(7).toString("hex")}`], {redirectStdout: filename})

Not saying this fixes your issue since I haven't looked closely yet, but it stood out to me.

[upbeat herald]

@rocky harbor @thorn moat Hey 🚀

I'm confused about the new source thing in the dagger.json, how can I access a parent directory?
Like if I have

codebase root parent
ci/
   # My module here

How can I access it? I tried to update source to .. but it gives me my modules files (in ci)

[sharp zealot]

Calling for bikeshedding... We're considering a default value for the "source" field of a new dagger.json. The question is: what should the default value be?

This would be a default path where the source code of a module would go. The equivalent of ./ci in most repos today.

[dense canyon]

👋 is there a way to define functions in a module that I want to make public so they can be used used from different modules but I don't want them to be callable via dagger call as the arguments don't make sense? i.e: functions with complex types as arguments.

cc @rocky harbor ?

[rocky harbor]

No not currently. We could consider updating the CLI to not expose functions with args it can't support yet though cause I get why that would be confusing

[dense canyon]

sounds like a good idea. Shall I open an issue?

[rocky harbor]

Yes please 🙏

[dense canyon]

https://github.com/dagger/dagger/issues/6608

[dense canyon]

I'm creating a module that needs to check if the host is effectively listening on a port to do something. Given that we've removed the Host API from modules, is there a way to achieve this currently? Given that we don't have the c2h network anymore

[rocky harbor]

You want to check from the module whether the caller's host (as in where the CLI runs) is listening on a port? Or check whether a port is being listened on in the module container?

[dense canyon]

the caller's host (as in where the CLI runs) is listening on a port?
this

Basically trying to use c2h with modules

[rocky harbor]

Oh okay that was never possible even when Host existed (good example of part of our motivation for removing that). In general it's not possible for modules to interact with the caller's host in any way, everything has to be an explicit argument.

[dense canyon]

makes sense. I can work around this, so no big deal. I was thinking in the use-case of some modules needing to interact with the CLI caller's service.

i.e: I start my app locally and I run a dagger call module which runs a benchmark against my app

I'm forced to daggerize my app now if I want this to work. As I'd need to connect them via services

Maybe this use case will become possible when we find the way to integrate modules with the plain SDK somehow 🤔

[rocky harbor]

You can accept an arg of type Service and pass it from the CLI like --svc tcp://localhost:1234. If you want to then interact with it direct from your code I think it would just be svc.Up(ctx) (though that will block so needs to be in separate goroutine/thread/etc.). We actually don't have coverage for that exact scenario in our integ tests yet though (just some related ones), so let me know how it goes if you try it 😄

[dense canyon]

nice, can give that a try 🙂

[dense canyon]

doesn't seem to be supported:

265: [0.23s] input: resolve: loadServiceFromID: up: failed to select host service: tunneling to non-Container services is not supported

that's ok, I can daggerize my app and keep everything in Dagger for the moment.

[rocky harbor]

Ah okay, that error sort of makes sense I guess given the previous use cases around up, but I suspect there's a path to allowing that to work since this use case makes perfect sense. cc @thorn moat

EDIT: or we could just add tunnel to the CurrentModule API, which is the module-specific implementation of various APIs that used to be on Host.

In the meantime you could bind svc to a Container and use it from a WithExec (that is covered in our integ tests). Or yeah daggerizing it all is an option too 🙂

[dense canyon]

we could add tunnel but thinking a bit about it, I agree it's confusing given that we have the runtime containers which kind of makes the system not to easy to reason about. Maybe sending Service via the argument is more clear? IDK...

[rocky harbor]

If we added tunnel there you'd still use it from Service args like dag.CurrentModule().Tunnel(svc), so I think it would work out nicely. By default though it would have the same error as above I'm pretty sure, so we'd need to fix that in either case.

[normal blaze]

Hey folks, just picking up my daggerization of Jumppad again, one of the things I am trying to do currently is upload files to GitHub actions archives. I am creating a module that accepts *File, however I need to read the contents as a binary representation to calll the GH api

I see that Contents returns a string, is there a way to get the raw bytes?

[sharp zealot]

there's no raw byte type in graphql , so I'm not sure how we encode the raw bytes from buildkit api into that gql string

her's the spec

[normal blaze]

Would it be possible to do some wrapping tricks, or even just have base64 strings as an option

[sharp zealot]

it appears we need to get on that 👍 im curious what you get today

[normal blaze]

The internal struct must handle binary as I can mount file to a container

But in this instance for a module I think it makes sense to deal with the file from the module code

Ah, but, looking at this, all the file operations are also graphql calls. I am guessing that when the dagger engine creates the container it mounts the file from a location to it. This is why files are represented by IDs. So the output of the file is not passed through the graphql api just a reference to it

[sharp zealot]

correct

[normal blaze]

I am playing with the idea of exporting a file to the host but then I suspect the module will not be able to read this file as the module is running inside of a container

actually that works

// example usage: "dagger call upload-archive --file ./file.tar.gz"
func (m *Actions) UploadArchive(ctx context.Context, file *File) error {
    fmt.Println("Uploading file")
    _, err := file.Export(ctx, "/tmp/file.txt")
    if err != nil {
        return fmt.Errorf("unable to export file: %w", err)
    }

    fmt.Println("File uploaded to /tmp/file.txt")
    d, err := os.ReadFile("/tmp/file.txt")
    if err != nil {
        return err
    }

    fmt.Println("File content: ", string(d))

    return nil
}

[restive shore]

I think that file is inside your container and not actually on your host.

[kind carbon]

That's ok because he needs to access the binary format in go to upload to GH archives.

[upbeat herald]

@thorn moat @rocky harbor When calling constructor, is it normal that the function is called multiple time?
One time without argument loaded and 1 time with, I could assert this behaviour when I tried to do some dag logic inside the constructor itself.

Example:

  constructor(source: Directory) {
    console.log("called with:", source)
  }

Displays:

called with: undefined
called with: Directory { ... }

I found that issue because I tried to use source inside the constructor and everything was breaking because source was undefined

(Maybe it's something broken on my side, I'm trying to find what)

[thorn moat]

can you repro with the Go SDK? haven't seen that before, but I've also never tried calling anything against the arguments

the fact that it's undefined and not null feels a little suspicious

[upbeat herald]

Maybe it's typescript specific, I'm trying something that will give me the answer in a sec

Ok it's typescript specific

And that's funny, it's when I load the typescript files, for some reason the constructor is called but it shouldn't 🙂 and it's called without the dagger argument indeed

[thorn moat]

huh weird (i say not knowing much about TS)

[upbeat herald]

https://tenor.com/view/nervous-sign-of-the-cross-praying-hoping-nico-santos-gif-21421248

[upbeat herald]

@thorn moat I fixed my issue but this creates another minor issue that I cannot workaroud without abstracting the default value notation in Typescript (which is I think not a good idea at all)

I have a question: in Go, if you have fields inside your object, do you expect the user to call the constructor if there are default value?

The thing is that in Typescript, constructor are optional:

class Foo {
   prop = "a" // works
}

BUT in order to make that happens, Typescript still require to call the constructor (but the assignment in done implicitely)
My 2nd question is, if I have

@object()
class Foo {
   a = "a"

   @func()
   zoo(): string {
     return this.a
   }
}

And do: dagger call zoo, do dagger will first call the constructor or it will directly call zoo?
In that case, we need to enforce constructor to assign fields

Ok I found a workaround to fix the side effet of my fix ^_^

@object()
class Foo {
   a = "a"

   // Placeholder constructor so dagger call constructor and init properties in the state
   constructor() {}

   @func()
   zoo(): string {
     return this.a
   }
}

[thorn moat]

at least in Go, the constructor is always called, I don't think there's a way to pre-define defaults for the fields at the moment but could be wrong (cc @deft rain - do we support pragmas for fields?)

checking for consistency with Python might be less confusing (cc @kind carbon)

[upbeat herald]

It's a specific Ts issue that made me does that.
Basically to call a function from a class, I need to instanciate the class but the constructor might requires arguments.
So instead I use Object.create that instanciate the class without calling the constructor.
The side effect is that it's not loading properties (logic because the constructor isn't called).
However, since dagger has its own state management, I can assign properties after the instanciation.

Meaning that everything works fine as long as properties are register in the state (aka calling the constructor one time to init field)

And since Go needs New to assign default value, Typescript will follow the same behaviour

@kind carbon For more context, my issue was:
1 - I have a constructor with required arguments
2 - I'm calling a function of this class

-> How do you instanciate the class to call the function without the required argument in the constructor
---> You can't -> boom error

Maybe you have the same issue in Python (I didn't tested)

[kind carbon]

Let me test this in python 🙂

[upbeat herald]

Is your constructor the default one or this a method you define? I see you have a create function, I guess it's not the standard typescript constructor

A way to work around my issue and fix all bugs in one time would be to change my constructor function into a named one like

@object()
class Foo {
  create(): Foo {
    init stuff in it
  }
} 

And basically I ignore constructor (meaning I create the class with the normal new Foo()), and then when dagger call the constructor, under the hood I call create

[kind carbon]

In python, the @object_type decorator generates a constructor automatically based on the fields. field(init=False) means "don't put this in the constructor" for example.

[upbeat herald]

What if I want to add logic in the constructor during my field initialization?

I hit the original issue because I did that:

  @field()
  project: Container

  constructor(source: Directory) {
    this.project = dag.node().container()
    // Extract package.json and yarn.lock to a temporary directory
    const dependencyFiles = dag
      .directory()
      .withFile("package.json", source.file("package.json"))
      .withFile("yarn.lock", source.file("yarn.lock"))

    // Get source without generated files nor useless files.
    const sourceCode = dag.directory().withDirectory("/", source, {
      include: [
        "**/*.ts",
        "tsconfig.json",
        "package.json",
        "yarn.lock",
        ".mocharc.json",
        ".eslintrc.cjs",
        ".prettierrc.cjs",
      ],
      exclude: ["node_modules", "dist", "dev"],
    })

    // Install dependencies and add source code.
    this.project = dag
      .node()
      .withPkgManager("yarn")
      .withSource(dependencyFiles)
      .install() // Install dependencies prior to adding source to improve caching
      .withSource(sourceCode)
      .container()
  }

[kind carbon]

Better make a 🧵 for this

[upbeat herald]

So I can call dagger --source=. call lint and this will work without extra call

Otherwise I need to make a function project that return an object with lint, build, etc..

[upbeat herald]

A what? 🤣

[kind carbon]

Thread 🙂

[upbeat herald]

Oh yeah sorry

I hit the original issue because I did

[deft rain]

we do support pragmas for fields, but only for private at the moment: https://github.com/dagger/dagger/blob/main/cmd/codegen/generator/go/templates/module_objects.go#L127-L133

we could add defaults potentially?

[thorn moat]

eh, don't think it's a big deal, constructor + arg pragmas feels fine to me, and will behave better for complex defaults like Container

seems nice-to-have, but for complex defaults you'll end up rewriting it in constructor style anyway

[normal blaze]

Yes that is correct 🙂 on a side note, there is no public API for creating archives, but this technique does work for adding release assets.

[spiral whale]

Hey guys, I just missed last release changes and trying to find some new functionality. How can I translate dag.Host().UnixSocket(dh) to new version? cc: @rocky harbor @deft rain

[wintry prism]

Hey guys, I just missed last release

[thorn moat]

Requesting anyone with bandwidth to help review the new TUI so we can sneak it in to v0.9.9 with higher confidence: https://github.com/dagger/dagger/pull/6588
Hoped to have it ready much earlier today so there'd be more time to review, but one thing led to another, as tends to happen with TUIs...
I recorded a bunch of demos if you want to just eyeball it, but it'd also be great to try and break it.
Big thanks in advance! - and special thanks to all who gave feedback: https://discord.com/channels/707636530424053791/1203178812729458708

[sharp zealot]

in the Go sdk , how can I drop to raw graphql queries?

[rocky harbor]

dag.c.MakeRequest, e.g. we do that for one of our tests here https://github.com/sipsma/dagger/blob/6163a9b5ac173f6eb75d292dcf9618ce6bd59f90/core/integration/module_test.go#L2579-L2579

[normal blaze]

Not sure if this is covered by the new TUI, but some feedback when using a module in a module. If the module function fails, even when the error is being discared, an error line is still output in the terminal.

➜ dagger call -m="./.dagger" get-version --token=GITHUB_TOKEN --src=.
✔ dagger call get-version [13.57s]
┃ 0.0.0                                                                                                                                                                                    
✘ nextVersionFromAssociatedPrlabel ERROR [1.06s]
✘ exec /runtime ERROR [1.05s]
┃ failed to get pull requests: GET https://api.github.com/repos/jumppad-labs/daggerverse/commits/09d1f68c6571e7de1a29a459878a9367d12da555/pulls: 422 No commit found for SHA: 09d1f68c6571e
┃ e1a29a459878a9367d12da555 []                                                                                                                                                             
• Engine: d35f3a4fa541 (version v0.9.8)
⧗ 16.89s ✔ 304 ∅ 31 ✘ 2

This resulted from the following code

    // get the next version from the associated PR label
    v, _ := dag.Github().
        NextVersionFromAssociatedPrlabel(ctx, owner, repo, ref, GithubNextVersionFromAssociatedPrlabelOpts{Token: token})

    // if there is no version, default to 0.0.0
    if v == "" {
        v = "0.0.0"
    }

    log.Info("new version", "semver", v)

    return v, nil

While NextVersion.. did actually produce the error listed in the terminal output, i would not expect it to be shown in the output as I am catching the error and disposing.

[warped canyon]

Anyone around to help me out real quick? I'm trying to move my dagger.json from a ci/ subdirectory to the project root. I moved the dagger.json, updated the relative paths for local modules, and added "source":"ci", but it seems like I'm still missing something

[deft rain]

sure, wanna pop into dev-audio for a moment?

[warped canyon]

yup!

[warped canyon]

module works on 0.9.8 so i'll stick with that today

[deft rain]

awesome 🎉

[sharp zealot]

Did you find the issue?

[warped canyon]

the issue was I thought the source changes were in 0.9.8, and then switched to main and one of my modules wasn't working on that version. So i'll be using 0.9.8 today

[sharp zealot]

Any chance we could get rid of that "engine: ..." version line that shows up in every single dagger call output?

(oh that might be part of the coming TUI improvement)

[warped canyon]

Here's the issue I ran into on main: https://github.com/dagger/dagger/issues/6624
feels specific to typescript dependency + codegen

[rocky harbor]

Looking into it, thanks for the report!

[thorn moat]

This is fixed with the new TUI 👍

[sharp zealot]

Note: we should hide dagger publish for 0.9.9 given 1) daggerverse launches a bit later, and 2) the publish flow may change

safer to hide for now

[sharp zealot]

Attention Zenith developers: we are getting closer to a release. 0.9.9, which drops tomorrow, includes our last batch of breaking changes (that we know of) before 0.10. After that it's going to be mostly bugfixes.

If you already have Zenith modules, here are some breaking changes to watch for:

1. **optional arguments (Go only)*. If you're using Optional[foo], that goes away in 0.9.9. There is a new system based on // +optional and similar Go comments. We don't have a dedicated docs section for that (yet), but there are examples spread across the Go SDK docs: https://docs.dagger.io/zenith/developer/go/457202/test-build-publish . Just search for "optional".

2. Access to module source code. If you're using dag.Host().Directory(".") for accessing your module's source code, you need to call dag.CurrentModule().Source() instead.

3. root field in dagger.json. If you've added this field, your module will break, you need to change the content and/or location of your dagger.json. See the next section.

4. Modules in monorepo. We've cleaned up and clarified how to embed Dagger Modules in a monorepo. It is still possible and encouraged, but there are new rules, which may break your current monorepo setup. If you relied on dag.Host().Directory() to access other parts of the monorepo from your functions, for example application source code, that is no longer possible. You need to pass source code explicitly as an argument. You're also encouraged to always initialize a module at the root of your monorepo. Then depending on what you need, either add source code to that module with dagger develop --source=PATH, or create sub-modules in subdirectories. We will add a section about Dagger in monorepos in the docs.

Thank you all for your help, we're getting close to the finish line!

[sharp zealot]

Also: new TUI is SAWEET!

[thorn moat]

Hey hey, we're doing the Zenith office hours in https://discord.com/channels/707636530424053791/911305510882513037 - feel free to pop in if you have any questions or topics to discuss!

[restive shore]

I went into each of my modules and ran dagger develop which updated the dagger.json to the new version. removed the parent directory dagger.json (daggerverse/dagger.json) didn't run into any issues with that.

[kind carbon]

@warped canyon and @restive shore, I just tried on my daggerverse and there's no need for source. This is in my root:

{
  "name": "daggerverse",
  "dependencies": [
    {
      "name": "hello",
      "source": "hello-world"
    },
    {
      "name": "poetry",
      "source": "poetry"
    }
  ]
}

Notice I changed the name for hello-world. So instead of specifying the path with -m, you just specify the name:

dagger -m hello functions

[restive shore]

and you made the daggerverse folder itself into a module right?

[kind carbon]

That was just:

dagger init
dagger use poetry
dagger use hello-world

[kind carbon]

Not quite. It's a module because it has a dagger.json, but there's no source. No SDK, no generated files for top level. Just dependencies that you can call directly from the CLI.

[restive shore]

ah interesting. Changing the name from hello-world to hello you did via a cli flag? or manually?

[kind carbon]

You should be able to:

dagger use hello-world --name=hello

But I got an error, not sure if it's just Python. So I changed it manually just to POC.

Target module may need updating, let me try with new one 🙂

[restive shore]

After doing this you still need -m?

[kind carbon]

Of course, but instead of it being a path, it's a short alias. Source could be several levels deep:

{
  "name": "dagger",
  "dependencies": [
    {
      "name": "python",
      "source": "sdk/python/dev"
    }
  ]
}

Becomes:

dagger -m python call ...

[restive shore]

ok.. that makes sense. Didn't see the benefit before

if they are not nested, then no need of it

unless you want to call your module by an alias instead of folder name

[kind carbon]

Name defaults to the name of the dir, yes. Can also help with using two modules with same name. And you can have external (and pinned) sources mixed with local ones. All accessible by a short alias from the root of your repo.

[restive shore]

What would be a use case to have external modules in the daggerverse folder? Individual modules will still have to install the external modules right?

[kind carbon]

Ok, so this worked:

dagger init --sdk=go wat
dagger use wat --name=bat

Which gives:

{
  "name": "daggerverse",
  "dependencies": [
    {
      "name": "bat",
      "source": "wat"
    }
  ]
}

Note: last path in dagger init is the target directory. No need to mkdir && cd anymore.

[kind carbon]

You could have an external module that you find useful for some task and have it accessible here. It's gonna be pinned and callable with a short name.

[restive shore]

I can't think of a use case right now but I'm sure there will be modules that help build modules.

[warped canyon]

what's the alternative for GetOr() with the new // +optional syntax?

in the docs page it shows

if nodeVersion == "" {
    nodeVersion = defaultNodeVersion
}

but there must be a proper GetOr() type of call, right?

or some way to provide a default at least

[warped canyon]

I might have found my answer @thorn moat https://github.com/vito/daggerverse/blob/main/go/main.go#L184

[thorn moat]

yep yep

advantage of this style is that the defaults can show up in Daggerverse / at the schema level

https://daggerverse.dev/mod/github.com/vito/daggerverse/go@4809c149d0ed114dc199bc17cfec6b51e4f27858#Go.gotestsum

[warped canyon]

very nice

it doesn't show the unspecified defaults for optionals though

like race: Boolean = false

[thorn moat]

probably because that's actually defaulting to null atm, like Boolean vs Boolean! at the schema level

[warped canyon]

Here's my git diff for updating optionals in one of my modules, if anyone needs a reference. The changes to everything except main.go were made by dagger develop automatically ~~https://github.com/kpenfound/dagger-modules/commit/434e1241b411809e3be7f7e4daa66dc896db3344~~
actually here's a better one with a default arg https://github.com/kpenfound/dagger-modules/commit/75312746d4654ee0679647bc04921e47ba47c92f

Another question on new optionals ❓
Before you could pass an optional from one function into another without having to default in both. I guess in this pattern we'd have to set the default in each function even though it's passed all the way through.
For example, in the last commit I linked, Attach() and Service() both had the optional docker version, but it was only used in Service() since Attach() wraps it, so before Attach() didn't have to parse/default the input

I guess I answered my own question 😂 just something to watch out for while updating!

[restive shore]

You can also do optional + default in constructor if you want to use the same arg in multiple functions

[warped canyon]

yeah that's probably a good call for that module. It's from before the constructor days!

[sharp zealot]

Going to try and fix my modules too. this is useful 😁

So far I'm getting very cryptic errors

for example try calling my hello module in 0.9.9

no idea what the error means

[warped canyon]

it is cryptic 😂 I come bearing secret knowledge that invalid type: invalid type means it doesn't know what Optional[] is

[restive shore]

dagger call errors are also kind of hard to get to. They appear at the top of the CLI help.

 dagger call --help
Error: module at source dir "redacted" doesn't exist or is invalid
Call a module function and print the result.

If the last argument is either a Container, Directory, or File, the pipeline
will be evaluated (the result of calling `sync`) without presenting any output.
Providing the `--output` option (shorthand: `-o`) is equivalent to calling
`export` instead. To print a property of these core objects, continue chaining
by appending it to the end of the command (for example, `stdout`, `entries`, or
`contents`).

Usage:
  dagger call [flags] [FUNCTION]...

I would expect them to appear at the end. Or even skip the help when it errors

[sharp zealot]

https://tenor.com/view/here-catch-present-gift-surprise-gif-16380292

[sharp zealot]

If the last argument is either a Container, Directory, or File, the pipeline will be evaluated (the result of calling sync) without presenting any output.
Providing the --output option (shorthand: -o) is equivalent to calling export instead. To print a property of these core objects, continue chaining by appending it to the end of the command (for example, stdout, entries, or contents).

This behavior needs to be changed. Lots of interesting content we can print

[restive shore]

If I delete a module directory created via dagger init --name module and re-run the same command dagger still thinks the module exists.

also cryptic error when I try to init a dagger module - dagger init --name sonar --sdk go sonar

Error: failed to generate code: input: resolve: moduleSource: withName: withSDK: withSourceSubpath: resolveFromCaller: asModule: failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name sonar --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

I am actually unable to init a module even with dagger init --sdk=go sonar

This now tells me the module exists, but it doesn't. No folder or reference in dagger.json dagger init --name=sonar --sdk=go --source=.

mkdir sonar && cd sonar
dagger init --name=sonar --sdk=go --source=.

^^^ does not work either.

Stderr:
Error: load package ".": package name is empty

Apologies if this is a bit of a messy message. Trying to record exactly what I'm seeing as I go through the progression.

I am not sure how this all go into this bad state.

I think some kind of state is cached in the engine.

If I cd into the sonar directory and do dagger init --sdk=go --source=. I get the following

Error: cannot update module source path that has already been set to "daggerverse/sonar/dagger"

But I can't find where that reference is coming from. It's nowhere in my folder structure.

tried a different (new) module name

mkdir sonar2 && cd sonar2
dagger init --sdk=go

gave me

Error: load package ".": package name is empty

I've somehow gotten into a bad state that doesn't let me create a module at all

[sharp zealot]

This did not happen. Only meetings. "kill me now" level of meeting burnout

[sharp zealot]

Small TUI nit @thorn moat . The module loading takes a lot of real estate in the terminal, because we show it in full detail. Any way we could collapse it to show only a top-level "load module" function call or something like that?

[thorn moat]

Yep, want to get to that soon

[latent trellis]

How do I depend on an other module in my daggerverse repo?

This seemed to work at first: https://github.com/sagikazarmark/daggerverse/blob/main/archivist/dagger.json#L7

But when I try to install it, it fails:

led to generate code: input: resolve: moduleSource: withDependencies: resolveFromCaller: as
initialize dependency module: failed to create module: failed to update module dependencies: failed t
th "/arc" escapes roo

I thought that's what root-for was for, but I can't really keep up with all these changes anymore.

[sharp zealot]

we are done with those changes 😁 Sorry it was a fast-moving couple of weeks.

In the end we removed root-for, and root. You just import at any other local path and it should just work, within the current git repo

[latent trellis]

Glad to hear. 😄

The problem arises when I try to import a module from my daggerverse that imports something else locally.

[sharp zealot]

might be a bug. that's supposed to just work in 0.9.9

[latent trellis]

Repro: dagger install --debug github.com/sagikazarmark/daggerverse/archivist

Wait a minute

Where did the dagger.json go from my repo root?

Is that even still necessary? 😄

Here is a more complete log

It complains about path escaping a root

So the local path import doesn't seem to work when installing the module remotely

[wintry prism]

When I want to start working with functions, I expected to.

1. mkdir mydir && cd mydir
2. dagger init
3. dagger install <cool modules>
   4 dagger functions # to see what I've got to play with...

Is that right?

[sharp zealot]

dagger init will create the directory for you now

[wintry prism]

This is me looking to play with functions on the CLI.

[sharp zealot]

In the future dagger install without first doing dagger init will install in your home directory. But at the moment that's not implemented

[wintry prism]

ah okay, I didn't get a new subdir with init in 0.9.9

in my empty dir

[sharp zealot]

I think it's an optional argument, probably defaults to .

[wintry prism]

I got a error from functions, but I see why. I had installed this mod and it needs an update

civo-cluster ➤ cat dagger.json                                         git:main
{
  "name": "civo-cluster",
  "sdk": "go",
  "root-for": [
    {
      "source": "."
    }
  ]
}

My error

Error: input: resolve: moduleSource: resolveFromCaller: asModule: initialize: module name and SDK must be set

not the best error 😉

but! I ran dagger develop on that module 👆 and it fixed up the dagger.jsonnicely 🙂

[sharp zealot]

root-for is deprecated btw. I don't think it even made it into a release. you can safely remove that field

[wintry prism]

yep. I like how it was fixed by develop

[wintry prism]

surprised I can't do this flow for installing a bunch of modules and then seeing what functions I could run from CLI with dagger functions, throws error

Error: input: resolve: moduleSource: resolveFromCaller: asModule: initialize: module name and SDK must be set

Figure we need to make that flow work or provide another way to get list of functions available from installed modules since dagger call --help will give help for dagger call and won't give function names.

https://github.com/dagger/dagger/issues/6644

[ashen ember]

nob question. I there any way to send log information back to the CLI output so I can show the users of my module (when run via dagger call ) some feedback?

For example, if I have this function

func (m *MyModule) TestOutput(ctx context.Context) (string, error) {
    log.Println("the command is doing such and such ...")
    fmt.Println("... and this and that")
    return "the-actual-output", nil
}

the result of instantiating it with the CLI is

$ dagger call test-output
✔ dagger call test-output [0.43s]
┃ the-actual-output                    

Is there any way of showing additional context in the terminal that's not necessarily the cmd output?

[wintry prism]

There are use cases for 1) "Daggerizing" a project with a Dagger CI implementation and 2) creating a module as a reusable component.

The result of the change decided here (#daggernauts message) for use case 1 may have an impact on use case 2. It seems probable that many would end up on a path where the module they create for Daggerverse going forward will have a dagger.json at the root with a dagger/ subdir with the module implementation unless they explicitly specify --source pointing to same dir as dagger.json.

path A

dagger init mymod # creates mymod dir
cd mymod
dagger develop --sdk python # creates dagger subdir

path B

mkdir mymod
cd mymod
dagger init --name mymod --sdk python # creates dagger subdir

.
├── dagger
│   ├── pyproject.toml
│   ├── sdk
│   └── src
└── dagger.json

dagger.json:

{
  "name": "mymod",
  "sdk": "python",
  "source": "dagger",
  "engineVersion": "v0.9.9"
}

So the flatter style (used up til now), (more appropriate for publishing component/utility modules?) with the module implementation at same level of dagger.json would require someone to always specify --source dir as same dir where dagger.json lives. Though it seems that when the value of source is indeed the same dir as where dagger.json lives, "source": is left absent in the dagger.json, indicating that the "true default" for source relative to dagger.json is ".".

mkdir mymod
cd mymod
dagger init --name mymod --sdk python --source .

which creates

.
├── dagger.json
├── pyproject.toml
├── sdk
└── src

dagger.json:

{
  "name": "mymod",
  "sdk": "python",
  "engineVersion": "v0.9.9"
}

In summary, it's as if source has two "defaults". It defaults to a dagger/ subdir below the dagger.json in dagger init and dagger develop while it defaults to the same directory as the one containing the dagger.json in the dagger.json.

[wintry prism]

There are use cases for 1) "Daggerizing

[latent trellis]

Any ideas? This is kinda blocking 0.9.9 adoption for me and I can't really figure out the solution

[restive shore]

is this just me?

[deft rain]

this is kinda weird, i can't repro it 😢

[restive shore]

I removed the engine didn't work. I'll try removing the volume also

I still can't 😦

This is the command I'm trying to run
dagger init --sdk=go daggerverse/sonar --name=sonar

[deft rain]

are there any directories above you that container dagger.json?

[restive shore]

above? no

dagger init --sdk=go --name=sonar --source daggerverse/sonar

This gives me "module already exists" although it doesn't.. Let me traverse outside of my root to see if there's a dagger.json somewhere

There are dagger.jsons in repos parallel to my root. None of them have a "sonar" module though. Even if it did, should't matter to this dagger.json right?

this is how it is from "root".

[sharp zealot]

@restive shore could you show the contents of the dagger.json in the current directory?

[restive shore]

Yes, I came out but will post when I get back home

[deft rain]

so i don't think you should need the top-level dagger.json any more

or am i hugely misunderstanding your setup?

if you get a chance, if it's not private, having the whole contents in a git reproducer would really really help

i can't see to make it happen no matter what i'm trying

[deft rain]

ohhh i think i might get it - are you at the root, and expecting that --source daggerverse/sonar should create a dagger.json in daggerverse/sonar?
that's no longer true in this version of dagger, the dagger.json is always created in the current directory, while the dagger code is put in the value of the --source directory

[restive shore]

I am not particular about where the dagger.json is. I can't create a new module regardless of where the dagger.json is. I can give more details when I get back. On Friday I posted here an account of what I was seeing.

[restive shore]

The contents of the modules are kind of sensitive but I can try to do a simple reproduce.

[latent trellis]

Can we expect a 0.9.10 release today?

[sharp zealot]

I think @strong ingot and @deft rain are done for the day, so the current plan is probably to do it in their morning (UK).

We have a few people on vacation this side of the Atlantic, so not sure if we are able to release smoothly before Europe is back online.

@mossy hazel will know better than me

[mossy hazel]

yeah, Gerhard and Justin confirmed for tomorrow morning (UK time), meaning our night in the US

[latent trellis]

Thanks!

[sharp zealot]

How do I tag my repo for versioning multiple modules?

For example if I want to version https://daggerverse.dev/mod/github.com/shykes/daggerverse/hello, is this the right command?

git tag -a hello/v0.1.0

[latent trellis]

That's why I'm thinking about splitting up my daggerverse repo into repo per module 😄

[wintry prism]

yes, that's the correct format to use

We need to document this in a couple of places, but right now Daggerverse will display git tags of two forms

• v1.2.3
• foo/v1.2.3 where foo is the subpath under your repo root that holds your module

[sharp zealot]

Thanks. How do I know if it worked? I pushed the tag and re-crawled, but don't see it appear anywhere

[wintry prism]

Today, we don't modify the module data in the db on recrawl for the same git sha, but I've been thinking about changing that so we could pick up a tag.

[sharp zealot]

I didn't specify a git sha, though

If the crawler doesn't detect new tags, that's just a bug

[wintry prism]

right, any git ref resolves to a git sha: branch, tag, sha, etc, which is the canonical represenation path@sha

it's the way it was implemented originally, we could def add that

[sharp zealot]

You're saying the crawler only picks up new commits on the default branch, but not new tags (if I understood correctly)

So as a stopgap, if I push a new commit and re-publish, it will detect all new tags? Or only new tags that point to the latest commit?

[wintry prism]

the tags detected in the new crawl are scoped to the commit, so that workaround would not work.
we're going to be cracking open the crawl logic this week to get ready to recrawl all modules to update descriptions, so I can likely tell you more soon.

in summary: commit and push code changes, git tag and push tags, then publish is the only way to have a release tag on a module version in Daggerverse right now.

[sharp zealot]

Just tried that, it looks like the crawl is hanging: https://daggerverse.dev/mod/github.com/shykes/daggerverse/hello@9d5ee5168db3e77f9e88f36b89df7ab7760579ad

This is what I see

[warped canyon]

Another 0.9.9 question, I thought this was supported now but maybe something is still missing.

Use case: I have an application that contains a dagger module. My dagger.json is at the root of the application's repo.

I dagger install a module from daggerverse I want to use only from the cli.

I call dagger call -m <module name> <function> <args>

I get
Error: failed to get configured module: failed to get local root path: input: resolve: moduleSource: resolveContextPathFromCaller: cannot resolve non-local module source from caller

[warped canyon]

This could be because it's an annotated tag, but I thought that was supported now

[wintry prism]

@sharp zealot Daggerverse is still on 0.9.8, so that's the likely issue.

[hexed flume]

I don't know if it's a known issue, when I try this command that is coming from the Zenith docs:
dagger -m github.com/vikram-dagger/daggerverse/fileutils call tree --dir .

I get the following error:

Error: response from query: input: resolve: host: directory: host directory .: error from sender: open .Trash: operation not permitted

From the Pass a directory as argument section

I'm using dagger v0.9.9

[warped canyon]

Are you calling this from your home directory on mac?

[hexed flume]

héhé ... yes !

that the thing

[warped canyon]

Yeah I'm guessing it's a permissions issue

[hexed flume]

yes indeed

[sharp zealot]

@hexed flume same problem for me.

[sharp zealot]

How can I reproduce?

[warped canyon]

for me, i'm in a project with a dagger.json, and:

1. dagger install github.com/kpenfound/dagger-modules/ruff
2. dagger -m ruff functions
3. or dagger -m ruff call check --directory .

[sharp zealot]

I haven't actually tried it, but yes that is supposed to work

[warped canyon]

request, related to logging: it would be cool to have ways (controlled ways) to break out of the default logging in certain cases. My use case is to create groupings within github actions output with controls https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines but it doesn't work because of the extra dagger markup on the output https://github.com/kpenfound/demo-react-app/actions/runs/7878503851/job/21496902242#step:3:257

or even some functions that specifically do this, but built in

func startGroup(name string) {
    _, _ = os.Stdout.WriteString(fmt.Sprintf("::group::%s\n", name))
}

func endGroup() {
    _, _ = os.Stdout.WriteString("::endgroup::")
}

so I could call something like dag.GitHub().StartGroup("foo")

[sharp zealot]

See https://github.com/dagger/dagger/issues/6442

[restive shore]

hey, just getting back to this. Here's how my dagger.json (in root looks like )

{
  "name": "mymodule",
  "dependencies": [
    {
      "name": "docker",
      "source": "daggerverse/docker"
    },
    {
      "name": "java",
      "source": "daggerverse/java"
    },
    {
      "name": "oauth",
      "source": "daggerverse/oauth"
    }
  ],
  "engineVersion": "v0.9.9"
}

I am fully stuck. I can't even cd into a brand new subdirectory and run dagger init --sdk=go --name=sonar

[sharp zealot]

I want to flag something in the new Go DX for optional arguments. I don't like encoding my default values in comments, because they are no longer in my code. I can't use them if I need any value more trivial than a string. I can't use them in any way when calling functions intra-module. At this point I think we should remove +default and keep only +optional. It's not a one-way door: we can add it later. But if we keep it now, we will fragment how devs manage their default values.

[sharp zealot]

https://github.com/dagger/dagger/issues/6651

[void widget]

i see it now, was it just a congestion ?

[sharp zealot]

Possibly. In any case, the tag is still not there

[sharp zealot]

Are doc: struct tags still a thing in the Go SDK?

[thorn moat]

don't think so, it's just comments on args now

[sharp zealot]

That's what I thought, found an old leftover in one of my modules, thanks

It's spring cleaning time!

Transitioning away from Optional on several modules is very painful. Error messages are useless.

I'm basically bruteforcing it

I take it back, there is useful information in the error message, it's just drowned in a lot of words:

Error: failed to generate code: input: resolve: moduleSource: withDependencies: resolveFromCaller: asModule: failed to create module: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name supergit --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stderr:
Error: generate code: template: module.go.tmpl:74:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to parse method Tags: failed to parse type reference: invalid type: invalid type

--> method Tags

Ooooh that one is in a dependency

Another clue:

--module-name supergit

(The module I'm debugging is not supergit, but depends on it)

[thorn moat]

yeah I ended up just doing one big grep and went through them in my editor.

our errors could definitely use some love in general; they're easy to ignore but in reality they probably account for most of the DX even for experienced users.

I tried to tidy up a common one (process exiting nonzero) but only make incremental progress, had to back out parts of it

[thorn moat]

how did the TUI look in this case? hopefully it would have also been grouped under an ID that mentions supergit, but it might not be obvious

[sharp zealot]

Yeah my brain already tunes out all module-related activity, there's a lot of depth and it moves too fast.

Looking at it more closely, I can't spot that particular information:

$ dagger functions
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.2s
✘ ModuleSource.asModule: Module! 0.8s
  ✘ Module.withSource(
      source: ✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
    ): Module! 0.8s
    ✘ ModuleSource.asModule: Module! 0.8s
      ✘ Module.withSource(
          source: ✔ ModuleSource.withName(name: ""): ModuleSource! 0.0s
        ): Module! 0.8s
        ✔ Container.import(
            source: ✔ Directory.file(path: "go-module-sdk-image.tar"): File! 0.0s
          ): Container! 0.3s
        ✘ Container.directory(path: "/src"): Directory! 0.4s
          ✘ exec /usr/local/bin/codegen --module-context /src --module-name supergit --propagate-logs=true --introspection-json-path /schema.json 0.4s
          ┃ Error: generate code: template: module.go.tmpl:74:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: faile
          ┃ ethod Tags: failed to parse type reference: invalid type: invalid type
          ✘ generating go module: supergit 0.2s
          ┃ writing dagger.gen.go
          ┃ writing go.mod
          ┃ writing go.sum
          ┃ creating directory querybuilder
          ┃ writing querybuilder/marshal.go
          ┃ writing querybuilder/querybuilder.go
          ┃ needs another pass...

Error: input: resolve: moduleSource: resolveFromCaller: asModule: failed to create module: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name supergit --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stderr:
Error: generate code: template: module.go.tmpl:74:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to parse method Tags: failed to parse type reference: invalid type: invalid type

I guess it is there in the call tree:

      ✘ exec /usr/local/bin/codegen --module-context /src --module-name supergit --propagate-logs=true --introspection-json-path /schema.json 0.4s

[thorn moat]

yeah not great

[sharp zealot]

I just ran into a missing Host call while porting one of my modules...

[last fox]

Sorry, this is probably answered elsewhere in here. But I got hit with this particular error moving some things over and wasn't able to fully resolve it with how our directory is structured: https://github.com/depot/daggerverse/tree/main/depot.

[sharp zealot]

It's probably item number one on this list of breaking changes: #daggernauts message

(change in how optional arguments are handled in Go)

[last fox]

Yup, looks like we missed this one when upgrading. Thanks!

[sharp zealot]

sorry about the breakage. We wanted to pull off the bandaid before getting more testers on board this week

[hexed flume]

For me the issue occur only when you run the command from the user HOME directory.
It works from within a subdirectory: ~/foo

[sharp zealot]

yup, same

[hexed flume]

When I'm trying to run ./dagger version from the section Export the build directory, I'm getting an error.

Workdir: /Users/slumbering/workdir/dagger-zenith/my-dagger-build
Error: zsh: exec format error: ./dagger

[deft rain]

hm, this looks like you've built a binary that's not native to the system you're on
if you run file ./dagger do you get something that's expected for your host system? (e.g. matching architecture and OS)

[hexed flume]

$ file ./dagger
dagger: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, Go BuildID=Agm8PU1NNZMQ9mVhifbR/BqFmEUGKL2oDo7K6OYdP/4HA2Xlm31NVwMkyQb7R7/88MNXVWHmgpBnTClQHfP, with debug_info, not stripped

[deft rain]

are you on linux? or mac?

[hexed flume]

mac

apple silicon M1

[hexed flume]

docs link: https://docs.dagger.io/zenith/quickstart/853930/directories

[hexed flume]

I've created an issue:
https://github.com/dagger/dagger/issues/6653
cc @bleak nest @warped canyon @deft rain

[latent trellis]

This is still happening:

fatal error: concurrent map iteration and map write

Every second CI run in my daggerverse repo fails due to it :/

cc @thorn moat

[deft rain]

do you have a stack trace?

[latent trellis]

It's a known issue: https://github.com/vito/progrock/issues/12

[latent trellis]

Ok, now that 0.9.10 is out, I can actually start using the new source thing.

Here is the next issue though:

I have a repo, dagger.json in the root, source is set to "ci". How do I get the repo root in my ci code? dag.CurrentModule().Source() seems to return the contents of the ci directory.

I guess I need to manually pass it to my functions, right? :/

[deft rain]

ahh i hit this as well - i think this is what you need to do now.
i think the idea is that you have modules that are composeable, but it is a bit of a pain to constantly pass "." in, it makes me want to wrap the cli in a shell script, which i know we want to avoid

[latent trellis]

Well, it marks an end of an era: I'm gonna have to stop using the Dagger CLI and wrap it in a shitty task runner, so that I don't have to pass the same arguments to it all the time. It kinda makes me sad.

But here is something that makes it so much worse: now that dag.Host() is gone, I have no way to exclude directories.

[deft rain]

yeah, i'm slightly inclined to work out a way to bring it back if we can - while the model works well for daggerverse-type setups, it's really not great for a specific case where i only really want to run the dagger module in the context of one repo

i'm guessing that's your use case?

[latent trellis]

yeah

dagger call --directory . ci

It uploads everything to buildkit. How do I tell it to ignore a bunch of directories with a lot of files that I don't want it to upload?

(It adds roughly 1-1,5 minutes to the build time)

[deft rain]

we discussed previously here: https://github.com/dagger/dagger/issues/6155
i don't have any particularly good ideas beyond what's in there though

[latent trellis]

I don't think I can update to 0.9.10 this way

[deft rain]

heard, the reasons you're giving are good - we will be trying to fix them, i'm literally hitting the same things as you in trying to modularize our own ci, so we'll need to get them ourselves ❤️

[ocean escarp]

since version 0.9.9 I can no longer use a custom runtime from a git repo,
I always get this error:

Error: returned error 422 Unprocessable Entity: {"errors":[{"message":"internal system error"}],"data":null}

[deft rain]

can you share your dagger engine logs?

[ocean escarp]

I always use my own sdk runtime from github like this, it worked fine in version 0.9.8 of dagger.

{
  "name": "demo",
  "sdk": "github.com/fluentci-io/daggerverse/deno-sdk@main"
}

7: moduleSource DONE
7: moduleSource DONE

6: resolveFromCaller
6: ...

12: dagger functions ERROR: returned error 422 Unprocessable Entity: {"errors":[{"message":"internal system error"}],"data":null}
12: [0.00s] Error: returned error 422 Unprocessable Entity: {"errors":[{"message":"internal system error"}],"data":null}
12: dagger functions ERROR: returned error 422 Unprocessable Entity: {"errors":[{"message":"internal system error"}],"data":null}

6: resolveFromCaller

✔ defaultPlatform: Platform! 0.0s
✔ moduleSource(refString: "."): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
● ModuleSource.resolveFromCaller: ModuleSource! 1.5s
  ✔ moduleSource(refString: "github.com/fluentci-io/daggerverse/deno-sdk@main"): ModuleSource! 1.5s
    ✔ git(url: "https://github.com/fluentci-io/daggerverse"): GitRepository! 0.0s
    ✔ GitRepository.commit(id: "main"): GitRef! 0.0s
    ✔ git://github.com/fluentci-io/daggerverse#main 1.5s
    ✔ GitRef.tree: Directory! 0.0s
✘ dagger functions 0.0s

Error: returned error 422 Unprocessable Entity: {"errors":[{"message":"internal system error"}],"data":null}
[ble: exit 1]

[deft rain]

oh i think this might be https://github.com/dagger/dagger/pull/6645
any chance you could try v0.9.10?

[ocean escarp]

I have the error in v0.9.9 then I updated to v0.9.10, but there is still the error

the error only appears if I do dagger functions -m "." or dagger functions

[deft rain]

oh interesting, so doing dagger call works?

[ocean escarp]

same error with dagger call, it only works if I load the module from github with -m

[dense canyon]

👋 is the -m flag relevant in dagger install? Just got bitten by trying to do dagger install -m relative_dir and found that dagger install requires a module name which got fixed by running dagger install $relative_dir. Just wondering if -m is of any use at all in install now

[restive shore]

it's used if you want to install a module (X) into another module (Y). dagger install -m Y X. I think it has the ability to do that from anywhere within the root of your module. I also think Eric wanted to change that behavior to show the source/destination to be more explicit. Something like dagger install --source X --destination Y

-m is where you pick the module you want to work on.

More context here - https://github.com/dagger/dagger/pull/6533

And https://github.com/dagger/dagger/issues/6605

[dense canyon]

👍 got it, makes perfect sense. I got confused since the -m flag was used differently in the past. Thx!

[restive shore]

Yes! So did I.

[sharp zealot]

yeah -m should probably be a global flag

[thorn moat]

50% hit rate seems mighty high, we haven't seen this in our ci at all. Wonder what we're doing different. 🤔 There's definitely a deeper issue at play but maybe it's time to add a lock + warning for dropped logs until we can find it. Better than a full panic at least.

[latent trellis]

All I know is it occurs in basically every PR I open in my daggerverse repo. One thing I noticed is that it only affects my helm and helm-docs modules.

[warped canyon]

How can I reproduce?

[deft rain]

potential idea - how would we feel about commands like dagger call/dagger develop defaulting to the dagger.json in the closest parent?
i find myself going into lots of sub-directories, and expecting to be able to call the module defined at the top of my git repo (sort of like how i do with git commands)

[sharp zealot]

yeah agreed

Dagger module in monorepo

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/6666

[ashen ember]

👋🏼 this is my first pass of a Dagger module and the reason I've been asking questions. Any feedback is appreciated! 🙂 https://github.com/chainloop-dev/chainloop/pull/502

[sharp zealot]

Problem: it looks like there's no way to query the ID of an object from the CLI?

Is it a quick cosmetic fix, or something deeper?

[kind carbon]

Cosmetic

You can use sync as it doesn't hide it yet, if your object has it.

[sharp zealot]

Nice! 🙂

Ok next problem... I need to load a directory by ID from the CLI...

😭

[thorn moat]

should be possible with a regular string flag + dag.LoadDirectoryFromID() at least

[sharp zealot]

Found a workaround which involves writing a helper module

By the way: those LoadXFromID top-level functions are hella ugly

Has anyone seen this error?

error: parse selections: parse field "loadDirectoryFromID": init arg "id" value as dagql.ID[*github.com/dagger/dagger/core.Directory] (DirectoryID!) using dagql.ID[*github.com/dagger/dagger/core.Directory]: proto: cannot parse invalid wire-format data

[thorn moat]

seems like the ID string payload is malformed somehow. maybe there's a linebreak splitting it

[sharp zealot]

yeah but it's valid base64

[wintry prism]

@thorn moat any chance you can share your grep invocations you use to update mods? err detect need for updates

[thorn moat]

i only remember grep Optional really

[wintry prism]

lol

[thorn moat]

i guess root-for too?

[wintry prism]

yep. Host().Directory(".") variants too eh

[sharp zealot]

I also had to grep for Opt( and \.GetOr(

Is there a way to introspect the current module? dag.CurrentModule() doesn't return a Module type. Is there a way to get one?

(I'm thinking no, but you never know)

[thorn moat]

don't think so. CurrentModule technically has a hidden Module field in core/ but it's probably hidden for a good reason (i.e. bootstrapping related)

[sharp zealot]

I guess that's part of the puzzle to solve if we want self-call in the future right?

[thorn moat]

possibly, I think some puzzle pieces were added recently as part of another PR too

[sharp zealot]

I have a question on the new call convention for optional arguments, specifically when the caller is a Go function.

is it expected that I can still pass optional arguments as regular arguments? eg. dag.Mod().Func("hello world")?

[thorn moat]

nope, that seems like it interpreted it as required. one gotcha I ran into is that // +default doesn't imply // +optional - you have to set both

[normal blaze]

Hey folks, I have been creating a module to fetch secrets from HashiCorp Vault, I just changed the interface for a couple of methods from (string,error) to (*Secret, error) and now when you see the generated code it is only showing a single output parameter of ?*Secret

https://daggerverse.dev/mod/github.com/jumppad-labs/daggerverse/vault@4f140335b8c62d8e04a2fddcf559341d988b1832

Interestingly I no longer need to pass the context to the function either

[wintry prism]

Hi @normal blaze 👋 I know work was done to make context and error not required where possible, so that's expected.

[normal blaze]

Ah, I thought that might be the case, my only problem is that my functions are not what I would call idiomatic go with that type error tuple

[thorn moat]

This is actually just how objects vs scalar values work in Dagger. When you return an object the client always has a "lazy" view of the call. This is what allows you to chain long sequences without constant err checks. So it's the same as e.g. container.withexec

[normal blaze]

Ok thanks, do you have an example of a pattern where I can return error info to give detailed info to the consumer of my module, or should I assume nil is an error and halt execution

I know the error will be printed in the logs but actually reading the output is not easy

[thorn moat]

Not sure what you mean by assume nil is an error; the client will always get a *Secret. I can see how this case might be a bit tricky to frame in a lazy manner though. The expectation is that you can just pass that *Secret value around and the module will be called when the secret value is actually needed

[thorn moat]

This should be better with the new TUI but yeah, module logging + errors definitely needs more attention (DX wise)

[normal blaze]

But the problem is when the get secret operation fails

for example in my main code I might have something like this....

[upbeat herald]

Is there a way to publish a module on daggerverse and see the git tag instead of the commit sha?

Because I'm working on a module named daggerverse-cockpit that aim to help modules maintainers to simplify the management of their module
https://daggerverse.dev/mod/github.com/quartz-technology/daggerverse/daggerverse-cockpit@f6dc7141a330a9cb62ed80198ac86380c3fd7664

I would like to create a GitHub Action that publish all modules when a new tag is release.
I think that would be call on the daggerverse if we had a way to see the tag itself

[wintry prism]

daggerverse-cockpit :: Daggerverse

[ashen ember]

👋🏼 is there any way to get the dagger cloud run UUID from inside a module?

[sharp zealot]

I see you figured out module versioning @upbeat herald 😄

[sharp zealot]

If you want a nice benchmark of running a function which calls a lot of other functions, from several other modules, try this:

time dagger -m github.com/shykes/daggerverse/dagger call engine versions

Runtime is 20s on a hot cache for me

[upbeat herald]

Yep haha! Sorry for the spam on daggerverse

[sharp zealot]

Who wants to try something cool 🙂

It requires an OpenAI key

dagger -m github.com/shykes/daggerverse/daggy call \
 do --token env:OPENAI_API_KEY --prompt "fetch the git repo github.com/dagger/dagger and list its contents. Then do the same thing for github.com/shykes/daggerverse please"

Powered by gptscript @obtuse lion 🙂

In case you want to try 👆

[obtuse lion]

Do i need to be running a dev version of dagger. I'm getting ┃ Stdout: ┃ json: error calling MarshalJSON for type *main.File: input: resolve: container: from: withExec: withEnvVariable: withExec: withWorkdir: withDirectory: withMountedCache: withExec: withMountedDirectory: withMountedCache: w ┃ xec: file: process "go build -o ./bin/dagger -ldflags -s -w -X github.com/dagger/dagger/engine.Version= -X github.com/dagger/dagger/engine.EngineImageRepo=registry.dagger.io/engine ./cmd/dagger" did not complete successf ┃ : exit code: 1 ┃ ┃

[sharp zealot]

that's just my module doing something stupid. It's the build of the dagger CLI that fails, of all things

[sharp zealot]

do you have a dagger cloud run url by chance?

or more hints from the logs?

also what version of dagger?

[obtuse lion]

i'm not logged into the cloud, but I can copy the whole logs. v0.9.10

Logs

[sharp zealot]

It's that same error again @thorn moat 😭

(pretend that was a silent notification, sorry!)

@obtuse lion oops I had forgotten to push my fix... Could you try again?

[obtuse lion]

yep

[sharp zealot]

You can also open a debug shell with dagger -m github.com/shykes/daggerverse/daggy call container terminal

You can pass your openapi token to the container:

dagger -m github.com/shykes/daggerverse/daggy call \
  container --token env:OPENAI_API_KEY \
  terminal

[obtuse lion]

holy crap it worked (at the amazing speed of AI)

[sharp zealot]

The Amazing Speed of AI ™️

[obtuse lion]

we'll just say there's room for improvement.

[sharp zealot]

I just realized Container.Terminal doesn't support ExperimentalPrivilegedNesting 😭 so I can't have a dagger call debug that just pops a shell (because I need dagger-in-dagger)

[thorn moat]

also ran into this... I've been wondering if we should move a bunch of those withExec opts into their own with things, like withExperimentalPrivilegedNesting

[sharp zealot]

Or... maybe we're ready to enable it by default? That was always the plan, we just added a safety period

[thorn moat]

mostly a security question right?

[sharp zealot]

Yeah

Just to give ourselves time to find red flags

There are still little things, like shared access to cache volumes

[sharp zealot]

anything we could do to improve caching of module load? When you use 3-4 dependencies it's almost 10 seconds to load everything on a hot cache

I wonder if it would be possible for dagql to start caching some calls itself ? 😇

[agile saddle]

First of all, great job to the team on Zenith. It’s starting to look really polished.

However I saw something in both Kyle’s dagger 101 demo and the typescript one that concerned me. It almost looked like on the video that changing the zenith module caused a cache invalidation causing the build of the base container to be redone in both functions for the dagger module that was called. With non Zenith dagger, my experience has been that this wouldn’t happen? Is there any major differences in how caching happens in Zenith?

[thorn moat]

PSA: we're in https://discord.com/channels/707636530424053791/911305510882513037 for this week's Zenith office hour!

[wintry prism]

updating my trivy module to ditch Optional and using the json // +default="latest" style.
Looks like I'll need to unwrap the json myself, or will that be taken care of with
https://github.com/dagger/dagger/pull/6683
I'll leave off with the quotes for a second...since they're being left in until we merge/release this

  ┃ input: resolve: container: from: invalid reference format
  ┃
  ✘ Container.from(address: "aquasec/trivy:\"latest\""): Container! 0.0s

will do this for now

// +optional
// +default=latest
trivyImageTag string,

[deft rain]

Yup you need to leave the quotes out until that one is merged and released sadly 😭

[dense canyon]

👋 just checking if there might be a valid use-case to remove the restriction around installing modules from an absolute path in your system.

I have the following scenario:

I'm developing a module which is located in ~/Projects/daggerverse/my_module

At the same time, while I'm developing my module I'm dogfooding it in a project that lives in ~/Projects/myproject. The problem is that I can't dagger install the local version of my_module neither by relative nor absolute paths so I kind of need to make symlinks to get this to work.

Thoughts?

[dense canyon]

ok, seems like using symbolic links also doesn't work. Had to manually copy the module to ~/Projects/myproject/my_module so myproject can import it

[wintry prism]

ok, seems like using symbolic links also

[wintry prism]

Excited for the new description comment/docstring at top of your module code file 🙂
https://github.com/jpadams/daggerverse/commit/5e198f2327b458fe9b313ecbbe8b99ab6d7b198b

// Finds vulnerabilities from container image ref or Dagger Container

package main

previous version

[ashen ember]

Awesome, thanks for the heads up. I've also updated my module

on another note. I've noticed that the link to source is broken. It's missing a / after the commit sha

https://github.com/chainloop-dev/chainloop/tree/438621bd4bae0deaca8efa6cecf4fcd615ee5301extras/dagger => https://github.com/chainloop-dev/chainloop/tree/438621bd4bae0deaca8efa6cecf4fcd615ee5301/extras/dagger

[wintry prism]

Thanks @ashen ember I understand that will be fixed when we get an upstream fix in the 0.9.11 release next week.
Definitely on our radar.

[ashen ember]

awesome!

[sharp zealot]

Trying now: running dagger modules from a plane 😅

[sharp zealot]

OK the new UX with dagger install && dagger -m <shortname> is REALLY good. Modules load much faster!

The improvement is even more noticeable on a plane

Thanks @warped canyon for the fix, and @deft rain for getting it in

@warped canyon @bleak nest we should avoid constructor arguments in docs examples for now. Two reasons:

1. it's one more thing to grok in the beginning, when there is already a lot to digest
2. constructor arguments are not displayed by --help. So they are hard to discover.

[bleak nest]

should we change this quickstart example then? it uses a constructor argument https://docs.dagger.io/zenith/quickstart/853930/directories#call-a-builder-function (I mean, change the module to use regular fn args, not switch to a different example)

[honest hearth]

awesome!

[bleak nest]

btw it looks like they do appear in --help

$ dagger -m github.com/kpenfound/dagger-modules/golang call --help
Call a module function and print the result.

...

Flags:
      --ctr Container    
      --focus            Only show output for focused commands (default true)
      --json             Present result as JSON
  -m, --mod string       Path to dagger.json config file for the module or a directory containing that file.
                         Either local path (e.g. "/path/to/some/dir") or a github repo (e.g.
                         "github.com/dagger/dagger/path/to/some/subdir")
  -o, --output string    Path in the host to save the result to
      --proj Directory

[sharp zealot]

oh, I tried dagger -m foo --help. Didn't think to add call

[sharp zealot]

Yes I think so. Will discuss with @warped canyon in person in Austin 🙂

[bleak nest]

OK. I have 2 guides in open PRs which also use constructor args, I'll change those as well

[rocky harbor]

Is there a way to introspect the current

[rocky harbor]

This should be fixed here: https://github.com/dagger/dagger/pull/6693

[deft rain]

Attention Zenith devs! I'm currently in the process of releasing v0.9.11 (the new SDKs should drop a little later today), and just want to flag a potentially breaking change for any Go module devs ⚠️

If you're using // +default pragmas for string values, then you'll need to update to start quoting those strings. This is a bit annoying for sure, but it's moving towards being a bit more explicit in declaring these as strings.

For example, before in <=v0.9.10:

func (m *MyModule) Foo(
  s string, // +default=foo
) error {

And after, in >=v0.9.11:

func (m *MyModule) Foo(
  s string, // +default="foo"
) error {

We'll be updating our docs ASAP, to reflect the new pattern, but just wanted to share an advanced warning!

[restive shore]

I was just testing this. How does the error message look to you? It looks like it's spitting out the entire generated go code for me.

[deft rain]

hm, it shouldn't spit out the whole code 👀
it should just print something like default value "foo" must be valid JSON: invalid character 'o' in literal false (expecting 'a')

[restive shore]

hmm this is what I'm seeing as soon as I update to 0.9.11. I assumed it was because of the defaults but wonder if it's something else

Error: failed to get module SDK: input: resolve: moduleSource: resolveFromCaller: asModule: failed to create module: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name docker --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stderr:
[omitting 67945 bytes]... environment variable value.
func (r *EnvVariable) Value(ctx context.Context) (string, error) {

That's from a dagger develop

[deft rain]

well that's unhelpful, and looks like something else

hm, there should be an error somewhere in that

[restive shore]

It's hard to find. I went all the way up for this

[deft rain]

that looks right, not sure why it errors - i'm guessing there's something in the logs you can't share the whole ones?

[restive shore]

That's pretty much it. It's literally the gen code that's output as stderr.

let me strip the last bit..

type TypeDefKind string

func (TypeDefKind) IsEnum() {}

const (
        // A boolean value.
        BooleanKind TypeDefKind = "BOOLEAN_KIND"

        // A graphql input type, used only when representing the core API via TypeDefs.
        InputKind TypeDefKind = "INPUT_KIND"

        // An integer value.
        IntegerKind TypeDefKind = "INTEGER_KIND"

        // A named type of functions that can be matched+implemented by other objects+interfaces.
//
// Always paired with an InterfaceTypeDef.
        InterfaceKind TypeDefKind = "INTERFACE_KIND"

        // A list of values all having the same type.
//
// Always paired with a ListTypeDef.
        ListKind TypeDefKind = "LIST_KIND"

        // A named type defined in the GraphQL schema, with fields and functions.
//
// Always paired with an ObjectTypeDef.
        ObjectKind TypeDefKind = "OBJECT_KIND"

        // A string value.
        StringKind TypeDefKind = "STRING_KIND"

        // A special kind used to signify that no value is returned.
//
// This is used for functions that have no return value. The outer TypeDef specifying this Kind is always Optional, as the Void is never actually represented.
        VoidKind TypeDefKind = "VOID_KIND"

)

[deft rain]

couple questions - what's your dir structure? and are you using interfaces? those are just wild guesses, but without a reproducer i kinda have no idea how to dig into that

[restive shore]

yeah understood. No interfaces. Dir structure is the same as when I was working with you for the go.work issue. Main project has a go.mod and go.work. Daggerverse is under ./daggerverse. Each module is a folder

❯ tree -d .
.
├── ci
├── cmd
│   └── cloak
├── daggerverse
│   ├── docker
│   │   └── querybuilder
│   ├── java
│   │   ├── querybuilder
│   │   └── resources
│   ├── myapp
│   │   └── querybuilder
│   ├── oauth
│   │   └── querybuilder
│   └── sonar
│       └── querybuilder
├── dist
├── examples
├── internal
├── pkg
└── reports

This may clarify some things.

[restive shore]

I was able to narrow it down. Same issue as before. Different behavior. If I rename go.work and go.mod it works.

[deft rain]

Agh what, I had a fix for this

[restive shore]

I know! Was trying to test that

[deft rain]

So I guess the issues we were seeing were subtly different, since mine is definitely resolved (there are some new tests as well)

[restive shore]

FWIW, my main go project has a dagger sdk dependency in go.mod. I wonder if that causes some conflict. I have a require ( dagger.io/dagger v0.9.0 ...

[restive shore]

hmm, IDK what's up but even a simple go module with just main.go in the root makes dagger throw that error.

0.9.11 error when go.mod is present in a top level dir

[restive shore]

I may have found a regression on 0.9.11. When I do a dagger develop in an existing module folder it seems to be always creating a dagger/dagger.gen.go. This isn't happening in 0.9.10. My source is set to source='.'

[void widget]

I am getting a somewhat similar issue. Could not workaround with the renaming of go.work / go.mod, tracking it down 😇

Error: failed to get module SDK: input: resolve: moduleSource: resolveFromCaller: asModule: failed to create module: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: failed to create module: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module-context /src --module-name go --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

Stderr:
Error: generate code: template: alias.go.tmpl:56:3: executing "_dagger.gen.go/alias.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to generate type def code for Go: failed to convert method Gotestsum to function def: default value "testname" must be valid JSON: invalid character 'e' in literal true (expecting 'r')

[void widget]

Thanks for the insights @thorn moat, the issue comes from the default field not being double quoted in a dependent module 🙏 PR on the way on your repo

[sharp zealot]

that's what I've been stuck on since yesterday

aaand that's the answer. thank you!!!

[void widget]

Agreed, same issue on my side, with the fact that I get this error when I specify a --source (to avoid creating the dagger subdir):

dagger develop --sdk=go --source=.
...
Error: cannot update module source path that has already been set to "daggerverse/dag"

[void widget]

Another weird bug encountered, can't seem to find the root cause atm.

I am able to call @upbeat herald 's module with the latest engine version:

dagger -m github.com/quartz-technology/daggerverse/postgres functions
Name                 Description
...
with-database-name   WithDatabaseName sets the name of the database that will be created on start.
with-init-script     WithInitScript adds a script to execute when the database is started.
with-port            WithPort configs an exposed port on the container.
with-version         WithVersion sets the version of postgresql to pull from the registry.

However, the moment I install it as a submodule, I have some weird bug:

dagger install github.com/quartz-technology/daggerverse/postgres@aecc35bce44a4f6447542bc96de102e3b8fed2ee --mod .

puis dagger functions:

# test.com/dagger
dagger/dagger.gen.go:5140:34: cannot use r.Client (value of type func() *Container) as "github.com/Khan/genqlient/graphql".Client value in argument to q.Execute: func() *Container does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
dagger/dagger.gen.go:5145:20: field and method with the same name Client
        dagger/dagger.gen.go:5114:2: other declaration of Client
dagger/dagger.gen.go:5150:11: cannot use r.Client (value of type func() *Container) as "github.com/Khan/genqlient/graphql".Client value in struct literal: func() *Container does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
dagger/dagger.gen.go:5159:11: cannot use r.Client (value of type func() *Container) as "github.com/Khan/genqlient/graphql".Client value in struct literal: func() *Container does not implement "github.com/Khan/genqlient/graphql".Client (missing method MakeRequest)
dagger/dagger.gen.go:5272:11: too many errors

The module's code doesn't seem to have one of the breaking changes due to the engine version bump (0.9.9 to 0.9.11), but I might miss some

[dim falcon]

Hello. I have a problem publishing my module: "github.com/puzzle/dagger-module-git-actions/git-actions"
I got the following error.

Error: generate code: template: module.go.tmpl:74:3: executing "_dagger.gen.go/module.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to parse method Push: failed to parse type reference: invalid type: invalid type

It is because of the Optional type of the prBranch parameter of the Push function: prBranch Optional[string],
When I remove Optional, it works.
Will Optional be supported?
Thanks for your help!

[void widget]

Hey 👋
The latest version supports optional fields, but it is not Optional[string] anymore (one of the recent breaking change).
See this example for an alternative: https://github.com/vito/daggerverse/blob/main/go/main.go#L16. It is now a comment on your function args.

Can you please change that and tell us if it fixed your issue ? 🙏

[upbeat herald]

I need to double check my module but last time it worked

[void widget]

it works when called directly, but fails when I install it as a dependency. Interested to know if you have the same behavior 🙏

[dim falcon]

Thank you!
It works but I cannot use isSet or GetOr methods on the param object any more.

[ashen ember]

I wrote my first actual Zenith pipeline using multiple modules [1] and it was a great experience. Good job everyone 🚀

[1] https://github.com/chainloop-dev/integration-demo/blob/main/chainloop-demo/dagger/src/main.go

[deft rain]

agh this is a unintended side-effect of https://github.com/dagger/dagger/pull/6680
Client and Query are now reserved fields on all imported fields, so this errors out

this is kind of very annoying, and it would be nice if this wasn't an issue - I'll try and take a play with this later today, and see if I can hide them again

see some of the complexity in https://github.com/dagger/dagger/pull/6680#discussion_r1492500523

[sonic vessel]

Hi, I try creating the Elixir SDK for Zenith at https://github.com/wingyplus/daggerverse/tree/main/elixir-sdk and found the error like this

    ✔ generatedCode(
        code: ✔ Container.directory(path: "/src"): Directory! 0.0s
      ): GeneratedCode! 0.0s
    ✔ GeneratedCode.withVCSGeneratedPaths(paths: ["sdk/**"]): GeneratedCode! 0.0s
    ✔ GeneratedCode.withVCSIgnoredPaths(paths: ["sdk"]): GeneratedCode! 0.0s
    ✘ Directory.diff(
        other: ✔ Container.directory(path: "/src"): Directory! 0.0s
      ): Directory! 0.0s

Error: failed to generate code: input: resolve: moduleSource: withName: withSDK: withSourceSubpath: resolveFromCaller: asModule: failed to create module: failed to update codegen and runtime: failed to diff generated code: TODO: cannot diff with different relative paths: "/" != "/src"

I belive I do something wrong with dag.GenerateCode but the error message didn't guide me what's wrong. 😦

[deft rain]

would appreciate any core team engineer thoughts on https://github.com/dagger/dagger/issues/6706
(tl;dr go versioning drives jed insane again)

[sharp zealot]

@bleak nest FYI https://github.com/dagger/dagger/pull/6707

Sorry for the delay.

Unfinished parts are marked with FIXME

[bleak nest]

Thank you, will work on this tomorrow!

[void widget]

Mmh, thanks 🙏 Any idea on why this module is touched and not another one ? Is it because of the go versions ? (testing that out atm)

[deft rain]

It's because there's a method defined called "Client"

If you call it something else, the issue goes away, but I have an idea for how to fix this

[void widget]

https://tenor.com/view/dumb-gif-4272870

Thanks, I shall read more carefully next time 🙏 😍

[kind carbon]

Elixir runtime module

[ocean escarp]

do we have a list of argument names that can't be used in functions? like --output

[sharp zealot]

you can dagger call FUNC --help

sorry I misunderstood. No.. I don't think we have that.

I would love to have that list too 🙂

Mostly it's the list of types in the core Dagger API

[sharp zealot]

Is anyone else getting a trailing % at the end of dagger call output?

$ dagger -m github.com/shykes/daggerverse/hello call hello
hello, world!%
$

[thorn moat]

this is possibly your shell seeing that there isn't a trailing linebreak; do you get the same from echo -n hello?

we debated the correct behavior here and figured the most correct thing is just relay whatever value came out, i.e. don't add an extra linebreak

[mossy hazel]

yep, that's the shell, there is no \n on the default message https://github.com/shykes/daggerverse/blob/main/hello/main.go#L40

[sharp zealot]

Makes sense. First time I see a shell doing that.

Yeah I see the dilemma

A small downstream consequence of conflating CLI command and function call

Maybe take a page from echo's book, and make it dagger call [-n] ? 🙂

Seems harmful enough to have the CLI add a trailing \n, in an opt-out way. If you're doing serious enough composition that it becomes a problem, you should be doing it in code anyway, making the problem moot

Otherwise, the risk is that devs start adding \n to their function's return values to make it look better from the CLI, making the problem worse

[thorn moat]

hmm actually this might be fixed by https://github.com/dagger/dagger/pull/6704 - if not I'll make it so. it was handled properly by the previous TUI: https://github.com/dagger/dagger/pull/6520#issuecomment-1916112176

[sharp zealot]

FYI https://github.com/dagger/dagger/issues/6710

What does this error mean, when running dev engine?

WARNING: failed to resolve image; falling back to leftover engine error="GET https://registry.dagger.io/v2/engine/manifests/4b825dc642cb6eb9a060e54bf8d69288fbee49
04: MANIFEST_UNKNOWN: manifest unknown"

As far as I know, I'm on latest main, with vanilla ./hack/dev

[thorn moat]

did a separate PR since the other one is already a bit of a grab-bag: https://github.com/dagger/dagger/pull/6711

[sharp zealot]

thanks!

FYI https://github.com/dagger/dagger/issues/6712

What's the trick again for fixing IDE auto-complete in a repo that has multiple golang modules? aka my daggerverse repo 🙂

[restive shore]

go work init, go work use [module]

be mindful of the versioning - https://discord.com/channels/707636530424053791/1209532081697325147

[sharp zealot]

Thank you!

[restive shore]

Does struct embedding work with modules? I remember there being some conversation about it but I can't find it. I am trying to get it to work but I keep getting a nil pointer.

[sharp zealot]

I'm getting this error in my IDE after adding all my daggerverse modules to root go.work

[restive shore]

you also have to go into each go.mod in your modules and change module name to something other than main. Dagger generates modules named "main" by default

[sharp zealot]

well that sucks

[restive shore]

Which I think can be changed to generate module name instead of main

[sharp zealot]

So out of the box, developing on a daggerverse-style repo in Go is broken?

What's weird is that I remember having functioning auto-complete in this repo at some point in the past, and I never had to rename package name

did something change? Or has it always been like this?

[restive shore]

IIRC it's always been like that

[sharp zealot]

Mmm looking into my repo, some of my older modules have a name other than main Either I manually changed them and don't remember, or dagger init behavior changed at some point

[restive shore]

It's possible I don't recall how modules worked some time before. So much has changed 😄

[deft rain]

Yeah this is on my list to see if I can fix

[restive shore]

found the culprit - https://github.com/dagger/dagger/issues/5975

[deft rain]

If you've had multiple main packages in the workspace it's never worked I don't think

[restive shore]

so @sharp zealot you are right. It used to be named with the module name.

[deft rain]

I think we need to go back to that annoyingly cc @rocky harbor

[sharp zealot]

Was the motivation for the change, that some module names are reserved by Go?

maybe prefix them with dagger_ or something?

(note: Go is annoying)

[deft rain]

Ahhh don't even 😭 I spent ages today trying to track down a go tooling library just throwing away the error that would have told me what the issue was

[sharp zealot]

Your error was not worthy

Yes I fixed it! Autocomplete is back in my life

Thank you @restive shore

[deft rain]

I think using the module name shouuuuld be fine here? Although, maybe we need to also consider the path (like foo/name and bar/name both being modules)

[sharp zealot]

For the record here's what I ran to fix it:

for d in *; do sed -i '' -e  "s/module main/module $d/" $d/go.mod; done

I think the --source flag in dagger develop might be broken

(trying to repro)

[deft rain]

I think it might only be "go" that's reserved (you can do "go get go@version") I think now to update the module go version/toolchain

[sharp zealot]

After initializing a fresh module with --sdk=go --source.., I get both dagger.gen.go and dagger/dagger.gen.go

[deft rain]

Ah no this is actually a feature! Although maybe the directory name is confusing

See https://github.com/dagger/dagger/issues/6661 for the logic as to why this is necessary

[sharp zealot]

So if I keep the default value of --source which is dagger, do I get dagger/dagger/dagger.gen.go ?

[deft rain]

Yeah... Not ideal tbf, but I don't really have a ton of better ideas (since the subpackage needs to be nearly importable from a utils subpkg)

We could have a directory "dagger.gen" maybe? But not entirely sure how that works, or looks

Potentially it could be in "internal" actually

And maybe "querybuilder" could go next to it?

[sharp zealot]

No strong opinion. I would say don't worry about it for now

[deft rain]

We can fix this in the future, you're right - with quite minimal breakage I think

[sharp zealot]

How do I check if a file exists?

[deft rain]

In go? I'd do _, err := os.Stat(...) and check err.
For dagger, I guess I'd call Size on a file and check if I get an error?

[sharp zealot]

Sorry I mean in the Dagger API

[sharp zealot]

Right but I need to differentiate regular errors from "no such file" error

Which I would rather not have to do (should file an issue for that) but since I'm stuck with that, I need to remember how to do it

In a SDK-specific way

[deft rain]

Mm, I don't think there's a way to really do it atm

It would be nice to have ".Exists"

[sharp zealot]

https://github.com/dagger/dagger/issues/6713

[sharp zealot]

What does Directory.WithFiles do? Copy multiple files at the same path? I don't get it.

File.Name returns the name of the file. Is that the full path? Or just the last component of the path? What happens if I copy that file at a different path in a different directory?

[sharp zealot]

Better:

dagger  -m github.com/shykes/daggerverse/utils call clean-go-workspace --source . export --path=.

because why not 🙂

[dense canyon]

related Solomon: #go message

[sharp zealot]

thanks I didn't remember where that thread was 😁

[dense canyon]

btw, not sure if this is a setting of the LSP or what, but in my vim setup I can open multiple modules within my daggerverse project with all having main as a package and my editor is happy

https://asciinema.org/a/p8NHfzSfySKhX9zzWaCYKpRuR

[ocean escarp]

It would be cool if we could add a category to the modules so that we could easily filter the modules by category in the daggerverse

[rocky harbor]

yes that was the motivation. I think we should switch it to the path in the repo. Technically just module name alone would leave open the case where multiple modules in the same repo have the same, which is obscure but still worth avoiding. For modules outside a git repo or in the root of a git repo can probably continue to use main? Or dagger_main, etc.

[latent trellis]

I asked for that helper method. It's basically an alias to dir.WithFile("", file1).WithFile("", file2). The file name is the last component of the path. When a file is added to a directory without a name, its original name is used.

[dense canyon]

noted. Thx for sharing <3. cc @ember walrus

[ember walrus]

Hey @ocean escarp! Yes, we plan to do that. We've discussed a few options for adding the metadata for categories. 1) keywords provided by the module developer, which would map to categories 2) Algorithmic analysis of the module, mapping to categories.

Which categories would you want to see?

[deft rain]

aha, ok i have a part 1 to resolve this in https://github.com/dagger/dagger/pull/6716

this removes the Client property entirely (and is a refactoring i've wanted for a while, it's a pain to have to pass client/query alongside each other everywhere)

[ocean escarp]

Great to hear that it's on the roadmap! As for categories, here are some ideas that come to mind: Code Quality, Deployment, Utilities, Testing, Publishing

I really like the new UI of Daggerverse

[warped canyon]

On 0.9.11 I'm seeing something new with dagger develop and the Go sdk. I thought I was crazy but I saw it happen to @wintry prism too 😂

Only with Go modules, when I dagger develop, I get a dagger/dagger.gen.go generated inside of my module which appears to be just the core api and not actually used.

Any chance this is already fixed on main?

[deft rain]

it should be actually used, but it is intentional!

see https://github.com/dagger/dagger/issues/6661 for the logic for why this is a thing

the top-level dagger.gen.go is full of aliases to the dagger/dagger.gen.go

[kind carbon]

The other sdks import dagger, rather than it being local. Wouldn't it be better eventually to do the same?

[warped canyon]

So I get 2 dagger.gen.go now and they're both committed? 😅

[deft rain]

we still a top-level one for the interfaces implementation

there is no way around this that i can think of 🤷

[kind carbon]

Would also be easier to transition to modules as right now you need to remove all the dagger. packages.

[kind carbon]

I've not dug into that yet. Not sure about what the limitation is.

[deft rain]

https://github.com/dagger/dagger/pull/6680#issue-2136906098

Initially this might sound simple - and prior to the addition of interfaces, it was, however, these add a lot of confusion. Specifically, interface codegen is added alongside the main code, and includes adding new methods on user-defined structs as well as creating an implementation of user-defined interfaces.

So where should these new definitions belong? They actually have to go at the same level as the definition of the struct/interface, so that 1. we can actually attach the methods, and 2. so we can actually access the interface in arg/return types (we can't just copy the interface into the sub-package, since it might also refer to user-defined concrete types, which we can't copy).

if you can work out a way where we can have it all in a sub-package, i am happy to try it 🙂

[kind carbon]

Can't you put most of it in the subpackage, and generate another one on the local package only for anything related to a user defined struct or interface?

[deft rain]

also see the above discussion

[warped canyon]

Kind of a meta note ignoring all constraints and not providing any suggestions... 😅

on the DX side it feels like we were at a spot where a "module" could just be really simple couple lines of code and all of the magic happens at runtime. Right now it feels like there's a lot to it. We have the json, multiple generated files + directories, and dependency files that all get committed and require the developer to be responsible for all of the above

[deft rain]

they don't need to be committed - only if you want to have linting/etc

[warped canyon]

Ah ok. We did change the default .gitignore to now commit them, right?

[deft rain]

that's pretty much exactly what's happening, except that the top-level also has the type-aliases to avoid breaking every single go module

[kind carbon]

Oh, so users can transition without removing dagger. from dagger.Container (for example) everywhere?

[deft rain]

yup 👍 should be possible now

there's even a test to mix-and-match these now to confirm you can

[ocean escarp]

There's a problem when you publish in Daggerverse, the page stays on this load even if the module is published

[warped canyon]

Is the git ref here @mod?

[ocean escarp]

this is the name of my git branch

[warped canyon]

Perfect. Yeah that sounds like it may be a bug related to a recent change 🙂 opening an internal issue for that

[sonic vessel]

I used dagger init by using local sdk path. My current working directory is

-rw-r--r--   1 thanabodee  staff    11K Feb 22 22:59 LICENSE
drwxr-xr-x  12 thanabodee  staff   384B Feb 22 23:03 compose/
drwxr-xr-x   6 thanabodee  staff   192B Feb 22 22:57 elixir-sdk/

Then I want to create a module by using dagger init --sdk=elixir-sdk --debug first-mod. The dagger throws the error that elixir-sdk doesn't exists under first-mod like this.

✔ defaultPlatform: Platform! 0.0s
✘ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
✔ moduleSource(refString: "first-mod"): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.withSourceSubpath(path: "dagger"): ModuleSource! 0.0s
✔ ModuleSource.withSDK(sdk: "elixir-sdk"): ModuleSource! 0.0s
✔ ModuleSource.withName(name: "first-mod"): ModuleSource! 0.0s


Error: failed to generate code: input: resolve: moduleSource: withName: withSDK: withSourceSubpath: resolveFromCaller: failed to collect local module source deps: failed to collect local sdk: missing config file /Users/thanabodee/src/github.com/wingyplus/daggerverse/first-mod/elixir-sdk/dagger.json

Then I try dagger init --sdk=../elixir-sdk --debug first-mod, it's now work perfectly.

[warped canyon]

Are you wanting to create the module source in a first-mod subdirectory or in the current directory?

[sonic vessel]

I want to create in the current directory. The directory layout should looks like:

-rw-r--r--   1 thanabodee  staff    11K Feb 22 22:59 LICENSE
drwxr-xr-x  12 thanabodee  staff   384B Feb 22 23:03 compose/
drwxr-xr-x   6 thanabodee  staff   192B Feb 22 23:12 elixir-sdk/
drwxr-xr-x   4 thanabodee  staff   128B Feb 22 23:07 first-mod/

[sharp zealot]

I am going to open a PR to remove --sdk and --source from dagger init.

[kind carbon]

Looks like it should be dagger init --sdk=./elixir-sdk first-mod, i.e., from the repo root.

[sharp zealot]

not worth it 👆 ..

(it was also not worth committing all those generated files, but that ship has sailed)

[sharp zealot]

👆

[warped canyon]

My commit updating example-voting-app from 0.9.9 to 0.9.11 20,759 additions and 83 deletions. 😱

[honest hearth]

To play around organizationally with zenith I started a polyglot monorepo to write some glue scripts that calls modules with both go and python. Any thoughts around how to manage dependencies that modules need?

The pattern I'm thinking right now is:

• Specify one module as the entrypoint for running other modules
• That one module must necessarily be implemented in a single language
• use that entrypoint module for loading shared deps/configuration for both go and python
• In practice this translates to a top level go.mod and pyproject.toml that other modules reference

So i am thinking, in this strict sense, I am forcing all of my go and python deps into these two top level files. The upside here is one place to manage everything. The downside is that the constellation of deps must necessarily be locked to the sources of truth defined in this one go.mod and pyproject.toml

I think this is antithetical to the design of modules though, isn't it? Dagger's CLI currently wants one go.mod and pyproject.toml per module. This is so that modules can run independently, presumably. So how do we manage and prevent the proliferation of a ton of tiny pyproject.toml and go.mod for all of these little modules? Are we supposed to write tooling to pull in and manage version bumping?

[warped canyon]

@dense canyon this is also related to what we were chatting about regarding platform team maintaining top level modules + app teams maintaining app modules + platform team maintaining utility modules. I talked with @honest hearth a bunch about this at civo navigate 🙂

[honest hearth]

Related question, I think more on the python side because this is a bit more standardized in golang I think.... To use the dagger python sdk, we are limited to using poetry right now for module package management? (Not saying poetry is bad per-se, indeed, we use it religiously at the place I work, just curious if using dagger modules currently has that requirement under the hood cc @kind carbon

[kind carbon]

No, see this: https://deploy-preview-6681--devel-docs-dagger-io.netlify.app/zenith/developer-guide/python/820256/module-structure#modules-as-python-libraries

TL;DR: as long as pip install . works (PEP 517).

[ember walrus]

Not sure if this is known - when you use incorrect syntax in the CLI to init a module, the error message isn't helpful. In the screenshot, I accidentally included an extra = when defining the SDK.

[sharp zealot]

What do you mean? That's super clear.

Yeah error messages in general need work

[thorn moat]

could be an epic of its own tbh. There's a lot to cover, but I think the payoff would be huge

do we have a parent issue/category for poor error messages?

[ember walrus]

https://github.com/dagger/dagger/issues/6655 found this related issue. No epic that I see, but I could make one

[deft rain]

There's quite a few different parts involved: we need better errors in the engine+cli, as well as in the SDKs (go is especially guilty for giving difficult errors atm)

also, related: we should aim to rework our error handling so that with places that it makes sense (like for generating go code), we should generate as many errors as possible, and not immediately bail out at the first one

additionally, having a way to do warnings nicely 😄 that would be super nice for when we do deprecations in module code in the future

(just my 2 cents on things i'd love to bring into the epic if we make it)

[thorn moat]

don't know if this qualifies as "nicely" but there is (progrock.Recorder).Warn which sends to the TUI

[deft rain]

mm - that works, but it would be really nice if we could attach warnings to each module? that way we could warn users on just the modules they're developing, we could connect that into daggerverse potentially (and warn authors about potential deprecations, etc)

super nitpick, tbf, it's definitely not a high priority ticket 😄

[thorn moat]

ah gotcha. I think that ties into the whole 'module best practices' idea, maybe a linting-style approach would would be nice (bonus points if it's able to link to specific parts of your code)

[deft rain]

massive agreement on this btw - having nice error messages genuinely does make me happy to use dev tools, so doing this could really lower the barrier to entry

[thorn moat]

elm is my go-to example for excellent error messages: https://elm-lang.org/news/compiler-errors-for-humans

would genuinely like to see this become a top prio as part of a general focus on polish post-launch

[deft rain]

sounds like you just volunteered to take it on 😄

[thorn moat]

i hear we have a dev on the team that loves to hunt down esoteric error messages though

[deft rain]

i did spent an embarassing amount of time just now tracking down a wonderfully descriptive message invalid request that i'd added as a little hack

[rocky harbor]

Made an issue for collecting together all unhelpful error messages: https://github.com/dagger/dagger/issues/6718

Seems like the easiest way to collect them together rather than creating 10k separate issues 😄

[sharp zealot]

Calling for feedback. We're making good progress on the developer docs for Zenith. But there is an unresolved question of how much to centralize. How much to write in SDK-specific sections vs. a common section.

Here is a preview link with 2 options (Options A and B visible in the sidebar). Please share your thoughts! https://deploy-preview-6681--devel-docs-dagger-io.netlify.app/zenith/developer-guide/

[restive shore]

I think i found a bug. If I have an error type in a go module struct dagger panics when doing dagger develop. Has anyone else run into this?

[rocky harbor]

Yeah that sounds like a bug, would not be surprised if we handle that poorly. In general I am not sure how/whether to handle that at all since I’m not sure what a go error type would map to in gql, but we should at minimum not panic and error out more clearly

You could store the error as a string in the meantime perhaps

[restive shore]

Yeah that's what I'm thinking of doing. The reason I have to "transport" the error is because I don't want to break the module chaining and there is no easy way to break out of an intermediate stage.

I will open an issue if there isn't one already

[rocky harbor]

Thank you, also feel free to include the use case too, curious to see if there’s any other approaches possible

[restive shore]

Yes it's possible that I'm doing something silly

[sharp zealot]

What does @field mean in the Typescript SDK?

[wintry prism]

@upbeat herald 👆 (silent mention)

[upbeat herald]

Maybe you can give a shot to the doc I wrote about in https://github.com/dagger/dagger/pull/6681

-> https://deploy-preview-6681--devel-docs-dagger-io.netlify.app/zenith/developer-guide/typescript/582020/conventions

If that's not clear enough, then I shall add more content based on your feedbacks 😉

And https://deploy-preview-6681--devel-docs-dagger-io.netlify.app/zenith/developer-guide/typescript/582020/conventions#fields-exposition

[wintry prism]

Noticing some errors are hidden by new TUI @thorn moat

is that known? progress=plain showed the error

[thorn moat]

First I’ve heard of it, do you have a repro?

[ocean escarp]

how to override the default shell in dagger call COMMANDS... terminal?

[ocean escarp]

I found it, it's the withDefaultTerminalCmd function

[deft rain]

mm, this was starting to really bug me while writing docs, so decided to try take a stab at reoganizing the structure to be a bit clearer: https://github.com/dagger/dagger/pull/6721

[rocky harbor]

Hey all, chatting about Zenith for the next hour in dev-audio for any who want join!

[dense canyon]

👋 just checking if the error message from dagger functions $function_name has been raised? My function currently returns a string and it was a bit suprising to see that dagger functions errors with Error: function 'validate' returns non-object type VOID_KIND

[rocky harbor]

That's definitely not expected, issue w/ repro would be appreciated 🙏

[dense canyon]

https://github.com/dagger/dagger/issues/6731

[sonic vessel]

I try publishing a module that written in Elixir. The dagger publish said:

Publishing github.com/wingyplus/daggerverse/wttr@ea6331e52fc93fdec260559fefc99161c2f3f863 to https://daggerverse.dev...

You can check on the crawling status here:

    https://daggerverse.dev/crawl/github.com/wingyplus/daggerverse/wttr@ea6331e52fc93fdec260559fefc99161c2f3f863

Once the crawl is complete, you can view your module here:

    https://daggerverse.dev/mod/github.com/wingyplus/daggerverse/wttr@ea6331e52fc93fdec260559fefc99161c2f3f863

When https://daggerverse.dev/crawl/github.com/wingyplus/daggerverse/wttr@ea6331e52fc93fdec260559fefc99161c2f3f863. The page still uploading the module.

Not sure if https://github.com/wingyplus/daggerverse/blob/main/wttr/dagger.json#L3 cause the issue.

[sonic vessel]

Oh the page is now changed to crawl error: ["error" "get module for ref: no rows in result set"].

[strong ingot]

cc @thorn moat @void widget 👆

[dense canyon]

checking

[dense canyon]

just tried to re-published and it seems to have worked now? 🤔 https://daggerverse.dev/mod/github.com/wingyplus/daggerverse/wttr@10cffdbede63f685f7c6ad4154a22e2682622811

[honest hearth]

I think other people have asked this before, but in the newer versions of dagger, what is the new equivalent of dag.Host()? e.g. how do I reference a file that I want to stage into a container? Could not find docs for this

[deft rain]

you need to explicitly pass it as a function input

or, if you want just scratch data that's not an input, you can use CurrentModule.Workdir

[honest hearth]

nice, i think the latter is what I want, since I am simply staging a script into the container that will always live in the current module's workdir

[honest hearth]

Does Dagger support optionals defined in go in a struct like this?

type Vault struct {
    // Github token for authentication
    // +optional
    GithubToken *Secret
}

func New(github_token *Secret) *Vault {
    if github_token == nil {
        github_token = dag.SetSecret("ghApiToken", os.Getenv("GITHUB_TOKEN"))
    }
    return &Vault{github_token}
}```

I notice that it seems to work in function arguments, but if you put it in a type like above it seems to always want this flag to be present, even if we add this `// +optional` syntax

[restive shore]

It does not. Only thing you can specify with pragmas in a struct is // +private

If you are looking to set defaults for a module "globally" for all functions you can use a constructor New()

eg: https://github.com/kpenfound/dagger-modules/blob/main/golang/main.go#L20

Oh I missed that you are already using a constructor in your example. You can set optionals and defaults there 🙂

[sharp zealot]

FYI @warped canyon @bleak nest: https://github.com/dagger/dagger/pull/6707/files#r1503201262

[honest hearth]

Oh I missed that you are already using a

[sharp zealot]

Non-urgent request: how hard would it be to have a flag on dagger develop to choose the .gitignore behavior? I don't necessarily need the default to change for everyone, but I personally would much rather prefer to continue git-ignoring generated files. Without a flag to do it for me, I won't bother, too much risk of accidentally breaking my module

I am hitting a mysterious error message, and could use a second opinion...

Repro:

dagger call -m github.com/shykes/daggerverse/core@v0.1.0 \
  git --url https://github.com/dagger/dagger \
  branch --name main \
  tree \
  entries

The error I'm seeing: Error: unsupported object type "Socket" for flag: ssh-auth-socket

[sharp zealot]

Does .gitattributes make it possible to hide generated files from the output of simple commands like git diff or git diff --cached ?

[sharp zealot]

Is there a way to expose all the ports in a service, natively? Eg. dagger call <SERVICE> up --native?

I was surprised that by default, up doesn't actually work. You need to manually pass ports to forward

[sharp zealot]

I'm hitting another issue, this time trying to call a dagger module from within another dagger module... Kind of hard to explain, here is the repro:

[rocky harbor]

I was surprised that by default, up

[thorn moat]

Does .gitattributes make it possible

[rocky harbor]

Definitely possible yeah. Though I'm curious if we did a better job hiding the codegen output everywhere (i.e. #daggernauts message) and maybe consolidated the generated files even more if this is still something others would care about. I personally have not really minded committing generated files (it's what you do for a lot of codegen like protobuf, etc.), but I have gotten the impression I'm the only one who feels that way 😄

[honest hearth]

Definitely possible yeah. Though I'm

[sharp zealot]

Hi everyone. Today is the big day! We're going to release Zenith, aka Dagger 0.10. We are calling for docs testers!

This release includes a complete docs rewrite. Later today the zenith docs (https://docs.dagger.io/zenith) will become the primary dagger docs. Now would be a great time to go through them, and tell us if you find any issues.

Thank you all, we're almost there!

[kind carbon]

The new version doesn't seem to be published, at least for me. Could by a caching issue on my side.

[sharp zealot]

Maybe requires manual publish? @bleak nest ?

[bleak nest]

checking

[bleak nest]

done

[sonic vessel]

I think I know now. I forgot to push the latest commit and I using --force to force publishing it. T_T

[dense canyon]

🎉

[sharp zealot]

Note that Daggerverse is not part of today's release: it is still under development and will remain "under the radar" for a few more weeks

Also not included today: an overhauled Dagger Cloud, to visualize custom functions (it only visualizes core functions today). That's in development, but not yet ready.

Let me know if you want to alpha test that 🙂

[honest hearth]

I'm working on a module that stages aws credentials into the container from the host machine using WithMountedSecret. I'm trying to conceptualize the how dagger organizes the host -> container interaction in this new post Host() land.

The module is a go module. It attempts to go find the user's home dir using homeDir, err := os.UserHomeDir()

However, os.UserHomeDir() returns /root, as if it's executed in the container, rather than my mac's home directory that would normally be output with echo $HOME. Where is os pulling from in this context? Is it the dagger engine's container? The module's container? How can I reference the host's home directory instead?

[sharp zealot]

The module's container (technically that one function call's ephemeral container).

You cannot access the user's home directory. Instead callers have to explicitly pass everything as arguments.

We're trading some convenience for better isolation of functions, which makes the platform better overall.

We also plan to add a layer of convenience on top. Will link to design discussions for that.

See #daggernauts message

[thorn moat]

now seems like a good time to remind everyone that the emoji exists (i forgot about it)

[honest hearth]

Continuing the above thought, does Dagger support mounting entire sensitive directories? I'm looking at the above workflow for AWS SSO that would be made much easier by mounting entire ~/.aws folder into the container. Is there a safe considered way to do that? WithMountedSecret appears to only target individual files. Looping through each file in the folder I need doesn't seem really a great way to handle this, since there may be an arbitrary amount of random files in ~/.aws/sso/cache

[sharp zealot]

I think mounting a directory is considered safe, since mounts are not persisted

[sharp zealot]

Zenith is officially released! Dagger 0.10 is out, and the new docs are shipped.

A huge THANK YOU to all of you, for investing time and energy into testing Zenith... Putting up with breaking changes.. Participating in the design, reporting bugs, suggesting use cases... All of it was so, so useful.

Today we're letting the dust settle... And tomorrow is launch day. There will be blog posts, social media, etc. etc. Should be fun!

[plucky ermine]

It feels like the previous dagger up command (now I think it would be dagger call $SOME_FUNCTION_RETURNING_SERVICE up is not documented at all. Am I missing something obvious in latest docs?

up is particularly hard to search for in our docs 😅

It feels like the previous dagger up

[plucky ermine]

One more thing that feels missing from the docs is a discussion around the daggerverse style repo with may modules in it, and how to use git tags to create the version.

[sharp zealot]

@plucky ermine would you mind opening issues to track these gaps in the docs as you find them? Would be super useful 🙏

[subtle plinth]

Today I feel like I've spent 90% of my time waiting for Dagger modules to fetch and compile rather than running any real pipelines 😅

[restive shore]

I see here in previous conversations that a module field always gets a getter function and I can't access a field as a field. Is this documented in zenith docs? I couldn't find an explanation or the signature of the automatic getter.

[strange arch]

Hello, my main module sources are in the dagger directory at the root of my Git repository and I’d like to access files in the root of my Git repository. Did I understand correctly that it’s not possible anymore and that I must pass the files as arguments to my functions?
Also, how can I add a directory passed as an argument to the working directory of my module? I’d like to use Helm Go packages to load a chart from my module code, but I can’t figure how to do it with latest versions of Dagger.

[restive shore]

While developing modules with Go, it feels weird that a return type (*Container, error) just drops the error for consuming modules.

[kind carbon]

Fields

[warped canyon]

Does querybuilder have to get committed now? I'm updating greetings-api which has an older .gitignore and the module is failing to codegen on another machine

[kind carbon]

It only needs to be committed if you're using some external linter (CI for example), without doing dagger develop first. Otherwise, you can gitignore.

[warped canyon]

Hrm ok. On a second machine where I've cloned the project, trying to dagger develop, I see

exec /usr/local/bin/codegen --module-context /src --module-name greetings --propagate-logs=true --introspection-json-path /schema.json 0.7s
      ┃ Error: exit status 1                                                            
      ✘ generating go module: greetings 0.4s
      ┃ writing go.mod                                                                  
      ┃ writing go.sum                                                                  
      ┃ creating directory internal/querybuilder                                        
      ┃ writing internal/querybuilder/marshal.go                                        
      ┃ writing internal/querybuilder/querybuilder.go                                   
      ┃ go: finding module for package github.com/kpenfound/greetings-api/ci/querybuilde
      ┃ go: github.com/kpenfound/greetings-api/ci/dagger imports                        
      ┃         github.com/kpenfound/greetings-api/ci/querybuilder: module github.com/kp
      ┃ greetings-api@latest found (v1.1.0), but does not contain package github.com/kpe
      ┃ reetings-api/ci/querybuilder

but dagger develop works on the machine where I've developed the modules, which has all of the ignored files

[kind carbon]

There's been a change in where the pacakges get generated to (internal). Do you have the same dagger cli version in both places?

[warped canyon]

Both are 0.10.0

[kind carbon]

.gitignore should be:

/dagger.gen.go
/internal

Adjust to where --source is.~

[warped canyon]

I have

/dagger.gen.go
/internal/querybuilder/
/querybuilder/

it was last modified with 0.9.0

[kind carbon]

Do you have workspace files?

[wintry prism]

Today I feel like I've spent 90% of my

[sharp zealot]

OK Daggernauts, the moment is approaching... We're finishing up the launch blog post

Here's a fun experiment, which I may or may not regret... 🙂 Here's the draft blog post, feel free to review and comment.

https://docs.google.com/document/d/1JTGkRR1kfg9oyh1Ar96XnDqiU6Z4IFUvKUHyTdnP3rc/edit?usp=sharing

When we say we develop in the open, we're not kidding 😛

@wintry prism @warped canyon what's a good example of a dedicated module repo, for the blog post?

[warped canyon]

as in someone's daggerverse repo?

[sharp zealot]

No, the other pattern

1 module = 1 repo

Also looking for a good example of a module embedded in an app repo

[warped canyon]

my kpenfound/greetings-api and kpenfound/example-voting-app have embedded modules, granted they're demo projects and not "real"

[sharp zealot]

That's fine. Which one would you be more comfortable linking to from the blog post?

[warped canyon]

Probably greetings-api because it covers cases like lint/test/deploy that voting app doesn't have yet

[dense canyon]

👋 does it make sense for moduleSource(refString:"foobar") { asString} to not return an error? cc @rocky harbor

[bleak nest]

Have opened an issue to add more docs around this

[sharp zealot]

I'm trying to hint at use cases of Functions beyond "regular CI". I know that's been a recurring theme, @obtuse lion you were the first to point that out. What are good examples of other kinds of pipelines which might be relevant?

As we developed and tested this feature with the community, we noticed something interesting: Dagger Functions are great for CI, but they can be applied to other problems as well. From data processing pipelines, to AI pipelines, to custom dev environments, [...]

[plucky ermine]

Thanks @bleak nest it was on my list, I appreciate you!