daggernauts

[spiral whale]

1password on Zenith 🦤 . I really love how easy to create a module and integrate it into my workflows.

[sharp zealot]

awesome 🙂 but wait how does $OP_SERVICE_ACCOUNT_TOKEN get expanded in that graphql file?

[spiral whale]

Good old ‘sed’ 🙂 it’s not expanded ( dagger query --progress=plain container < <(sed "s/$OP_SERVICE_ACCOUNT_TOKEN/$OP_SERVICE_ACCOUNT_TOKEN/g" examples/examples.gql) )

[sharp zealot]

I just ran my first Zenith function from a stable dagger build 🙂 0.8.8

@kind carbon how do I play with the latest and greatest of CLI invocation?

[kind carbon]

It's in https://github.com/dagger/dagger/pull/5882. I'm trying out @thorn moat's fix for getting an argument's type so you'll pretty soon be able to have Directory and Secret args if it all works out.

[sharp zealot]

https://tenor.com/view/oh-boy-gif-24157752

[sharp zealot]

https://tenor.com/view/why-are-you-torturing-me-torture-annoying-over-it-cant-deal-gif-15666755

[kind carbon]

File support:

[thorn moat]

https://tenor.com/GV5n.gif

[spiral whale]

This is like a trailer for the final episode of your favorite TV show. 😵

[strong ingot]

Can you share an mtr daggerverse.dev -rwnbc 10 via DM? Would like to dig into this one.

BTW, I suspect that it's the same reason that registry.dagger.io doesn't work for you.

FWIW, this is a temporary setup, I expect us to migrate it over the next few weeks.

[sharp zealot]

https://tenor.com/view/teaser-discord-xd-sj-waiting-gif-19503918

[sonic vessel]

Learning Zenith - Jenkins DSL 👀

[restive shore]

whoa! That is very close!

[restive shore]

Will PM you. Having trouble getting mtr to run

[kind carbon]

@sharp zealot @deft rain I pushed a few arg types support to https://github.com/dagger/dagger/pull/5882.

[sharp zealot]

When you want to try @kind carbon 's CLI awesomenedd, but are too lazy to fetch his PR into your local dev environment...

echo '{dagger{engine{dev(repository:"https://github.com/helderco/dagger", branch:"helder/dev-2744-pass-arguments-to-functions-as-subcommands"){cli(operatingSystem:"darwin", arch:"arm64"){export(path:"./superdagger")}}}}}' | dagger -m github.com/shykes/daggerverse/dagger query

./superdagger call --help

(Requires Dagger 0.8.8)

[upbeat herald]

Is there a way to update the dagger.json to not publish a package? I'll create a module to tests my other modules but it's not something that should be used by anyone else than me

It's kind of a replacement of a possible mage file, with a method integration-test, lint etc...

[kind carbon]

Did someone ask for ⛓️?

You can check the query that's executed in debug mode.

[thorn moat]

@sonic vessel Small local dev requesst for the Elixir SDK - when sdk:elixir:generate fails, it seems to leave my local state with the entire sdk/elixir/... removed, which I have to git checkout before I can try again. Haven't had time to look into why it's happening, it's not a big deal just slightly annoying. If you can find time to fix that I'd appreciate it 🙏

[rocky harbor]

You can check the query that s executed

[sharp zealot]

Starting a quick "project management" thread to talk about Kubecon countdown...

[bleak nest]

I need some help creating and testing Node and Python examples for the services v2 guide...Helder/Tom/JF, can you please have a look whenever possible at https://github.com/dagger/dagger/pull/5557 where I've put inline comments for the snippets I need your help with? Thanks!

[rocky harbor]

debug module tests

[sharp zealot]

@rocky harbor linear bikeshedding, if you don't mind I'm going to move dagger shell and dagger serve under call functions from the CLI since that's the big killer feature and in my mind, each verb we want to ship on top of call should be part of it

[rocky harbor]

Yeah that makes sense, they're just variations on ways of calling/interacting w/ functions

[sonic vessel]

Let me take a look on next week. 🙏

issue created https://github.com/dagger/dagger/issues/5901 🙂

[thorn moat]

thanks! ❤️

[rose hinge]

Hi, I'm trying the module feature but I have an issue.
It seems when a ref of my object is send to a children, private attributes are not transfered.
If I try to put in public the attribute (secretManagerClient) I have a graphql schema error related to the type: secretmanager.Client where the Client type seems to create a conflict.
I don't really understand what I'm doing wrong or how to deal with that, someone have an idea ? (the code is available here: https://github.com/Dudesons/daggerverse/tree/main/gcp-secrets-manager)

[upbeat herald]

secretManager.Client need to ba a marshallable object, I think that might be the reason

Hello guys, I made 3 modules (Golang, Golangci-lint, Node) to test some dependencies capabilities of Zenith and explore way to test module through a ci module (that should be private, it's not usable in another context)

https://github.com/quartz-technology/daggerverse

I'll also add tests for Golang and a python Module soon, it's pretty fun to play with Dagger Zenith 🚀
Using concurrency definitely helps for tests btw 😄

[rose hinge]

ok but this type is from the gcp package so I'm just using it. Should I change my code and instantiating the client directly in Get/Set secrets method and maybe just give in a Credentials call credentials information ?

[upbeat herald]

I think so yeah

[rose hinge]

ok thank you I will try that

[obtuse lion]

the dagger playground localhost CORS approach is super cool 🙂

for playground modules should be able to have examples, like godocs does. Or the same in daggerverse

i'm having a lot of fun with modules, i'm doing some really stupid things, but I will say it's pretty hard to understand how to create modules that don't suck. Like what's a good pattern. I'm sure since this is all so new yall are still figuring out the best style. But in general, adding methods to container I think is a difficult to understand model. I've seen over and over again that people really really struggle with the idea of CI tasks running from a containerized context. Add things to directory makes more sense to be. Like Host().Directory(".").Gobuild(). Then the consumer just feels like they are "running a command." I'm building something with dagger that is a bit off the wall and hope to be able to share it soon (not convinced it's a good idea, but it's fun).

[upbeat herald]

You can check my modules for some pattern's ideas 😄 I'm still trying to find out the best way to create a module tho haha

You also can extend any dagger type with method (directory included) but I'm not sure you can retrieve the parent's container for now so .Directory(".").GoBuild() is not possible for now

[sharp zealot]

@obtuse lion yeah I no longer extend core types personally

My favorite pattern is straightforward but awesome: my own structs with their own state and methods

[obtuse lion]

i'm trying to rewrite one of my projects CI pipelines using zenith, so fooling around with different approaches. I struggle with the idea of "what should the output be". For example, gobuild is the core thing I need to do. Should the result of that function be a that it wrote the binary back to the host, or should it return a directory that has the binary in it. The later is definitely cooler, but it's hard to understand how to use it. I'm doing this all from the perspective of it I rewrite everything with zenith whats the chances my engineers don't get confused.

[sharp zealot]

if it helps i started doing this for our own ci 🙂

[obtuse lion]

I really struggle with the concepts of things existing in a parallel containerized/cached universe. I saw this with gh action where they started with a dockerized model, but then what took off is the javascript approach where everything runs in the same context.

[sharp zealot]

https://daggerverse.dev/mod/github.com/shykes/daggerverse/dagger

[obtuse lion]

I hope there's some happy medium where it can look like things are all running in the same context, but really they do fancy things on the side and don't have to pollute the host like GH actions do.

[sharp zealot]

A simpler implementation of the pattern: https://daggerverse.dev/mod/github.com/shykes/daggerverse/supergit

[obtuse lion]

This is cool, but I still struggle with complexity that you are basically creating a virtual workspace. It exists "somewhere" in dagger. That's hard to grok unless everything you are doing is in that virtual space. Like if your dagger "script" is "first checkout, do stuff" Then it's fine because everything is in one virtual place. But if your starting from the perspective that you already have everything checked out locally on disk (how all CI flows start). It's hard to mentally reconcile what you have locally with what is happening virtually elsewhere. Sorry if that confusing to explain. Like the full mental model of how to best use dagger hasn't quite clicked with me in a way that I can easily explain it to someone. Everything I do with it so far is super fun and cool, but I just know I can't easily give it to someone else, they will be so confused. Not that it's a flaw of dagger, just that I don't understand the right way to hold the phone 🙂

[obtuse lion]

I do really like that pattern you have with the git commands holding there own state.

I would vote to not allow extending core types. It still doesn't seem right to me. My two cents. I think the method chaining approach really makes sense from the perspective that you are essentially doing a fluent api builder pattern. But I don't think it should be abused to the point that try to do everything in one chain. So self referencing method are nice, but adding to other types not so much.

[wintry prism]

experimented with goplantuml

go get github.com/jfeliu007/goplantuml/parser
go install github.com/jfeliu007/goplantuml/cmd/goplantuml@latest

goplantuml -show-compositions -show-connection-labels -show-aliases -show-aggregations -show-implementations -show-options-as-note  ./supergit > supergit.puml

Then pasted contents of supergit.puml in https://www.plantuml.com/plantuml/ and hit submit.

[latent trellis]

Now that zenith is merged, do I still need a local dev engine? Or can I enable it with an environment variable?

[latent trellis]

Once there is an easy/ier way to chain the outputs of different graphs, there is really no need for extending core types.

[sharp zealot]

Yeah I now agree with you

[sharp zealot]

no env variable needed, we just hid some commands and flags, but they’re all there

[wintry prism]

Not that I love UML, just wanted a fast way to visualize a module's structure. Wondering what the preferred tools are for such things in Golang ecosytem...

[obtuse lion]

The WithX style doesn't really represent cardinality well. What I mean is the ambiguity of a method WithFoo, is that replacing Foo or appending Foo. I'm trying to understand the style where it seems like WithX function should be used to build an object, essentially aggregate state.

Is there a public version of querybuilder somewhere? Not the generated one in ./internal?

[rocky harbor]

Not at this exact moment but we're making that change in a PR https://github.com/dagger/dagger/pull/5882 (not sure if commit w/ that is pushed to pr yet or not, but we were talking about it the other day)

we could most likely split it out to its own quicker PR if helpful

[obtuse lion]

yeah, i keep copying the internal package into my own project for what I'm doing.

[kind carbon]

Yes, that's been done already.

[rocky harbor]

Yeah it's not always clear, even in our core API sometimes. One option is to just use names like WithAppendedFoo and WithReplacedFoo, with the obvious tradeoff of verbosity.

[obtuse lion]

I'd vote for WithFoo for replacing as that seems like the default behavior and WithAppendFoo for appending

Is it intentional that dagger mod --help doesn't show any help about init/use

[obtuse lion]

Am I missing an secret env var?

[kind carbon]

Our API appends on lists/maps (like WithEnvVariable), but replaces otherwise (like WithWorkdir).

[obtuse lion]

Yeah, my comment is just that there's no way to really know if you are operating on a list/map. Unless I'm missing something.

[rocky harbor]

That makes sense, though there is a somewhat interesting exception in our core API in that WithExec is appending an exec operation to the chain of operations to perform. But WithAppendedExec sounds weird, so not sure what the exact answer is here (or if it matters, maybe just go with what's intuitive and rely on docs to disambiguate as needed?)

[obtuse lion]

Like the funky thing is you aren't really working on a list/map with something like WithMountedDirectory() or WithExec(). I struggle mentally with this where the WithX fields aren't just adding state to a struct (which would imply that order doesn't really matter)

[kind carbon]

You have a hint from the getters. With workdir you have Container.Workdir(), with the env vars you have Container.EnvVariable(name) and Container.EnvVariables().

[obtuse lion]

I just pulled this PR, much nicer to have the querybuilder API. Thanks

[rocky harbor]

Is it intentional that dagger mod help

[obtuse lion]

I'm trying to do a multicall type thing where I want this code to run in the context of a module or not. The only way I seem to know if I'm in a function call is if CurrentFunctionCall().Name() blows up. Could there no a better flow that doesn't result in an error to know if I'm in a function call?

[rocky harbor]

There definitely could be, it would be trivial to add an API that just returns a bool based on whether you are in a module or not. That just wasn't really a use case we've thought about before 😄 Curious about the details whenever you have time. In general, we want users to execute while containerized as much as possible, so we just want to be careful to not rush into making these sorts of use cases too easy necessarily (though happy to be proven wrong)

[obtuse lion]

don't make any changes for me, I'm most definitely doing something wrong. I'm going to proceed forward with just checking for an error. If I can get this working I'll show you all the dumb thing I'm doing. Is there still a weekly call on friday?

[rocky harbor]

Yep, Friday at 9am PST, looking forward to hearing about it!

[obtuse lion]

I'm supposed to be launching a new product, but this is way more fun.

[rocky harbor]

Thoughts on extending core types (and generally types from outside your module)…

Conceptually, it’s all total sugar. In terms of functionality,query{container{foo(arg:”abc”)}} has the same effect as query{foo(container:”<container id>”, arg:”abc”)}

func(c *Container) Foo(arg string) has the same effect as func Foo(c *Container, arg string)

etc.

So, by that argument, it does feel like our raw graphql API doesn’t need to support extending other types. Sugar is not very important on that level.

In terms of our codegen across various languages, sugar does matter though. I have a million thoughts on the cases where extending other types is genuinely useful vs when it’s an anti-pattern. But it just devolves into the same arguments programmers have been having with each other for decades about programming styles, the best language, the best API, etc. It’s honestly just kind of boring at this point.

What’s more interesting IMO is that whatever choice Dagger makes for our official SDKs in terms of whether methods can be added to core types, it would still be possible for someone else to say “that’s awful” and then go make their own SDK that encodes their own opinions. That’s possible because of our “consumer-centric” model where codegen is done client-side and because of the work over the last month to allow SDKs to be implemented as modules and then published+consumed by anyone.

So to be concrete, if we didn’t allow core type extensions in the graphql schema or in our Go SDK, someone else could still implement a AltGo SDK that does codegen with methods extending core types. Using the example above, they could find what would have been methods like func Foo(c *Container, arg string) and instead codegen them as func(c *Container) Foo(arg string). There would be some interesting heuristics there, but definitely possible.

[obtuse lion]

Is the point your making is that sugar can be added after the fact by a consumer and therefore that is reason to possibly delay allowing it? My general approach is just that when it comes to UX, conservative is better. Once it's proven it can be added later. So it seems like having the ability to bake the feature a bit more outside of core might be good.

[rocky harbor]

Yes exactly, I'm sort of thinking out loud so sorry that the point wasn't clear, I'm still figuring it out 😅

Basically, the first important point is that it's not needed at all in terms of graphql schema, no utility there. The second important point is that I realized even if we don't implement this ability in the raw graphql schema, then it's still possible for codegen'd clients to choose to do it. Basically, I'm realizing whatever choice we make isn't really a one-way door, which is great.

I've heard dramatically different opinions on the feature, so we'll upset someone no matter what. So since all else is equal, 100% agree that just starting more conservative is probably the better option.

[obtuse lion]

upsetting people is sometimes a feature 🙂

So I'm trying to create my own sdkRuntime. Which seems like it's just "create a module like" https://github.com/dagger/dagger/tree/0f875dc80c5d2cac080f6add961fef7851cfa685/sdk/go/runtime. The one thing that's not obvious to me is how to publish it. What is the expected format for the docker container?

[rocky harbor]

One thing right away, there's a PR out to refactor all that (goal is to make it simpler): https://github.com/dagger/dagger/pull/5900

[obtuse lion]

tbh, what I really want is a sdkRuntime that just runs a binary. Eventually i want to create a real sdkRuntime, but right now it would be easier to bootstrap my madness with one that just runs foo. Right now I'm hacking this by using the go sdkRuntime that has an init function that block and runs my real main, which just calls my actual binary. Super hacky.

but I'll definitely read through this PR.

[rocky harbor]

Ah interesting (and very creative workaround ), that makes total sense honestly, maybe we should make that one of our first sdk runtimes... It would be super generic and re-usable for many (if not most) other sdk runtime implementations.

[rocky harbor]

It's very new, so take with lots of salt, not vetted yet

[latent trellis]

This is probably a terrible idea, but I managed to get zenith generated code run with Mage: https://github.com/portward/cerbos/pull/13 😂

[rose hinge]

Hi, is it possible to have a list as an input for a method in a module ?

I tried something like that: func (f *Foo) WithBars(e []Bar) *Foo { // ... return f }

but I have this error: ```dagger mod sync
• Engine: ceb73106b135 (version v0.8.8)
⧗ 3.75s ✔ 51 ∅ 7 ✘ 2
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to call module to get functions: failed to get function output directory: process "go build -o /runtime -ldflags -s -d -w ." did not complete successfully: exit code: 1

[oblique reef]

get your os's args or use a popular library to pull them in

[rose hinge]

I'm sorry but I don't really understand what you suggest. If I comment this method the dagger mod sync is working correctly

[ocean escarp]

Is it possible to pass environment variable from the host to the dagger module?

[bleak nest]

I need some help creating and testing

[kind carbon]

You can send the value as you're calling, like dagger call deploy --token $API_TOKEN. But... we could also interpret a function's EnvVariable parameter to interpret the value of the flag as the key to fetch the OS var from. Like dagger call deploy --token API_TOKEN . You'd then need to token.Value(ctx). Not sure when that would be preferred over just sending the value though. And it may be worse for reusability, when consumed by other functions (having to instanciate an EnvVariable).

[kind carbon]

I used a bad example with $API_TOKEN because that would be a Secret (which is supported by the cli in https://github.com/dagger/dagger/pull/5882), not a string, but you get my point.

[ocean escarp]

ok, so I need to pass as an argument to the function. Now, if my function needs several optional parameters?

[kind carbon]

Just keep passing flags 🙂

Python module runtime unlocked! 🎉 Thx @rocky harbor!

[rocky harbor]

(PR if you want to track: https://github.com/dagger/dagger/pull/5900, just some cleanup now that we've verified it's working, should be merged in the next day or two so everyone can start writing Python modules!)

[rocky harbor]

It should work, but just tried it too and it didn't, filed an issue to fix it here: https://github.com/dagger/dagger/issues/5904

[deft rain]

oh i ran into this earlier while looking into https://github.com/dagger/dagger/issues/5864 🙂 pretty much every single "weird" way of constructing inputs works a bit unexpectedly

e.g., variadic params also don't generate compileable code

[kind carbon]

I actually added support for []string to use with Alex’s apko module but haven’t committed 🙂

[ocean escarp]

My problem is that when I register the module function, I still don't know how to say that these arguments are optional.

[rose hinge]

Hi I have a question when using modules, is it possible to use modules in a code which is called with dagger run ... or I have to use another command or doing something different ?

[deft rain]

Today, you'll need to use dagger query, like in https://docs.dagger.io/labs/project-zenith/#creating-your-first-module
But we're working on dagger call, which will let you directly call functions from whatever programming langauge your module is in from the CLI (so no need to write graphql queries)

[rose hinge]

ok so until the dagger call, do you recommend I wrap my script in a module and call it with dagger query or using a bash script which contain gql queries (eg: I need to request secrets for my pipeline and inject it in a container)

[wintry prism]

Getting this after new module creation with v0.8.8

I remember seeing advice of changing the go.mod to match suggested format. Is there an issue for this already?

[kind carbon]

Not sure. In this case it should be changed to 1.21, not 1.23.

[wintry prism]

My module has other complaints about secrets and such, so maybe that caused something to go wrong...not sure.

[kind carbon]

Python runtime

[obtuse lion]

I realize now that if my first step of my dagger pipeline is to git clone, dagger makes so much more sense to me. Which leads me to two observations:

1. It's surprising difficult to do an good git clone for CI, just look at all the crap actions/checkout does
2. If you are using GH action, you have to do a checkout to get the dagger to then run the checkout... kind of weird.

[warped canyon]

100% agree. With the ability to execute remote modules, no more need to checkout outside of dagger 🚀

[rocky harbor]

Yes agreed on both points.

What @warped canyon said, with the dagger CLI you can run e.g. dagger query -m github.com/org/repo/subpath@commit .... When you do that, the git repo holding the module is pulled in the dagger engine (not to your local fs), so it's all nice and cached and efficient. I almost think of it as the equivalent of being able to specify a remote docker image in a docker run command, except our equivalent in dagger are remote git repos 🙂

Also, a few Zenith iterations ago I did some early experimentation w/ an updated Dagger Github Action that basically just installs the dagger CLI for you and then parses the GH env vars to figure out what repo you want to checkout and uses that as your module. So I ended up being able to run my CI steps w/out including the standard actions/checkout@v3 step first. Ultimately just a marginal performance improvement, but pretty nice still.

[thorn moat]

you need to bump to go 1.21, which frustratingly changed the default go.mod version format in a backwards-incompatible way, and the message is pretty confusing. your local pre-1.21 Go is unhappy seeing a full MAJOR.MINOR.PATCH semver number that 1.21 produces, where it previously only supported MAJOR.MINOR

[sharp zealot]

The performance improvement is not always marginal: for example @visual blaze and @vivid hatch are looking to move git checkout into Dagger because they use lfs for large game assets, and that GHA checkout action takes longer than the rest of their CI combined! A common problem in gaming.

[wintry prism]

regarding passing secrets around from GraphQL or CLI, what's the best strategy for now? I was trying out a HelloSecret module which is just Solomon's HelloWorld with a greeting that's a Secret. I realized I might need generics or something since my naive approach was to overload the function name with different arg types. Since I figured I might want to pass a Secret from code and a SecretID from GraphQL. But does Secret() not get an existing secret in an engine by ID? Or are they scoped to a session? Saw another discussion about introducing a GetSecret(), but maybe the solution is just adding the ability to pass a Secret from the dagger cli, eh?

type HelloSecret struct {
    Greeting *Secret
    Name     string
}
...
// Change the greeting
func (hello *HelloSecret) WithGreeting(greeting *Secret) *HelloSecret {
    hello.Greeting = greeting
    return hello
}

// Change the greeting
func (hello *HelloSecret) WithGreeting(greetingID *SecretID) *HelloSecret {
    s := dag.Secret(greetingID)
    return WithGreeting(s)
}

[kind carbon]

With https://github.com/dagger/dagger/pull/5882 you can pass by the CLI: dagger call with-greeting --greeting $MY_SECRET. The CLI will make the c.SetSecret for you with that value, before passing it to the query.

[wintry prism]

Improve cli for calling module functions...

[plucky ermine]

I have a dumb question that I feel was already answered but I cant seem to find it.

It seems like none of the modules actually import the dagger SDK - I dont fully understand how this part works.

The reason I ask is because I am not sure if its possible to get your text editor to show the dagger SDK definitions and do autocomplete type things without importing the dagger SDK?

[deft rain]

Codegen 🎉
So instead of importing a library, we generate code in dagger.gen.go.
If you run "dagger mod sync" then the generated code is created, and then copied locally so your ide picks it up 🙂

[kind carbon]

And that's done locally only because of the IDE in fact, otherwise it wouldn't need to.

[warped canyon]

If you're having trouble with autocomplete in Go and you have a monorepo of modules, open the module directly in your editor rather than the parent directory

or setup a go.work, but I don't have instructions handy to help with that

[deft rain]

Or you can use go workspaces - go work init && go work use ./subdir 🙂

Jinx @warped canyon 😂

[plucky ermine]

Thanks everyone ❤️ I feel like I get bit by this once a week haha, I will get the hang of this soon.

[plucky ermine]

this solved my issue - I am getting this tatood on the top of my right hand

[wintry prism]

For those who didn't look inside the thread above. Secrets passed via the CLI! 🚀
https://github.com/jpadams/daggerverse/tree/main/helloSecret

[quick wind]

I notice all mods are written in Go so far and I recall seeing something about Go being the only supported SDK for now? Is there a timeline for other SDK support? Waiting on Python to demonstrate this internally

[kind carbon]

Oh yeah, Python works on main actually: https://github.com/dagger/dagger/blob/b74df2479214020502fcb917d77325f00bd4d4e1/core/integration/module_test.go#L707-L740. I'm working on making it better (mod init, codegen, etc..), and expanding what's supported.

[deft rain]

@rocky harbor @thorn moat would you be free around now to catchup around option types? https://github.com/dagger/dagger/pull/5907

[thorn moat]

@rocky harbor fyi: had to undo your recent change to have core/ depend directly on internal/mage/ 😦 https://github.com/dagger/dagger/pull/5557/commits/1807036ff2c4a71e0878eecaed2c6497b68de2b1

[thorn moat]

@rocky harbor do you have a moment to https://github.com/dagger/dagger/pull/5557? Looking to get this finally merged, tired of dealing with merge conflicts 😵‍💫 - docs are "done enough," better to just continue in a separate PR at this point

[rocky harbor]

Yep! I have one other quick fix thing to finish, I'll take another quick look in a few mins

[wintry prism]

Going to build a quick staticcheck module.

If I did something like WithStaticcheck(*Container) (*Container) I could install via go install but that would mean I need to assume the container provided has golang. Otherwise, I could do a dance to get a binary in there. Or I could instead do something like Staticcheck(*Directory) ... that assumes a directory with a golang project (though it will return 0 in a random directory, which is :/)

What makes most sense for linters (or other checks) in general? Did the previous dagger checks work yield lessons?
(disclaimer, my brain/body are still on UTC+1 though I'm supposedly here in UTC-7 land)

[rocky harbor]

If I did something like `WithStaticcheck

[thorn moat]

services v2 is merged

[wintry prism]

https://github.com/jpadams/daggerverse/tree/main/goVersion
uses 👆 https://github.com/jpadams/daggerverse/tree/main/staticcheck (just started)

[wintry prism]

services v2 is merged :elmofire:

[deft rain]

What are people's thoughts on being able to query "the constructor" for an object?

e.g. I have a GitRepository - I want the remote that it comes from, but it feels like a specific case of something more general (I want the http URL that a File came from, etc)

I think it's slightly more apparent with Zenith - e.g. I have a github module that takes "git urls", but I'd love to be able to just take core git objects, and grab the url from there (I didn't construct the input type, so I don't have access to the constructor). Although mayyybe that's against the spirit of things 😄

[rocky harbor]

I think it's slightly more apparent with

[wintry prism]

Is it okay to write something like this where the action is taking place inside the go sdk runtime container rather than in a golang Base or similar?
https://github.com/jpadams/daggerverse/blob/main/goVersion/main.go

[rocky harbor]

Yes, I think it's probably one of the more interesting features Zenith enables; you don't have to wrap your code up in a WithExec to call it, you can just write it 😎

It won't come into full fruition until we add more cache control (which isn't scoped for Kubecon); so right now the only downside is that it will be uncached when first invoked in a session (cached on any subsequent calls in the session). But once we add the full cache control, you should be able to make even that code fully cached based on the inputs

[spiral whale]

I just discovered that I can use a local path for module dependency. It's baffling to me that I haven't noticed it until now. 😅 😬

[rocky harbor]

We probably have a docs bug then 😄 Fortunately we're starting back up on fleshing the Zenith docs out early next week, so hopefully that will help with things like this

[wintry prism]

Hi All! Project Zenith Community Call happening in the ⁠#911305510882513037 channel right now! 🙂

[spiral whale]

No, I think I'm the buggy one. I read early version of the docs before but didn't checkout updates later on 😄

[warped canyon]

failure on sync/init on 0.9.0: Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to create module from config ::.: failed to get runtime: failed to import go module sdk tarball: no manifest for platform linux/arm64 and tag

maybe I'm too early 🙂

[spiral whale]

probably, I get same error as well but then realise SDK releases didn't finish yet

[rocky harbor]

failure on sync/init on 0.9.0: `Error:

[kind carbon]

@rocky harbor, I seem to have a limit on 12 functions in a module. 13 and it breaks:

Error: query module objects: returned error 502 Bad Gateway: http do: Post "http://dagger/query": net/http: HTTP/1.x transport connection broken: EOF

That's on the query for fetching the module's objects. You think that's returning a response that's too large? I'll probably have to cut down and make more queries.

Nope it's not the query. 😅

Even if I select only name, it's the same thing.

[spiral whale]

I just created this issue https://github.com/dagger/dagger/issues/5937 and it seems it exists in 0.9.0 as well. Do I making something wrong?

{
  "name": "gale",
  "sdk": "go",
  "root": "../.."
}

Shouldn't this dagger.json needs to import parent directory as dag.Host().Directory(".") it's kinds blocker for me to convert gale since I'm not able to access source code

[rocky harbor]

Got back on the issue!

[rocky harbor]

@Erik Sipsma, I seem to have a limit on

[spiral whale]

I'm not able to use my module remotely at the moment. I'm unsure if this is a bug or I couldn't do it properly. What is the correct way to call a module repository using subpath daggerverse/gale

[ocean escarp]

so since dagger version 0.9.0, sdk runtimes will all be included in the dagger engine container, no more need to pull an external sdk runtime image?

[upbeat herald]

Dagger shell with Zenith is absolutely amazing

[strong ingot]

@kind carbon ?

[kind carbon]

Yes, that’s right.

But only for built-in ones.

[wintry prism]

push your changes! I want to try 🙂

dagger shell -m github.com/quartz-technology/daggerverse/redis cli --entrypoint /bin/sh

[kind carbon]

Fyi, I'm about to push several fixes for the cli.

[upbeat herald]

I did a special package called redis-shell, not published, if you directly use daggerverse/redis, you will first need to setup the redis server/cli with password and stuff

But I did that: https://github.com/quartz-technology/daggerverse/tree/main/redis-shell

Unpublished because it's not usable except for testing dagger shell, but it shouldn't be imported, feel free to clone the repo to try it 😄

[wintry prism]

Fyi, I'm about to push several fixes for

[rocky harbor]

@ocean escarp We did the initial setup required to support including the built-in SDKs w/ the image, but it started getting way out of scope of the original purpose of the PR it was included in (which was just to get the Python SDK Module working), so it's not complete yet.

Issue for finishing removal of internet deps to run builtin SDKs here: https://github.com/dagger/dagger/issues/5913

[quick wind]

I realise it's only been a few days since I asked but are these updates likely to make it into 0.9.1?

[kind carbon]

Codegen? I think so.

[restive shore]

Where can I find the zenith getting started guide? It would be a useful pin on his channel.

[rocky harbor]

Current Experimental Zenith Docs: https://docs.dagger.io/labs/project-zenith/

[rocky harbor]

Agree we should pin it; I'm not sure if I have the privilege to do that? Or I'm just missing where the "pin message" button is 🙂 cc @sharp zealot @sleek nest in case one you have the admin rights to pin it

[warped canyon]

Playing around with module DX without extending core types. Here's how that looked over a couple of iterations:

Original DX with the Golang module extending the Container type for GoTest:

func (g *Greetings) UnitTest(ctx context.Context) (string, error) {
    b := dag.
        Golang().Base("1.21").
        WithDirectory("/src", project()).
        WithWorkdir("/src").
        GoTest([]string{"./..."})

    return b.Stdout(ctx)
}

First update to the module to keep the API about the same but without extending core types:

func (g *Greetings) UnitTest(ctx context.Context) (string, error) {
    base := dag.
        Golang().Base("1.21").
        WithDirectory("/src", project()).
        WithWorkdir("/src")
    b := dag.Golang().Test(base, []string{"./..."})

    return b.Stdout(ctx)
}

Didn't like the look of that much, so I updated the module's API to embrace a world without extending core types:

func (g *Greetings) UnitTest(ctx context.Context) (string, error) {
    return dag.
        Golang().
        WithProject(project()).
        Test([]string{"./..."}).
        Container().Stdout(ctx)
}

Where I landed feels better than the original DX, IMO. Especially with the Container() accessor letting you basically chain back out to the core API. Here's the module: https://github.com/kpenfound/dagger-modules/blob/main/golang/main.go

[sharp zealot]

https://tenor.com/view/good-good-good-palpatine-emperor-palpatine-lightsaber-gif-19907846

[rocky harbor]

Looks good, I am curious to see how annoying the wrapping/unwrapping pattern gets once we have modules with lots of deps that require wrapping/unwrapping all over the place, but that's sort of par for the course at least in Go, so not that big of a concern.

One nitpick: this is currently encoding an assumption that .Test() is implemented by running a single Exec on a Container, which is why .Container().Stdout() works as expected.

That's not too crazy an assumption in this particular case, but you could imagine that the implementation changes to use multiple execs or even do something else entirely like running go test direct in the module container (once the runtime container is more easily customizable).

So I think it might be better to encourage patterns where you return objects with the intended API, with the unwrapper methods like .Container only used as an escape hatch. So in the Golang module it would be more like:

type TestResult struct {
   // fields...
}

func (mod *Golang) Test(...) *TestResult {
   // ...
}

func (res *TestResult) Output() string {
   // ...
}

That way if the implementation of Test updates, the implementation of Output can also be updated and callers don't ever need to change anything.

[warped canyon]

Yeah totally agree. I suspect my return types and struct properties will change quite a bit once I start mixing in more modules

I added a WithContainer() function to the golang module too, so there's potential to break in/out of the module multiple times

[spiral whale]

A noobie question, how do we export files and directories to host machine using dagger call?

[rocky harbor]

There's a separate command for that, dagger export dagger download

Has same behavior in terms of flags and calling conventions as dagger call, just a different verb to indicate you want to export what the function returns rather than just call it and see the output

[spiral whale]

🙏 it seems name is download just to keep for future referance. dagger download cluster kubeconfig --internal --export-path ./kubeconfig able to export file. 🚀

[rocky harbor]

Oh sorry you're right, I have no idea why I had dagger export stuck in my head 😅 I guess because it's the internal api

[spiral whale]

please don't read my previous message. I forget to remove my own flag 😄

[spiral whale]

same here, I read code couple of times but didn't hit the spot. I was searching for export as well 😄

[sharp zealot]

I’ve actually wondered if we should make Container.StdX return the concatenation of all execs for the container, instead of just the last one

[sharp zealot]

not too late to change the verbs… I don’t think the UX paint is dry yet

[rocky harbor]

Can just make export an alias of download too. I think my confusion is just due to the fact that I have insider knowledge and was thinking about the graphql api, which is called export. For someone new just using the dagger cli, download is more immediately obvious I'd suspect.

[rocky harbor]

There's times where you'd want that and times where you wouldn't, so it's hard to say. I guess the more general issue I am getting at there is just that users shouldn't need to make any assumptions about the Container being operated on internally. So that includes Std* apis but also pretty much all of the other ones too. Instead, modules should create their own types that expose exactly the APIs they support (i.e. TestResult w/ a Output field)

Another example might be a future update to Golang.Test that splits test execution across multiple container Execs in order to get better parallelization/caching. At that point, there isn't even a single Container to call Stdout on.

[wintry prism]

pinned!

[warped canyon]

For the Golang.Test function specifically, I'd probably either grab the stdout in the function and store it in a struct in the module, or redirect the stdout/stderr into a file and reference that. In a case where my whole pipeline executes in one container, I could see some value in having an easy way to get all of the stdout, but I don't know how common that will be

[spiral whale]

tried shell first time with kind and k9s and loved it

export DAGGER_MODULE=github.com/aweris/daggerverse/kind
dagger call cluster --name  kind-with-dagger create
✔ build "dagger call" [1.71s]
├ [1.45s] loading module
├ [0.26s] loading objects
✔ running "dagger call cluster create" [12.53s]
┃ cluster kind-with-dagger created
• Engine: 61b34bf9d33e (version v0.9.0)
⧗ 15.23s ✔ 79 ∅ 13
dagger shell k-9-s --name kind-with-dagger
✔ build "dagger shell" [1.69s]
├ [1.43s] loading module
├ [0.26s] loading objects
✔ running "dagger shell k-9-s" [32.21s]
✔ shell [29.68s]
• Engine: 61b34bf9d33e (version v0.9.0)
⧗ 34.95s ✔ 100 ∅ 10

[wintry prism]

Doing a quick docs refresh!

[wintry prism]

translating examples from GraphQL and dagger query to dagger call (leaving both in for now)

Do we need two calls to dagger call to handle the multiple values returned by this gql query?

dagger call hello-world --message "I am a potato" message
dagger call hello-world --message "I am a potato" from

echo '{potato{helloWorld(message: "I am a potato"){message, from}}}' | dagger query

[rocky harbor]

I believe so at the moment. I think it would also be reasonable for the CLI to print every "trivial" field on an object if the user doesn't select any, but I don't believe we've implemented that right now

[wintry prism]

https://github.com/dagger/dagger/pull/5956/files

[thorn moat]

This would be really cool, since you could interactively traverse the DAG (by repeating commands). Could be worth showing all the functions available too.

[rocky harbor]

Oh yeah, that reminds me of all the ideas from way back when of integrating the full interactive tui with everything. Can walk the whole dag, evaluating as you go, some tmux-like thing for viewing output of different execs etc

I want the tmux thing specifically to be able to reimplement bincastle someday 😄

But yeah, to start just being able to distinguish trivial resolver fields from “expensive” ones would be helpful here

[sharp zealot]

I think it’s reasonable to print all the scalar fields if the query ends in a struct. I wouldn’t worry about “expensive” fields until that actually becomes a problem

Also we should talk about dagger print as a more specialized version of dagger call with less noisy output and perhaps more formatting options

[rocky harbor]

The thing I wouldn't want is a function that is, e.g., implementing a deployment of something and returns a string to be called as part of that. To be clear, by "expensive", I just mean implemented as an actual resolver w/ code rather than as a trivially selected field in the parent object.

[deft rain]

Have we considered putting the main contents of the go codegened dagger.gen.go into the internal package? There's a lot of private fields on objects now, including querybuilder things, and it would be nice if we could have the boundary properly defined.
For accessing dag easily, we could still generate a top-level dagger.gen.go that creates a little helper.

[kind carbon]

Main thing is it was just faster to do it this way, but we know we want to put as little as possible in the user's host. We still need to be compatible with non zenith runs so the current codegen in zenith just adds a bit to the dagger.gen.go that's generated by mage and shipped with the SDK releases. At one point the querybuilder package was moved into internal but I had to move it back out to use it in the CLI. The zenith codegen can improve for sure, we just thought it's a lesser priority for now. Still, ideas on how to make it better are welcome at any time. As a reminder, codegen on the host is only needed for the IDE, and having LSP work correctly. It'll still be generated on the fly inside the runtime container so it works without those files locally.

[sharp zealot]

Hey now that there is a Service type, would it make sense to support passing it from the zenith CLI as a URL? 😍

I can't think of a specific killer use case (yet) but it feels like it could unlock a bunch of powerful stuff

dagger call integration-tests --db localhost:4242

[wintry prism]

Had a quck bit of fun to convert https://docs.dagger.io/757394/use-services/#example-mariadb-database-service-for-application-tests to Zenith 🙂

Broke up into a Drupal Base(), a MariaDB AsService(), and a Drupal Test Run()

https://github.com/jpadams/daggerverse/blob/main/drupal/main.go
https://github.com/jpadams/daggerverse/blob/main/mariadb/main.go
https://github.com/jpadams/daggerverse/blob/main/drupalTest/main.go

{
  "name": "drupalTest",
  "sdk": "go",
  "dependencies": [
    "github.com/jpadams/daggerverse/drupal@552d439f15073b62054c46f5e072be6d18b74c90",
    "github.com/jpadams/daggerverse/mariadb@552d439f15073b62054c46f5e072be6d18b74c90"
  ]
}

One bit I found confusing was that in Drupal().Base(ctx) the Container().Sync(ctx) returns (*Container, error) https://github.com/jpadams/daggerverse/blob/main/drupal/main.go

But by the time you go to use it in DrupalTest().Run() it doesn't want you to use Drupal().Base(ctx) but instead Drupal().Base() and drops error from the return of Base() too. If you try to use it in the original form, you get an error:

cc @manic lynx 👆 Instant Dagger demo with dagger call -m github.com/jpadams/daggerverse/drupalTest run

[manic lynx]

Hello. 🙂

So dagger call is for remote urls? And ragger run is for local?

[wintry prism]

dagger call, dagger up, dagger shell, etc are part of the new Project Zenith experience for running Dagger from the CLI and creating reusable, composable, multi-language modules 🙂

[manic lynx]

Got it!

[wintry prism]

Another fun example above: #daggernauts message

export DAGGER_MODULE=github.com/aweris/daggerverse/kind
dagger call cluster --name  kind-with-dagger create
dagger shell k-9-s --name kind-with-dagger

[manic lynx]

Sweet!

[wintry prism]

As folks create modules, we're starting to put them here: https://daggerverse.dev/ (some early docs there).

dagger run is still great for running your Dagger pipelines written in the current mode as it gives you the TUI and makes it easy to connect to Dagger Cloud, etc.

[whole maple]

Is there any option to print log statements inside a module? Sometimes it would very useful to print out additional infos during a dagger call ...

[spiral whale]

I think you're talking about the focus flag. just sent --focus=false to see the logs or --progress plain to disable TUI

[whole maple]

--focus=false works like a charm 🙂 Thanks

[spiral whale]

I made changes in the KinD module and to this example no longer valid for main branch. it needs to use following DAGGER_MODULE to work correctly

export DAGGER_MODULE=github.com/aweris/daggerverse/kind@66c7436dbc296e319bfa9d27236b8e9d5da2015c
dagger call cluster --name  kind-with-dagger create
dagger shell k-9-s --name kind-with-dagger

Updated README for anyone interested: https://github.com/aweris/daggerverse/tree/dc1c6d7de0aed058a9e5c32fc32c6d84a847569b/kind#commands

[wintry prism]

@rocky harbor @warped canyon @kind carbon am I confused or is this the way that the machinery works to make module authoring easier in the Go SDK by making context and error optional in some cases? Seems to only be affecting me in the "main" module that uses the two "utility" modules. It seems like I'm being required to drop context and error in my "main" module in this case (not an option to keep them).
https://docs.dagger.io/labs/project-zenith/

[kind carbon]

If you remove .Sync(ctx) you can remove the context parameter and the error return.

[wintry prism]

Yes, I remember there being discussion about that, but I couldn't find the reference. Why would someone use Sync or not use it? Apparently it's not needed in order to return type *Container

So it keeps things lazier?

To leave it off?

[kind carbon]

No, it's never needed if that's what you finish the function with.

[wintry prism]

hmmm

[thorn moat]

Maybe a good rule of thumb: if you don't know whether you should Sync you probably don't need to - especially when building a module to be consumed by someone else

[thorn moat]

What you're running into with Base() vs Base(ctx) is the fact that the client code doesn't reflect the server-side implementation's signature; it'll always be "lazy" just like the rest of the SDK. This is 100% intentional, and the same way the existing SDK codegen works. All those methods like Container.WithMountedDirectory are backed by resolvers that take a context.Context and return an error, but that's invisible through the GraphQL API

[wintry prism]

Ok, now I'll drop that and push it.

[warped canyon]

Yeah I don't know where the pattern of putting Sync(ctx) in functions started, I'll admit I put it in a few functions as well just following that convention, but we really shouldn't

[wintry prism]

now to try to dagger mod sync on drupalTest and see what happens

err

dagger mod use the latest version of drupal

[thorn moat]

you could probably point these to local paths instead of having to push first

try setting "root": ".." and "dependencies": ["../drupal", "../mariadb"]

[wintry prism]

but I want to simulate the experience of building a "main" module from "utility" modules I don't control

[thorn moat]

ah ok 👍

[wintry prism]

mod use and mod sync ok!

[warped canyon]

when you sync from your "main" module it should update your dependency as well, right? so you don't have to use the new version

[wintry prism]

but how do I stay pinned to a dep version if I want to? Didn't try it that way, so I don't know if it auto-uses the latest version

[wintry prism]

yeah, clearly we need to more clearly document exactly when to use it and why. Plus @thorn moat 's rule of thumb 😇

Working in simpler form! Thanks all! 🙌

dagger call -m github.com/jpadams/daggerverse/drupalTest run

I was really pleasing to break up the (small) wall of code in the example into three tiny modules, I must say 😄
Just a toy example at this point, but can imagine growing the utility modules up to their final form.

[wintry prism]

One bit of confusion was during initial module creation. I wanted to have a cool name for my MariaDB module so I used
dagger mod init --name=mariaDB --sdk=go but I put it in a directory on disk, that I subsequently commited to git, named mariadb, and then the name of the module in Golang is MariaDb.
So I need to dagger mod use github.com/jpadams/daggerverse/mariadb to keep consistent with GitHub and then use MariaDb in my code:
https://github.com/jpadams/daggerverse/blob/main/mariadb/main.go#L3
The original "cool" name is known only to me...😂

[warped canyon]

on the flip side, I think it's probably less confusing for module consumers since name casing will be globally consistent

[wintry prism]

yep, I think I was most surprised/vexed by DB -> Db, tbh

[restive shore]

I agree. I've put sync within functions (non zenith), in my SDK because I expect the functions to complete a unit of work. Letting the consumer decide when to use sync could confuse them when the functions don't do anything without them explicitly calling sync . A problem that can be solved with good documentation for sure.

[warped canyon]

Yeah, totally depends on the function. Some are more of a component of a unit of work, like a base image or an intermediate step. I think those cases probably shouldn't Sync()

[restive shore]

Can we please add this to the zenith docs? This was super useful

[restive shore]

Just noticed. The example main.go created with dagger mod init has a sync(ctx) in it. So that could be a good place to add some comments or another example function that doesn't do sync.

[sharp zealot]

Yeah sync should be removed from the default

good catch

[rocky harbor]

fix for that^ https://github.com/dagger/dagger/pull/5963

[sharp zealot]

First attempt at using 0.9 Zenith commands, getting a cryptic new error:

% dagger version
dagger v0.9.0 (registry.dagger.io/engine) darwin/arm64

% dagger -m github.com/shykes/daggerverse/helloWorld functions
▶ init
  ✘ connect ERROR [0.33s]
  ├ [0.33s] starting engine

⧗ 0.36s  ✔ 1 ✘ 1
Error: new client: remote is not a valid dagger server (expected "github.com/dagger/dagger", got "4d7c9f1e8edf")

[sharp zealot]

🙏

I really feel the need for dagger print (or equivalent). When I call a function with the primary goal of getting output to the terminal, I find dagger call very noisy and frustrating. I struggle to find the actual output in the noise; if I pipe it to another command, it's unclear if it works reliably

% dagger -m github.com/shykes/daggerverse/helloWorld call message
✔ build "dagger call" [1.86s]
├ [1.72s] loading module
├ [0.14s] loading objects
✔ running "dagger call message" [0.18s]
┃ Hello, World!                                                                                                 
• Cloud URL: https://dagger.cloud/runs/2cbfe452-872b-4d58-9670-d2207c051260
• Engine: 1b8f5d0852ba (version v0.9.0)
⧗ 3.35s ✔ 44 ∅ 10
% 

% dagger -m github.com/shykes/daggerverse/helloWorld call message | cat -e
✔ build "dagger call" [1.75s]                                              
✔ build "dagger call" [1.75s]
├ [1.61s] loading module
├ [0.14s] loading objects
✔ running "dagger call message" [0.31s]
• Cloud URL: https://dagger.cloud/runs/60441005-4870-471d-9822-23bc878d7931
• Engine: 1b8f5d0852ba (version v0.9.0)
⧗ 3.48s ✔ 44 ∅ 10
% 

[kind carbon]

Use --silent.

[kind carbon]

% dagger -m github.com/shykes/daggerverse/helloWorld --silent call message | cat -e

[sharp zealot]

Nice 🙂 thanks

I still would love a command that still shows me some TUI goodness while it's running, but then it all disappears once it's completed

• formatting options, and the ability to print as much as possible as text

ie. if I dagger print a container, maybe it could information about the container; etc

[warped canyon]

getting a codegen error using Optional on dagger@main 🧵

[sharp zealot]

Bikeshedding alert... dagger mod use ... shouldn't we change that verb to something more standard like add or install?

[warped canyon]

What's the dagger verb for H2C? serve? or is it not in yet?

[sharp zealot]

up?

[warped canyon]

yup that's the one, thanks

[sharp zealot]

Is it possible to tag a field to be serialized but not exposed in the API?

In other words, a struct field that is public in Go (so saved in the object state) but not public in GraphQL.

Example of the problem:

$ dagger -m github.com/shykes/daggerverse/helloWorld greeting
Error: input:1: helloWorld.greeting Cannot return null for non-nullable field HelloWorld.greeting.

I would love a way to either:

1. register a constructor function for my module (then I could initialize my fields, and wouldn't need to hide them in shame,

or

2. hide fields I can't properly initialize (then I can wrap them in accessors)

Has anyone seen this one before?

Error: query module objects: returned error 502 Bad Gateway: http do: Post "http://dagger/query": rpc error: code = ResourceExhausted desc = trying to send message larger than max (16799505 vs. 16777216)

https://dagger.cloud/runs/0d5ac4a1-d8a4-4267-b29c-70499ea80ed3

This was a very strange run... Sent one simple query in the local graphql playground, and it ran for a long time.. look at the run: https://dagger.cloud/runs/6fdab7ca-2074-41fb-8fad-5e934b00ded0

The query: ```graphql
{
supergit {
remote(url: "https://github.com/dagger/dagger") {
tags {
id
name
}
}
}
}

It's like somehow the same code in the same module became very expensive to run in the latest version

Here's the code of the tags function:

// Query the remote for its tags
func (r *Remote) Tags(ctx context.Context, opts RemoteTagOpts) ([]*RemoteTag, error) {
    var (
        filter *regexp.Regexp
        err    error
    )
    if opts.Filter != "" {
        filter, err = regexp.Compile(opts.Filter)
        if err != nil {
            return nil, err
        }
    }
    output, err := container().WithExec([]string{"git", "ls-remote", "--tags", r.URL}).Stdout(ctx)
    if err != nil {
        return nil, err
    }
    lines := strings.Split(output, "\n")
    tags := make([]*RemoteTag, 0, len(lines))
    for i := range lines {
        commit, name := tagSplit(lines[i])
        if filter != nil {
            if !filter.MatchString(name) {
                continue
            }
        }
        tags = append(tags, &RemoteTag{
            Name:   name,
            Commit: commit,
        })
    }
    return tags, nil
}

The query does return, with what looks like double-digit number of tags as result

It kind of looks like the module runtime is re-executed as many times as there are tags in the result?

That does seem to be the case... Filtering the tags so that the returned is shorter, seems to shorten the run time (and I believe the number of re-execs) in a way that feels proportional

[rocky harbor]

fixing sdk module generated code

[rocky harbor]

Based on the error message you most likely hit the bug that was fixed here 2 days ago: https://github.com/dagger/dagger/commit/8c33cee7b141269b05d59f48d5b2fed46bfdef52

So will be fixed in the release tomorrow

[rocky harbor]

The module shouldn't be re-exec'ing when resolving "trivial" fields like name (it only execs for functions), but due to the problem fixed in that commit above, I wouldn't be surprised if we were copying so much data while running requests that it became noticeably slow proportional to the amount of tags just because metadata in the context was getting copied around.

So worth another try tomorrow (or w/ ./hack/dev if brave)

[main jewel]

Hello, quick question regarding code organization. Should I create one module per job (like lint, test, etc) or I can create only one module for my project and expose multiple methods?

[kind carbon]

I’d create one module with multiple functions but it’s up to you really 🙂

Easier to manage.

[main jewel]

That was my first guess also, and here comes the naming 😄

[spiral whale]

Hey, I was trying to test dagger main to see what is changing for gale and I'm a little bit confused about the passing configuration. Did we remove passing Opts Struct support completely?

[deft rain]

yes, pretty much - the idea is that we want to eventually have all structs that the module declares be IDable (like container, directory, etc)

[spiral whale]

This is what I'm getting when I try to sync at the moment

dagger mod sync
✘ asModule(sourceSubpath: "daggerverse/gale") ERROR [1.51s]
• Engine: 04daf1dd8578 (version devel ())
⧗ 2.94s ✔ 153 ∅ 2 ✘ 1
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to install module dependencies: failed to install dependency "repo": failed to install module schema: schema validation failed: input:2029: Undefined type RepoOptsID.

      | ...
 2026 |         
 2027 |         
 2028 |         """
 2029 |         opts: RepoOptsID!
 2030 |     ): Directory!
 2031 |     """
 2032 |     
      | ...

[deft rain]

there is the potential to add opts structs back one day, but with a much more explicit way (instead of detecting if it's possible using pointers, etc)

[spiral whale]

ohh 😢 , then what about the default values?

[deft rain]

😢 there's a summary of the discussion here: https://github.com/dagger/dagger/pull/5907#issuecomment-1768953359

But, the problem is, there's no way in go to do proper default values - the problem gets worse when you imagine having default Container values, etc
To do default values, you need to use code - if you use an Optional, then you can do option.GetOr("defaultvalue"). The only difference is that this isn't understood at the engine level (so auto-docs generation won't pick up on it), but you can still have the default value.

[spiral whale]

I understand the reasoning and it makes sense. 🙏 Thanks for the explanation.

[deft rain]

I'm half hoping someone can come up with a good way that makes default values make sense in go 😦
But we couldn't think of something that felt natural for go programmers, and wasn't "too magical". One of the issues is we can't enforce the default values for intra-module function calls, so it does feel a bit confusing to have different behaviour inside and outside the module.

[thorn moat]

PSA: https://daggerverse.dev/ now requires a LICENSE file in the repo, so there's a whole lot of stardust this morning. The LICENSE file can either be in the module dir, or a parent dir, so if you just place one in the root of your repo you're good. Here's an example error with more details: https://daggerverse.dev/mod/github.com/vito/daggerverse/apko@506841fde4a8a38e5aaa475336d67cd4e2d5a0b8

[latent trellis]

I'm trying to convert an existing project still using mage to plain dagger call .

Currently, most of my (Go) projects have a ci submodule (an actual module) that I'm replacing with a new one generated using dagger mod init.

When I'm trying to run one of my build targets (dagger call --mod ci lint), the workspace becomes the ci subdirectory though and I don't have access to my actual project root.

I considered moving the dagger module to the project root which may work for application type projects (though I very much dislike the querybuilder package in the project root), it may not work well for libraries (I don't want dagger dependencies in my go.mod).

There is of course a chance that I'm missing something obvious. Any ideas?

[kind carbon]

You must have a Host().Directory() in your pipeline. Use a *Directory parameter in the function instead. Then in dagger call you can set the path to the directory for that parameter: dagger call -m ci lint --src ..

This decouples the app code from the pipeline code, so you can run that from anywhere.

[latent trellis]

Exactly. I guess that's a good workaround, though it would be nice to have like a dag.Workdir() or something that tells the module where dagger is being run from.

[kind carbon]

Another option is to use "root": ".." in your dagger.json. Then the module will have the parent dir's files, but that ties the module to the repo.

[latent trellis]

I don't mind that very much since this module is mostly for this specific project.

It may not fit perfectly into the current vision for Dagger, but I'm kinda using it as a task runner.

Instead of doing something like dagger call --mod github.com/.../some-go-build-module build I just want dagger call --mod ci build

Or even better, set DAGGER_MOD=ci in an env var and just call dagger call build

[kind carbon]

Yeah, than use "root".

[latent trellis]

One problem with that is it's gonna upload the entire directory as build context which I don't want (I have some large nix generate files in there). Is there an exclude option as well?

[kind carbon]

You can add exclude and include to dagger.json.

[latent trellis]

Nice!

[kind carbon]

They're defined here: https://github.com/dagger/dagger/blob/ed98d775ad923c56f130bbf5237fda0399fde1b4/core/modules/config.go#L25-L29

[latent trellis]

Are those relative to root or dagger.json?

[kind carbon]

Relative to root.

[latent trellis]

So dag.Host().Directory(".") should be the root, right?

Because....it's not

[kind carbon]

Maybe you need https://github.com/dagger/dagger/blob/ed98d775ad923c56f130bbf5237fda0399fde1b4/sdk/python/runtime/main.go#L106-L113 \cc @rocky harbor

[spiral whale]

no it's not. It's the module directory. I'm using https://github.com/aweris/gale/blob/d31b15785957e0dc621756d8ab56540b440d5711/daggerverse/source/helpers.go#L12-L18 this to get root from relative directory as workaround

[rocky harbor]

We need to rename root I think. Ultimately I think the actual model is pretty understandable, but I feel like the naming throws people off (feel free to disagree):

1. Your current working directory in the Module is always the same as the one containing dagger.json
2. By default, that's all that matters and all you need to be aware of
3. In the exceptional case where you need to load local paths that are "out-of-tree" (i.e. not a subdir of your current working dir), then you have to set what's currently called root
4. If you set root, your current working directory doesn't change in that it's still the directory containing dagger.json; the only difference is that the whole root gets loaded and your current working directory is now a subdir of that.
   • This results in the exact same directory structure you see on your local disk (up to the root path)

Would it help if we changed root to be named something like loadedParentDir? Maybe that would make it more clear that it's always going to be a parent dir rather than the "root" you are executing in or something

[latent trellis]

Couple ideas: projectRoot, contextDir

Context is already familiar from the container build terminology

[rocky harbor]

I forget if it has the same behavior though; in Dockerfiles if you reference path ., does that point to the root of your context dir?

[deft rain]

could we just map Directory(".") to the root? to me, that feels the most obvious behaviour.

[rocky harbor]

I think that results in more mental gymnastics ultimately, and potentially some technical problems. When I'm writing the code for my module, it's very simple if the paths I'm writing are actually relative to the file. So if I have a file foo.txt next to my main.go, I want to access it at ./foo.txt. I don't want to think about where I set my root and then do the path arithmetic to figure out what I need to reference in my code (and then update all my code if I have to change root)

[warped canyon]

That problem predates zenith, right? With the SDKs today you'd typically reference files relative to the workdir rather than where the dagger code is

Not saying we shouldn't change that, though. People trip on it all the time

[rocky harbor]

Yeah, I think Zenith gives us the opportunity to fix it since we now are actually loading that code into a container and have control over this sort of aspect

Basically, I want the Module container to mirror what I see on the host as closely as possible. I think that means that your workdir in the module container has to be the dir containing your module code.

For example, I have an IDE plugin that adds autocomplete suggestions based on files available relative to the source code file I'm editing. That wouldn't work if we set the workdir to be root. And obviously I don't care that deeply about one dumb IDE plugin 😄 It more just feels like evidence that the assumption of workdir being where your module source code is (rather than root) is on the right track.

If no one else agrees then obviously we should change it though 🙂

[thorn moat]

Agree with above - it's also very common for go test for example; tests always run from their package dir, making it easy to reference testdata etc.; making it a moving target is a pain in the arse

[restive shore]

Silly Q, I have an SDK i'm wrapping dagger Go SDK with. My plan is to slowly move sub components to a zenith module within the same repo. To access those external modules do I have to make my SDK a module as wel? I tried doing dagger mod use -m mymodule but that didnt' do anything (although it succeeded).

tldr: Can I access dagger modules from a non-module regular go module? I want to use the dag construct to call my module functions.

[latent trellis]

What do you think about this?

In a project, where I'm constantly adding --mod ci to all commands, an env var would help.

[kind carbon]

Try DAGGER_MODULE=ci

Or the abs path to it.

[latent trellis]

It works! Thanks!

[kind carbon]

direnv it 🙂

[latent trellis]

Already did 😉

[rocky harbor]

Silly Q, I have an SDK i'm wrapping

[latent trellis]

Thank you all for the assistance!

I managed to get my mage-based setup migrated to zenith/dagger modules: https://github.com/openmeterio/openmeter/pull/358

dagger call test

https://tenor.com/view/magic-gif-26166638

[sharp zealot]

Oh wait I just realized we removed optional arguments? 😦

[thorn moat]

see https://discord.com/channels/707636530424053791/1167084892723617912 - it's not gone, you use an Optional type now, but the ability to specify default values at the "schema level" is indeed gone (for now?)

[sharp zealot]

Oooh I understand now - thought that was only for fields. Thanks

[sharp zealot]

I got a little further with 0.9.1... But still stuck 😦

% dagger -m github.com/shykes/daggerverse/supergit functions repository with-remote
✘ load functions ERROR [1.99s]
├ [1.99s] loading module
┃ Error: failed to get loaded module ID: input:1: git.commit.tree.directory.asModule failed to cal
┃ module "supergit" to get functions: failed to get function output directory: process "/usr/local
┃ in/codegen --module . --propagate-logs=true" did not complete successfully: exit code: 1        
✘ asModule(sourceSubpath: ".") ERROR [1.03s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true ERROR [0.62s]
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error callin
┃ ModuleMainSrc: runtime error: index out of range [1] with length 1                              
┃ Usage:                                                                                          
┃   codegen [flags]                                                                               
┃                                                                                                 
┃ Flags:                                                                                          
┃   -h, --help             help for codegen                                                       
┃       --lang string      language to generate (default "go")                                    
┃       --module string    module to load and codegen dependency code                             
┃   -o, --output string    output directory (default ".")                                         
┃       --propagate-logs   propagate logs directly to progrock.sock                               
┃                                                                                                 
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error callin
┃ ModuleMainSrc: runtime error: index out of range [1] with length 1                              
✘ generating go module: supergit ERROR [0.25s]
├ [0.10s] go mod tidy
┃ writing dagger.gen.go                                                                           
┃ writing go.mod                                                                                  
┃ writing go.sum                                                                                  
┃ creating directory querybuilder                                                                 
┃ writing querybuilder/marshal.go                                                                 
┃ writing querybuilder/querybuilder.go                                                            
┃ needs another pass...                                                                           
• Cloud URL: https://dagger.cloud/runs/4e245e34-2e29-4e50-8a95-c17814c7d583
• Engine: 778c2c78e68a (version v0.9.1)
⧗ 3.38s ✔ 35 ∅ 4 ✘ 4

Hard to debug because I'm not getting a filename or line

[deft rain]

Already got a fix in https://github.com/dagger/dagger/pull/5972

Yeah the lack of filenames and stack traces is insanely annoying - I have a local thing to work around this, but go templates recover from panics without giving any useful info as to how to recover.

[sharp zealot]

It looks like we had a few regressions over the last 1-2 weeks, is that right?

[deft rain]

There's been a couple I think - part of it is that some of the paper cut fixes require more structural changes, and these affect the code in unexpected ways. And then we didn't have a super wide range of integration tests, that we've been adding as we fix those regressions to make sure it doesn't do it again.

Though, I think I'd have preferred to land the big opts change a bit less close to the release - I think that's just a bad timing thing from my side, sorry.

[rocky harbor]

Not your fault at all, that's just how the timing worked out. This is just gonna happen when we're moving quickly and iteratively. Like you said, we're expanding our set of integ tests every fix we make, but the amount of cases there are to cover is enormous; small differences here and there can be easily missed and require follow ups to fix and add coverage for.

I think the fundamental problem is that working fast+iteratively like we are is contradictory w/ being reliant on waiting for official releases to get everything out to users who are also working fast+iteratively on stuff.

The thing is, we build+publish both engine images and CLIs on every push to main. There's improvements to be made here, but let me see how hard it is for someone to use those. It might be a one-liner (much easier than ./hack/dev stuff)

[rocky harbor]

To use the engine w/ Justin's fix in place, you can use the engine built off main like this:

export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://registry.dagger.io/engine:main@sha256:80be3e5462f52fd2203265ee019c7b6f36071c71ca5c8b1d85bd872866475742
dagger -m github.com/shykes/daggerverse/supergit functions repository with-remote

Not ideal, but it's at least just a single env var. Maybe it's worth doing another patch release tomorrow; I'll ping Gerhard (just don't want to here atm since he just logged off for the night a little bit ago).

[rocky harbor]

Either way, running that command I get an actual expected error due to the updates we made to remove the special opts type from the Go SDK. I think you'll need to just update this code to directly accept Filter as an arg rather than wrapping it in the struct: https://github.com/shykes/daggerverse/blob/main/supergit/remote.go#L49-L83

The updated docs have an example: https://docs.dagger.io/labs/project-zenith/#creating-your-first-module (search for For example, to take multiple parameters (some of which can be optional))

[restive shore]

Found out that you can't name the module "go"

~/G/g/F/P/go ❯❯❯ dagger mod init --name Go --sdk go                                                                                                         bugfix/prevent-stack-smashing ✱ ◼
✘ asModule(sourceSubpath: ".") ERROR [0.92s]
✘ .....
✘ generating go module: Go ERROR [0.09s]
├ [0.02s] go mod tidy
┃ writing dagger.gen.go
┃ writing go.mod
┃ writing go.sum
┃ writing main.go
┃ creating directory querybuilder
┃ writing querybuilder/marshal.go
┃ writing querybuilder/querybuilder.go
┃ panic: go: internal error: missing go root module
┃
┃ goroutine 1 [running]:
┃ cmd/go/internal/modload.mustHaveGoRoot(...)
┃         cmd/go/internal/modload/buildlist.go:104
┃ cmd/go/internal/modload.newRequirements(0xe0?, {0x4000218b40?, 0x543be0?, 0x40001720f0?}, 0x5aedc0?)
┃         cmd/go/internal/modload/buildlist.go:118 +0x584
┃ cmd/go/internal/modload.updateUnprunedRoots({0x71df30?, 0xab31e0?}, 0x0?, 0x40000f20a0, {0x0, 0x0, 0x3d0f40?})
┃         cmd/go/internal/modload/buildlist.go:1465 +0x820
┃ cmd/go/internal/modload.updateRoots({0x71df30?, 0xab31e0?}, 0x0?, 0x40000a80c0?, {0x0?, 0x2?, 0x40000a80c0?}, {0x0?, 0x0?, 0x40000a80d0?}, ...)
┃         cmd/go/internal/modload/buildlist.go:781 +0x6c
┃ cmd/go/internal/modload.loadModFile({0x71df30, 0xab31e0}, 0x40000fe280)
┃         cmd/go/internal/modload/init.go:939 +0x134c
┃ cmd/go/internal/modload.LoadPackages({0x71df30?, 0xab31e0}, {{0x0, 0x0}, 0x4000119ef0, 0x1, {0x0, 0x0}, 0x1, 0x1, ...}, ...)
┃         cmd/go/internal/modload/load.go:345 +0x31c
┃ cmd/go/internal/modcmd.runTidy({0x71df30, 0xab31e0}, 0x40000ea4c8?, {0x400009e1a0?, 0x54c8e0?, 0x10edbc?})
┃         cmd/go/internal/modcmd/tidy.go:127 +0x204
┃ main.invoke(0xa70ea0, {0x400009e1a0, 0x1, 0x1})
┃         cmd/go/main.go:268 +0x4f0
┃ main.main()
┃         cmd/go/main.go:186 +0x754
┃ needs another pass...
• Engine: c8919d4606c0 (version v0.9.1)

[rocky harbor]

That's good to know, creating an issue. I'm actually not immediately sure exactly where a conflict would happen with that name...

[restive shore]

One more.. the generated go.mod is go 1.21.2. But I am using go work and it sets the workspace version to my local go version which is go 1.21.3. Now when i do a dagger mod use ../mymodule I get the following. This is with engine and go SDK 0.9.1

Only thing that worked was downgrading my go.work. I tried upgrading the module go.mod to 1.21.3 but that didn't work. Is that because the go SDK the engine is using is 1.21.2?

[rocky harbor]

Yeah we are using 1.21.2, so that seems correct. I'll update us to 1.21.3 (we should anyways + it's a one line change), but in general I'm not sure there's a ton of great options here.

We don't want to override the users version of course.

We do want to support configurable go versions (and SDK parameters in general), but that is going to require we pull Go toolchains at runtime (we can't plausibly pre-package every patch version of Go in our engine image), which obviously can become problems for users who are contending w/ stringent corporate firewalls.

[rocky harbor]

I haven't use go.work enough to know off the top of my head, can you specify just 1.21 rather than 1.21.3? That might be simplest if it works

[restive shore]

let me try that with a dummy module

hmm that sort of worked? But not really. I manually set go.work to 1.21. But the dagger mod get failed with

~/G/g/F/P/dummy ❯❯❯ dagger mod use ../docker                                                                                                                bugfix/prevent-stack-smashing ✱ ◼
✘ asModule(sourceSubpath: "dummy") ERROR [0.49s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true ERROR [0.15s]
┃ Error: load package ".": err: exit status 1: stderr: go: module .. listed in go.work file requires go >= 1.21.0, but go.work lists go 1.21; to update it:
┃         go work use
┃ go: module . listed in go.work file requires go >= 1.21.2, but go.work lists go 1.21; to update it:
┃         go work use
┃ go: module ../go listed in go.work file requires go >= 1.21.2, but go.work lists go 1.21; to update it:
┃         go work use

After I did go work use

It set my go.work to

go 1.21.2

toolchain go1.21.3

use (
        .
        ./docker
        ./dummy
        ./go
)

[thorn moat]

Haha I ran into this exact same thing yesterday. You just can't have a go.mod with module go - it's a really cryptic error, would be fun to figure out what wires that's crossing in the Go stack. cc @restive shore - this could be avoided by not naming the go.mod after the module though. I mainly did that because it looks nice, and the name doesn't really matter, so we could maybe just decide on a static one

You can also just change the go.mod directly to have a different name, if you want to have the module named "go"

[rocky harbor]

Oh we name the go module in go.mod with the dagger module name? I didn't know that. Feel like we could also just name it main. It's not importable now so I think it's pretty much arbitrary (unless there's some go submodule nonsense at play)

[thorn moat]

Yeah, we can just go with main or something, it's not importable either way since it's package main, though you do have to know about it if/when your module has any internal helper packages, since the imports would look like "main/foo" which is a little odd

Either way you can literally just change the go.mod if you don't like it, since the codegen will pick up from an existing go.mod and use that for existing imports like the querybuilder

[rocky harbor]

Yep exactly, as long as they can change it then it makes sense to just default to something safe like main. I'll append fixing that to my queue

[latent trellis]

Anyone saw this error before?

3: build "dagger call" ERROR: failed to get loaded module ID: input:1: host.directory.asModule failed to call module "ci" to get functions: failed to get input file: failed commit on ref "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022": "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022" failed size validation: 162 != 24: failed precondition

3: loading module [8.36s]
3: build "dagger call" ERROR: failed to get loaded module ID: input:1: host.directory.asModule failed to call module "ci" to get functions: failed to get input file: failed commit on ref "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022": "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022" failed size validation: 162 != 24: failed precondition

Error: failed to get loaded module ID: input:1: host.directory.asModule failed to call module "ci" to get functions: failed to get input file: failed commit on ref "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022": "layer-sha256:46995db4b389b894d512d6d1114535e02adf710a54ca68837fead903aef5c022" failed size validation: 162 != 24: failed precondition

GHA

[rocky harbor]

No... that is an extremely deep buildkit error. It's saying a container image layer has an unexpected size.

1. Do you have DAGGER_CLOUD_TOKEN set w/ caching enabled? Or are you using one of the _EXPERIMENTAL_DAGGER_CACHE_CONFIG env vars to use upstream caching?
2. Is this reproducible? Or just a one off?

[deft rain]

A quick google implies this comes from containerd: https://github.com/containerd/containerd/issues/2306

[latent trellis]

I have DAGGER_CLOUD_TOKEN set, I have a few cache mounts if that's what you mean. I did not set. _EXPERIMENTAL_DAGGER_CACHE_CONFIG

Seems to be a one off (or rather happens inconsistently), but basically immediately after I merged my dagger module rewrite, it started happening.

Here is the workflow run for reference: https://github.com/openmeterio/openmeter/actions/runs/6659370631/job/18098290440?pr=362

Dagger cloud run (if you can see it): https://dagger.cloud/runs/488f1fc0-f16a-4747-baee-0a26d7ef406c

Another weird thing I see on Dagger cloud is that basically all builds timeout. I do have the docker stop thing in the GHA workflows though.

From what I can tell, it's always this step.

[ember walrus]

@latent trellis which engine of the version are you running? Are you using services in your pipeline?

[rocky harbor]

[bot] Re-generate Node.js client · openm...

[wintry prism]

This happening for anyone else who is updating a module's dependencies?

drupalTest ➤ cat dagger.json                                                                                                                                      git:main
{
  "name": "drupalTest",
  "sdk": "go",
  "dependencies": [
    "github.com/jpadams/daggerverse/drupal@687eda1df9ec2e72187365d6142adfec4e8b4485",
    "github.com/jpadams/daggerverse/mariadb@552d439f15073b62054c46f5e072be6d18b74c90"
  ]
}
drupalTest ➤ dagger mod use github.com/jpadams/daggerverse/drupal                                                                                                 git:main
✘ stdout ERROR [0.58s]
✘ git://github.com/jpadams/daggerverse#687eda1df9ec2e72187365d6142adfec4e8b4485 ERROR [0.57s]
┃ fatal: Not a valid object name 687eda1df9ec2e72187365d6142adfec4e8b4485^{commit}

Turns out that after a brutal rebase on my part that wiped out the SHAs I was using, now drupal@687eda1df9ec2e72187365d6142adfec4e8b4485 doesn't exist any more.

My intent is to update this to the current ref via dagger mod use github.com/jpadams/daggerverse/drupal or dagger mod use github.com/jpadams/daggerverse/drupal@e5470174bfd86b9858dd5fd3db381835b579a5b4

I can re-create the module of course as one option.

I move my go code out. Delete all the mod contents. dagger mod init, dagger mod use my deps, copy back my go code, dagger mod sync. That works.

But it seems I should be about to force an update even if the old module dep doesn't resolve. If the NEW module ref doesn't resolve, it should prob stop me or warn me 🙂

[rocky harbor]

Yeah it shouldn't fail like this, making an issue. There's definitely some corner cases here... like in theory it's possible to have completely different modules at different commits in a single git repo + subpath. That would imply calling dagger mod use github.com/jpadams/daggerverse/drupal could actually be expected to just append the newest commit to the list of dependencies rather than update the existing entry in the list of deps.

But that's probably such an obscure thing to want to do that maybe we should just ignore it and have the behavior be to replace the existing github.com/jpadams/daggerverse/drupal dep with the latest version.

[sharp zealot]

Following up...

 % export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://registry.dagger.io/engine:main@sha256:80be3e5462f52fd2203265ee019c7b6f36071c71ca5c8b1d85bd872866475742
% dagger -m github.com/shykes/daggerverse/supergit functions
✘ load functions ERROR [18.62s]
├ [18.62s] loading module
┃ Error: failed to get loaded module ID: input:1: git.commit.tree.directory.asModule.serve failed 
┃  install module schema: schema validation failed: input:2270: Undefined type SupergitRemoteTagOp
┃ ID.                                                                                             
┃                                                                                                 
┃       | ...                                                                                     
┃  2267 |                                                                                         
┃  2268 |                                                                                         
┃  2269 |                 """                                                                     
┃  2270 |                 opts: SupergitRemoteTagOptsID!                                          
┃  2271 |         ): [SupergitRemoteTag!]!                                                        
┃  2272 | }                                                                                       
┃  2273 | """                                                                                     
┃       | ...                                                                                     
✘ checkVersionCompatibility(version: "0.9.1") ERROR [0.00s]
✘ serve ERROR [0.00s]
• Cloud URL: https://dagger.cloud/runs/d6d62873-7cb3-40ee-b4a9-14fff58d7e21
• Engine: 656c662afeb1 (version main)
⧗ 1m15.6s ✔ 166 ✘ 3
%

[rocky harbor]

#daggernauts message <- I think that will fix it

[sharp zealot]

Can't tell if the real error is the "undefined type" or version compat check, or both?

[rocky harbor]

That one is an intentional backwards incompatible change

[rocky harbor]

Agree the error message is useless here either way

[sharp zealot]

Is the checkVersionCompatibility actually an error? Or just a warning maybe?

But thanks, I suspected it was related to optional args, I'll switch focus to fixing it 🙂

(It looks like we have error message / debugging experience polish work in our future)

[rocky harbor]

It's showing up as an error in the progress output, but allows everything else to execute. It's just happening because you're overriding the engine image to the one on main. I chatted w/ Gerhard, we will most likely do another patch release tomorrow to get these fixes out, so should go away then

[rocky harbor]

Yes, very much so 🙂

[sharp zealot]

Unfortunately I'm still getting the compat check error even after fixing the code

 % export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://registry.dagger.io/engine:main@sha256:80be3e5462f52fd2203265ee019c7b6f36071c71ca5c8b1d85bd872866475742
% dagger mod sync
✘ asModule(sourceSubpath: ".") ERROR [21.06s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true ERROR [7.27s]
┃ failed to check version compatibility: input:1: checkVersionCompatibility failed to parse engine version as semver: No Major.Minor.P
┃ ch elements found                                                                                                                   
┃                                                                                                                                     
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to convert m
┃ hod Tags to function def: failed to convert param type: invalid type: invalid type                                                  
┃ Usage:                                                                                                                              
┃   codegen [flags]                                                                                                                   
┃                                                                                                                                     
┃ Flags:                                                                                                                              
┃   -h, --help             help for codegen                                                                                           
┃       --lang string      language to generate (default "go")                                                                        
┃       --module string    module to load and codegen dependency code                                                                 
┃   -o, --output string    output directory (default ".")                                                                             
┃       --propagate-logs   propagate logs directly to progrock.sock                                                                   
┃                                                                                                                                     
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleMainSrc: failed to convert m
┃ hod Tags to function def: failed to convert param type: invalid type: invalid type                                                  
✘ checkVersionCompatibility(version: "0.9.1") ERROR [0.00s]
✘ generating go module: supergit ERROR [5.38s]
• Cloud URL: https://dagger.cloud/runs/49ffd002-50f8-4b44-b499-453e75e677dc
• Engine: d88eaa6d1c7b (version main)
⧗ 28.96s ✔ 18 ✘ 4
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to call module "supergit" to get functions: failed to get function output directory: process "/usr/local/bin/codegen --module . --propagate-logs=true" did not complete successfully: exit code: 1

[rocky harbor]

The compat check error can be ignored, it shows up as error but is just a warning. The real error is the one about invalid type.

Do you have the latest code pushed? I'll take a look at it. There was another set of fixes just merged, can try with that.

Also, I'm going to copy paste your code into our integ test suite since it seems to be catching quite a bit 😄

[deft rain]

This seems like Kyle's error from yesterday - I know he resolved this by removing dagger.gen.go

There's something weird going on with how we do go package parsing I think

[sharp zealot]

lol

ok I'll remove

Ah, not I get real go errors 🙂

with real line numbers etc

Question! How do I check for an unset Optional[string]? Just empty string?

[deft rain]

If you call Optional.Get then it returns the string and a bool that's true if it was set and false if it wasn't - there's also a helper .GetOr that returns the provided default if it wasn't set.

These need doc comments actually, I'll do that ASAP tomorrow

[sharp zealot]

Oh!

OK, thanks

How do I provide the default?

[deft rain]

Optional.GetOr(default) should do it

[sharp zealot]

dagger functions could use a little polish... Is anyone already working on that? Otherwise I might make it my stretch goal contribution

[rocky harbor]

Helder is focusing on Python modules, so no, have at it 🙂

[thorn moat]

Out of curiosity: has anyone had any use cases for an interactive module? Specifically prompt-style request-responses; cases where dagger shell doesn't quite cut it. I'm planning to implement support for prompts in Progrock so that we can do basic interactive things on the CLI, and supporting it at the protocol level is interesting to think about, but it might be a terrible idea.

[sharp zealot]

Yes but of the async variety

Lots of workflows require eg. a manual approval step, which may take days to secure. Not having a primitive for that leaves some Dagger pipelines doomed to being forever sliced up within some other system that does

[ember walrus]

I put my feedback in the thread Alex started above 🙂

Agree with you

[sharp zealot]

Are we sure the .gitignore works? I feel like my changes to generated files are not being ignored, and are in fact being committed and pushed. Or is that the idea?

Is it possible to pass an array of strings as an argument?

do I json-encode it?

[wintry prism]

here's an example I haven't tried
https://daggerverse.dev/mod/github.com/aweris/daggerverse/gh@0640e36c1a3e3be5bc8becb2c1a95ced6683002e

hmmm errored out for me...

dagger call -m github.com/aweris/daggerverse/gh --help

[warped canyon]

Can anyone think of a more clever way to expose 2 services to the host with dagger up other than creating a proxy module that forwards both services on different ports?

[wintry prism]

here's an example I haven't tried

[rocky harbor]

So I was adding what I thought would be a very simple integ test where we init+call a module with a dependency, change the dependency, and then call again to verify we get the new output from the updated dep.

Turns out this doesn't work in the current module integ tests because of the way ExperimentalPrivilegedNesting works. That flag gives the exec dagger API access back to the engine, but it specifically also does it back to the same session as the original client.

This means that even when our integ tests do dagger query/call/mod/etc. calls across execs, it's all back to the same session. This breaks in the test case I'm adding (and all the more complicated ones we need to add) because you are trying to re-load modules back into the same session, which expectedly doesn't work.

Two relatively quick options to fix:

1. We can run the dev engine as a service inside the dev engine and use service bindings to point to it rather than using the ExperimentalPrivilegedNesting flag. I tried and while it works it's stupendously ugly and makes retaining cache across runs while not creating conflicts between parallel tests very difficult.
2. We can change the behavior of ExperimentalPrivilegedNesting to not connect back to the same session but instead create its own new one.
   • We still internally need the ability to have execs connect back to the same session in order for modules to work, but that can be a completely internal flag.

I'm currently going with 2 since it's fastest and best long term IMO, but it is technically a breaking change. It's such an obscure thing that I would be extremely surprised if any user even notices, but want to double check now if anyone has any objections or notices any flaws. @thorn moat @sharp zealot @deft rain (also IIRC the playground uses this flag but shouldn't be impacted by the behavior change, right @dense canyon?)

[sharp zealot]

Yeah 2. seems like the behavior I expected

otherwise you can’t really “virtualize” zenith

[thorn moat]

2 sounds good to me too, I've been aware of these semantics but can't remember if they're load-bearing anywhere off the top of my head; will keep thinking about it

I might be mostly thinking of the module use case, which it sounds like we'd keep, so 👍

[rocky harbor]

Yep it'll be an internal flag we can set on an Exec from our code in schema/module.go, but not in the graphql API. If a user wants that behavior, they have to use a module

[rocky harbor]

Pushed as a commit here since it's needed in order to write tests for that PR: https://github.com/dagger/dagger/pull/5964
(also finished updating the description to help reviewing, gonna squash remaining TODOs now)

[restive shore]

As someone currently utilizing ExperimentalPrivilegedNesting and not using modules would this break things? I'm using it to run tests using the dagger engine.

[sharp zealot]

Per my understanding, no it would break nothing

[rocky harbor]

No you should be good, though im actually somewhat surprised you’d need it for that use case, generally you should only need InsecureRootCapabilities for that

PrivilegedNesting is specifically needed when the exec wants access back to the dagger api serves by the engine its running in

[sharp zealot]

I think the daggerverse crawler is hitting the same error I did earlier... 😭

[thorn moat]

if this is fixed on main I can just do a quick bump

(assuming it's in the engine / SDK runtime / codegen)

[sharp zealot]

That would be amazing thank you. Yes it is fixed in main.

[thorn moat]

@sharp zealot ok, now it's just not happy that there's no LICENSE 😛

you can just put one in the root of your repo, it'll find it for all of them

[sharp zealot]

Ha ha thanks 🙂

@thorn moat I'm also stuck on another module which depends on supergit. It keeps erroring on a version of supergit that it doesn't match the latest (I think) but I can't fix it..

[thorn moat]

which module is that?

[sharp zealot]

github.com/shykes/daggerverse/dagger

 % dagger mod sync
✘ asModule(sourceSubpath: ".") ERROR [0.84s]
✘ exec go build -o /runtime . ERROR [0.40s]
┃ # dagger                                                                                           
┃ ./engine.go:59:19: dag.Supergit undefined (type *Client has no field or method Supergit)           
┃ ./engine.go:59:63: undefined: RemoteTagsOpts                                                       
┃ ./worker.go:223:34: cannot use registrySvc (variable of type *Container) as *Service value in argum
┃ t to worker.WithServiceBinding                                                                     
┃ ./worker.go:224:41: cannot use privateRegistry() (value of type *Container) as *Service value in ar
┃ ment to worker.WithServiceBinding("registry", registrySvc).WithServiceBinding                      
┃ ./worker.go:231:26: worker.Endpoint undefined (type *Container has no field or method Endpoint)    
┃ ./worker.go:231:40: undefined: ContainerEndpointOpts                                               
┃ ./worker.go:266:39: cannot use worker (variable of type *Container) as *Service value in argument t
┃ w.GoBase.WithExec([]string{…}).WithMountedDirectory("/app", dag.Host().Directory(".")).WithMountedD
┃ ectory(utilDirPath, testEngineUtils).WithEnvVariable("_DAGGER_TESTS_ENGINE_TAR", filepath.Join(util
┃ rPath, "engine.tar")).WithWorkdir("/app").WithServiceBinding                                       
┃ ./worker.go:267:34: cannot use registrySvc (variable of type *Container) as *Service value in argum
┃ t to w.GoBase.WithExec([]string{…}).WithMountedDirectory("/app", dag.Host().Directory(".")).WithMou
┃ edDirectory(utilDirPath, testEngineUtils).WithEnvVariable("_DAGGER_TESTS_ENGINE_TAR", filepath.Join
┃ tilDirPath, "engine.tar")).WithWorkdir("/app").WithServiceBinding("dagger-engine", worker).WithServ
┃ eBinding                                                                                           
• Cloud URL: https://dagger.cloud/runs/94ebf59c-0a98-4e9d-af17-0903aa689849
• Engine: 2ff796165b11 (version main)
⧗ 1.22s ✔ 13 ∅ 4 ✘ 2
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to call module "dagger" to get functions: failed to get function output directory: process "go build -o /runtime ." did not complete successfully: exit code: 1

[thorn moat]

some of that is legit - https://github.com/shykes/daggerverse/blob/e46c35c682d1a94f2ff867aa6439e9ac298a8f25/dagger/worker.go#L223 needs to be updated for services v2

some of it is also the removal of Opts struct support

[sharp zealot]

sorry wrong command

that was after manually removing the supergit dependency

 % dagger mod use github.com/shykes/daggerverse/supergit
✘ asModule(sourceSubpath: ".") ERROR [3.04s]
✘ checkVersionCompatibility(version: "0.9.1") ERROR [0.00s]
✘ exec go build -o /runtime . ERROR [0.22s]
┃ # dagger                                                                                           
┃ ./dagger.gen.go:4277:12: dag.LoadSupergitCommitFromID undefined (type *Client has no field or metho
┃ LoadSupergitCommitFromID)                                                                          
┃ ./dagger.gen.go:4277:37: undefined: SupergitCommitID                                               
┃ ./dagger.gen.go:4517:12: dag.LoadSupergitRemoteTagFromID undefined (type *Client has no field or me
┃ od LoadSupergitRemoteTagFromID)                                                                    
┃ ./dagger.gen.go:4517:40: undefined: SupergitRemoteTagID                                            
┃ ./engine.go:59:63: undefined: RemoteTagsOpts                                                       
┃ ./worker.go:223:34: cannot use registrySvc (variable of type *Container) as *Service value in argum
┃ t to worker.WithServiceBinding                                                                     
┃ ./worker.go:224:41: cannot use privateRegistry() (value of type *Container) as *Service value in ar
┃ ment to worker.WithServiceBinding("registry", registrySvc).WithServiceBinding                      
┃ ./worker.go:231:26: worker.Endpoint undefined (type *Container has no field or method Endpoint)    
┃ ./worker.go:231:40: undefined: ContainerEndpointOpts                                               
┃ ./worker.go:266:39: cannot use worker (variable of type *Container) as *Service value in argument t
┃ w.GoBase.WithExec([]string{…}).WithMountedDirectory("/app", dag.Host().Directory(".")).WithMountedD
┃ ectory(utilDirPath, testEngineUtils).WithEnvVariable("_DAGGER_TESTS_ENGINE_TAR", filepath.Join(util
┃ rPath, "engine.tar")).WithWorkdir("/app").WithServiceBinding                                       
┃ ./worker.go:266:39: too many errors                                                                
• Cloud URL: https://dagger.cloud/runs/c83d2d5d-fbd0-4339-9b90-c9235ee82c0f
• Engine: 2ff796165b11 (version main)
⧗ 1m19.1s ✔ 158 ∅ 3 ✘ 3
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule failed to call module "dagger" to get functions: failed to get function output directory: process "go build -o /runtime ." did not complete successfully: exit code: 1

mmm maybe it is the same

sorry I've been juggling a bunch of different error states, might be confusing myself

let me fix the legit stuff & see what's left

[thorn moat]

all good, there's some interesting other errors there too, could be something

[sharp zealot]

how do I pass a value to an Optional[...] argument when calling the function directly?

[thorn moat]

I think there's a constructor for it in dagger.gen.go

yep - Opt(foo)

[sharp zealot]

@thorn moat something's weird, it's like it's still using an old version of supergit ?

./engine.go:59:63: cannot use Opt("^v[0-9\\.]+") (value of type Optional[string]) as SupergitRemoteTagsOpts value in argument to dag.
┃ pergit().Remote(engineUpstream).Tags  

(side note some of that output is getting truncated in a weird way)

Just realized what time it is for you... Sorry for pinging!

[thorn moat]

dumb question: is it bumped? e.g. dagger mod use github.com/shykes/daggerverse/supergit@e46c35c682d1a94f2ff867aa6439e9ac298a8f25

[thorn moat]

np, I'm trying to stay up late to sleep on the flight 😛

[sharp zealot]

 % grep -r SupergitRemoteTagsOpts * | wc -l
       0

[sharp zealot]

I'm running this is from a dagger.json wiped of dependencies

So I'm assuming it's getting the last commit from github?

[thorn moat]

should be yeah (just saw the first line of the output)

maybe run with --focus=false to see which commit it's fetching?

(i will likely randomly disappear soon though, good luck if so)

[sharp zealot]

◀──┤ CACHED exec git ls-remote --symref https://github.com/shykes/daggerverse HEAD
█   │ [0.45s] git://github.com/shykes/daggerverse#main
┃   │ e46c35c682d1a94f2ff867aa6439e9ac298a8f25        refs/heads/main                                                                                             
█◀──╯ CACHED exec git rev-parse HEAD

oooooh

It's not a core llb Git, it's an exec of git... and it's cached 😫

looks like the right commit though

Giving up for tonight. Thank you for you help 🙏 and have a safe flight

[rocky harbor]

Oh yeah we should change that, it also results in another dep on pulling from dockerhub (which complicates users w/ corporate firewalls) https://github.com/sipsma/dagger/blob/79904d71aeaa44c279f89584548b361d62be0f1b/core/modules/resolver.go#L300-L300

Might be easiest to just add to our core git api?

[thorn moat]

might not even be needed now that you can get the commit from a ref

https://play.dagger.cloud/playground/qi0rk0te42b

this code was originally in Daggerverse where it made more sense to YOLO, should definitely be fixed

[rocky harbor]

Yeah that's what I was wondering at first, but i saw the code was for getting the default branch, which I thought was different, unless there's some way of using ref->commit to determine default branch

[thorn moat]

oh, yeah, we'll need that too. at least the ref => commit part is done (those are the two places alpine/git is used)

i've wanted a 'default branch' API before, it's awkward having to specify one all the time

I wonder if we should just start using the git CLI directly in the Git APIs at this point. We already need it in there for when Buildkit shells out to it. 😛

[rocky harbor]

Oh yeah that's what I figured we'd do for the default branch one since it's not in the upstream APIs (LLB or provenance) and I don't think it really makes sense for it to be

I think buildkit has a TODO to migrate to a go implementation of a git porcelain iirc, but just shelling out sounds more robust honestly

[thorn moat]

yeah I agree, would rather be looking at git man pages than a bespoke API tbh (go-git is nice, its API surface area is huge)

[rocky harbor]

Quick fix: https://github.com/dagger/dagger/pull/5987 Uses the new Git.commit API Justin just added where possible, and a cache buster for the default branch part for now.

[wintry prism]

Anyone seen this error for dagger mod sync? Error: module sync is not supported for git modules

trivyScan ➤ ls -al                                                                               git:main*
total 264
drwxr-xr-x  11 jeremyadams  staff    352 Oct 27 06:57 .
drwxr-xr-x  13 jeremyadams  staff    416 Oct 26 16:56 ..
-rw-r--r--   1 jeremyadams  staff     39 Oct 26 15:04 .gitattributes
-rw-r--r--   1 jeremyadams  staff  10758 Oct 26 15:04 LICENSE
-rw-r--r--   1 jeremyadams  staff     93 Oct 26 15:04 README.md
-rw-r--r--   1 jeremyadams  staff  95241 Oct 26 15:04 dagger.gen.go
-rw-r--r--   1 jeremyadams  staff     51 Oct 26 15:04 dagger.json
-rw-r--r--   1 jeremyadams  staff    192 Oct 26 15:04 go.mod
-rw-r--r--   1 jeremyadams  staff   2788 Oct 26 15:04 go.sum
-rw-r--r--   1 jeremyadams  staff    753 Oct 27 06:57 main.go
drwxr-xr-x   4 jeremyadams  staff    128 Oct 26 15:04 querybuilder
trivyScan ➤ cat .gitattributes                                                                   git:main*
/dagger.gen.go linguist-generated=true
trivyScan ➤ dagger mod sync                                                                      git:main*
• Engine: 64bf1cf27353 (version v0.9.1)
⧗ 2.89s ✔ 16 ∅ 1
Error: module sync is not supported for git modules

[deft rain]

do you have DAGGER_MODULE set by any chance

[wintry prism]

oooooh. Might indeed

That was it, thanks @deft rain !

(yeah the majority of the error message or bits to build it, didn't seem to be in our code base)

[deft rain]

we've changed the error message as well apparently: https://github.com/dagger/dagger/blame/4cb8bc93467c5e3162c0f111212443a3d7d88805/cmd/dagger/module.go#L171

which is why you probably couldn't find it 😢

[restive shore]

Tell me if this is the wrong place to ask questions like this.

A module's private fields will not be persisted.

As I understand (and seeing in practice) I can't use the WithFunction* pattern in go to set private fields of a module. Is there a different/preferred pattern for fields that I don't want to expose via a module? It's not clear in current docs.

One more question. Is this still true? (from the doc)

When referencing another module as a local dependency, the dependent module must be stored in a sub-directory of the parent module.

I was able to refer to a local dependency from a parent folder by setting root: .. in dagger.json. Maybe I'm understanding it wrong or more likely doing something wrong.

[deft rain]

nope this is the right place 🎉
for this one, looks like this was asked before: #daggernauts message
i'll open a tracking issue on github, since it's difficult to track things through discord

[restive shore]

Thank you! I am not sure I fully understand where that conversation landed in the end. It's probably because my lack of understanding of certain core decisions. I'll try to summarize what I understand,

All fields of the module need to be public to be used with the API. There is no way to hide certain fields and initialize them with initializer functions. Use Optional for defaults.

[deft rain]

yup, today all fields must be public - you can't hide them.
you also can't initialize them to defaults, there are two things here:

• we could maybe have a "constructor" thing for this, which we don't have
• you just can't do defaults elegantly in go, so it's difficult to do neatly

here's the issue i promised you: https://github.com/dagger/dagger/issues/5991

[restive shore]

Thank you! Makes sense. What about this? ^^

[deft rain]

I think this is https://github.com/dagger/dagger/issues/5914

woops

no it's this one: https://github.com/dagger/dagger/issues/5862

[warped canyon]

What's the latest on python modules? Do we have any docs on how it differs from Go today?

[rocky harbor]

@kind carbon is working on adding support for objects (right now you can only register a set of top-level functions), so I think the idea was to wait for that to be done before getting going on the docs.

We can help you out in the meantime if you want to get going; otherwise the closest thing to docs are the integ tests 😄

[kind carbon]

Yeah, sorry about that, I'm taking longer than expected 🙂

[warped canyon]

No rush at all, do it right 💪

I guess I'll start with mod init --sdk=python and see what I trip over 🙂

[rocky harbor]

No worries! If there's anything I can do to help (e.g. the Zenith API is getting in your way, etc.) let me know

[rocky harbor]

I think this might be the most fleshed out example of using Python so far, may actually be helpful https://github.com/dagger/dagger/pull/5964/files#diff-3f816215b584a031057b440089ceb90784c3d4257816d7facfa3dcec14fe0a94R1106-R1116

[warped canyon]

Awesome thanks. I'm going to see if I can do this one in python right now #1167259035662307398 message

[kind carbon]

Here's a peek for classes: https://github.com/dagger/dagger/issues/5991#issuecomment-1783081659

[warped canyon]

oh I guess I need to think about how to do this one without objects lol

[kind carbon]

You'll be able to use classes if you need to. Many users won't. So the top level functions are a convenience for a class with the name of the module and no fields.

You can also have both. Top level functions for the main module's "object", but also classes for additional objects.

[warped canyon]

Nice, yeah just thinking about the modules I've worked on so far, probably 70% wouldn't need them and 30% would

[kind carbon]

There's already support for default values and descriptions everywhere.... just not documented.

You can also chose a different name than the function/field/class:

@function(name="from")
def from_(address: Annotated[str, "The address to pull from"]) -> dagger.Container:
    """Pull a container image."""
    ...

This way you're not limited by reserved/soft keywords.

[restive shore]

When I run dagger call module, shouldn't it print the Pipeline() labels by default? I am not seeing them. I can do --focus=false but that's messy.

[wintry prism]

@restive shore can you join the Zenith call in progress in #911305510882513037 by any chance?

[restive shore]

oh man! I am in the office today so don't have access to my PC, let me try to join.

[sharp zealot]

Unfortunately I seem to still be stuck...

[restive shore]

I may have hit a blocker with modules. I am able to do dagger call module from my local machine but when I try it in CI, it times out when it tries to pull the go modules because of proxy. How do I make my corp proxies available to the engine spawned off of dagger call? It fails when it tries to do the codegen for the modules initially.

The env that dagger call is executing in has the proxies set.

I don't have the same issue when I do run non-zenith dagger code with go run main.go

[rocky harbor]

I may have hit a blocker with modules

[latent trellis]

Would be nice to have a "Go to source" button on the daggerverse UI. Since most modules are actually subpaths of a repository, just copying and pasting the URL doesn't work. (Also, I would still have to go to the specific commit).

[sharp zealot]

I think there is 🙂 Click on the github logo

[latent trellis]

never occured to me it could be a link. 😄

I noticed in some modules that there are some exported fields for storing state: https://github.com/kpenfound/dagger-modules/blob/main/golang/main.go#L15

Is that intentional? Does it have any meaning? (I'd probably default to non-exported fields which is why I'm asking)

[warped canyon]

Yeah right now private fields won't work (unless this got fixed recently?)

[latent trellis]

Won't work as in? They'll lose the state between calls?

[warped canyon]

Something about codegen. When they're serialized they're not included I think

[latent trellis]

To be clear, I don't want to access those fields (from Dagger/GraphQL) directly. I only want to set them via With* calls and access them with a Container() function like in the example above.

[sharp zealot]

Is this failing for anyone?

dagger -m github.com/shykes/daggerverse/dagger functions

[latent trellis]

yes

[latent trellis]

I wonder if it makes sense to write defensive APIs. For example: WithSource can only be called after a container is set (this would require a second struct that is returned by WithContainer).

The alternative is a flat API (WithContainer and WithSource on the same struct) and storing intermediary inputs and compiling the final container when Run/Lint/Test whatever is called.

The first one is more work, but makes it less likely and extra WithContainer somewhere messes something up. The second one is probably easier to write.

This:

type Spectral struct{
}

func (m *Spectral) WithVersion(version string) *SpectralContainer {
    return &SpectralContainer{dag.Container().From(fmt.Sprintf("%s:%s", defaultImageRepository, version))}
}

func (m *Spectral) WithImageRef(ref string) *SpectralContainer {
    return &SpectralContainer{dag.Container().From(ref)}
}

func (m *Spectral) WithContainer(ctr *Container) *SpectralContainer {
    return &SpectralContainer{ctr}
}

func (m *Spectral) WithSource(src *Directory) *SpectralContainerWithSource {
    return &SpectralContainerWithSource{
        &SpectralContainer{
            dag.Container().From(defaultImageRepository).
                WithWorkdir("/src").
                WithMountedDirectory("/src", src),
        },
    }
}

Versus this

type Spectral struct {
    Container *Container
    Source    *Directory
}

func (m *Spectral) WithVersion(version string) *Spectral {
    m.Container = dag.Container().From(fmt.Sprintf("%s:%s", defaultImageRepository, version))

    return m
}

func (m *Spectral) WithImageRef(ref string) *Spectral {
    m.Container = dag.Container().From(ref)

    return m
}

func (m *Spectral) WithContainer(ctr *Container) *Spectral {
    m.Container = ctr

    return m
}

func (m *Spectral) WithSource(src *Directory) *Spectral {
    m.Source = src

    return m
}

The more defensive API can also protect against accidentally calling a function without the necessary state (eg. source).

[rocky harbor]

Yeah I agree the version where you have your own SpectralContainer type is going to be a lot safer overall. And even though it's slightly more verbose in terms of the types, it's probably still saving lines of code + brain power to have fewer permutations of possible states to handle

Also in terms of consumers, I think it would pretty much be the same end DX

[latent trellis]

The happy path would certainly be the same.

The error path would be slightly easier on both sides: you would get an early warning that you are trying to do something invalid. On the module development side, you don't have to check if a source has been set for example. The type system takes care of that by eliminating invalid states.

Thanks for the feedback!

I asked, because I haven't seen this style in modules yet. Most modules implement a flat API.

[rocky harbor]

I asked, because I haven't seen this style in modules yet. Most modules implement a flat API.
Totally, we're still all figuring it out, so these sorts of questions+discussions are super helpful. Appreciate it

[latent trellis]

Is there an example for running multiple containers?

Here is what I did:

func (m *Lint) All(ctx context.Context) error {
    var group errgroup.Group

    group.Go(func() error {
        _, err := m.Go().Sync(ctx)
        if err != nil {
            return err
        }

        return nil
    })

    group.Go(func() error {
        _, err := m.Openapi().Sync(ctx)
        if err != nil {
            return err
        }

        return nil
    })

    return group.Wait()
}

Ideally, I'd like to have the same behavior as returning a container (fail if errors, print stdout). Sure, I can do it manually...but I just don't want to. 😄

[upbeat herald]

Idk what happened to Zenith but all my ci in my daggerverse is broken, and I only have this as output:

Error: make request: input:3: ci.lint failed to call function "Ci.lint": failed to get function output directory: process "/runtime" did not complete successfully: exit code: 2

[latent trellis]

I remember I had a similar issue and I had to upgrade to the latest version (ie. the go sdk in the module AND the dagger binary as well)

[upbeat herald]

Ohhh okay

That's strange because the goSDK is generated so I shouldn't have to update it... I also use dagger binary from main

[latent trellis]

Is it possible to define a "default" action for a command?

For example:

• dagger call lint go -> run golangci-lint
• dagger call lint -> run all linters

(For the moment I have an "all" action that runs all actions.

[warped canyon]

I think I'm hitting this on main right now.

Getting a mix of:
ERROR: query module objects: returned error 502 Bad Gateway: http do: Post "http://dagger/query": net/http: HTTP/1.x transport connection broken: EOF
and
Error: query module objects: returned error 502 Bad Gateway: http do: Post "http://dagger/query": rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7264280 vs. 4194304)
https://dagger.cloud/runs/c709d9da-d809-4475-9d80-845c0c4ccb5e

Looking at my log output, the ID for [0.53s] [DEBUG] dagger.mod._module: output => '"core.Module:eyJzb3... is crazy long. Like 5 screens long

[latent trellis]

Kafka service module: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/kafka@93d0e7724f12e78d6f9c3917d4e1f52812e9b5c1

I noticed types in a module are prefixed with the module name which is weird when I already add the prefix.

For example, I have a KafkaContainer type in there that's called KafkaKafkaContainer in the module docs.

I can't call it just Container, because there is already a Container type in there. 😄

Any ideas?

Also: I'm testing these Dagger modules with Dagger. 😄

while(usingDagger) {
    mind.blown = true
}

[upbeat herald]

If you created a module kafka, you can just add a method on your struct called Container and that should work and let you call it container 😄

type Kafka struct {}

func (k *Kafka) Container() {...} 

[latent trellis]

The reason why I have a separate struct is to create a strict API (ie. the KafkaContainer struct doesn't expose a WithContainer or a WIthVersion function)

[upbeat herald]

I see, maybe call it Instance then?

[latent trellis]

I was thinking Context but it's such an overloaded term (like Manager 😛 )

Maybe Node when thinking about the graph context.

[upbeat herald]

Node make sense

[sharp zealot]

FYI @wintry prism I am getting a LICENSE file from dagger mod init but not NOTICE file

@warped canyon are you still getting module errors on main?

I'm getting this on a brand new module:

 % dagger mod sync
✘ generatedCode ERROR [0.49s]
✘ exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [0.44s]
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleM
┃ nSrc: cannot define methods on objects from outside this module                                          
┃ Usage:                                                                                                   
┃   codegen [flags]                                                                                        
┃                                                                                                          
┃ Flags:                                                                                                   
┃   -h, --help                             help for codegen                                                
┃       --introspection-json-path string   optional path to file containing pre-computed graphql introspect
┃ n JSON                                                                                                   
┃       --lang string                      language to generate (default "go")                             
┃       --module string                    module to load and codegen dependency code                      
┃   -o, --output string                    output directory (default ".")                                  
┃       --propagate-logs                   propagate logs directly to progrock.sock                        
┃                                                                                                          
┃ Error: generate code: template: module:56:3: executing "module" at <ModuleMainSrc>: error calling ModuleM
┃ nSrc: cannot define methods on objects from outside this module                                          
✘ generating go module: ttlsh ERROR [0.19s]
├ [0.04s] go mod tidy
┃ writing dagger.gen.go                                                                                    
┃ writing go.mod                                                                                           
┃ writing go.sum                                                                                           
┃ needs another pass...                                                                                    
• Cloud URL: https://dagger.cloud/runs/2c31f105-2075-490b-a2af-abd6a47f1356
• Engine: d0277984c27c (version devel ())
⧗ 1.38s ✔ 18 ∅ 3 ✘ 3
Error: failed to automate vcs: failed to get vcs ignored paths: input:1: host.directory.asModule.generatedCode failed to get modified source directory for go module sdk codegen: process "/usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json" did not complete successfully: exit code: 1

% go vet
go: errors parsing go.mod:
/Users/shykes/dev/shykes/daggerverse/ttlsh/go.mod:3: invalid go version '1.21.3': must match format 1.23

[warped canyon]

what's your host go version?

[wintry prism]

Yep, I updated my host go version to

go version go1.21.3 darwin/arm64

to avoid this contention between go.mod formats between these go versions

[sharp zealot]

% go version go version go1.20.1 darwin/arm64

[wintry prism]

I wonder if we could do something to smooth that out for folks 🤔 Otherwise we should document a minimum required go version

[warped canyon]

Yeah you're running into this: #daggernauts message

[sharp zealot]

That only explains the 2nd error message, right?

I'm still getting an error on dagger mod sync (first part of that snippet) which I'm guessing is unrelated to go version on the host (I'll still update of course).

[wintry prism]

I'll clone your repo and try

[sharp zealot]

On the go version front: if everyone is going to run into this issue, should we hold off on upgrading to the latest Go?

[warped canyon]

oh yes I missed that first one. Are you on main or 0.9.1?

[sharp zealot]

main

To be clear I'm getting this even when initializing a brand new module

[wintry prism]

cannot define methods on objects from outside this module

[sharp zealot]

Oh you're right, there's actually a real error message in there

[warped canyon]

I'm on https://github.com/dagger/dagger/pull/6014 but I am able to create new modules atm.

[sharp zealot]

It's probably because I'm overloading a core type in there

This part confused the hell out of me:

Usage:                                                                                                   
┃   codegen [flags]                                                                                        
┃                                                                                                          
┃ Flags:                                                                                                   
┃   -h, --help                             help for codegen                                                
┃       --introspection-json-path string   optional path to file containing pre-computed graphql introspect
┃ n JSON                                                                                                   
┃       --lang string                      language to generate (default "go")                             
┃       --module string                    module to load and codegen dependency code                      
┃   -o, --output string                    output directory (default ".")                                  
┃       --propagate-logs                   propagate logs directly to progrock.sock                        
┃                                                                                          

I assumed it was another regression

We need to work on those error messages

[warped canyon]

yeah that's been leaking through with any codegen errors for some reason

[sharp zealot]

(I know I'm stating the obvious)

[wintry prism]

I used to do that with Trivy scan and moved to required style https://github.com/jpadams/daggerverse/blob/main/trivy/main.go#L32-L34

[sharp zealot]

I added an issue to the quality sprint: https://linear.app/dagger/issue/DEV-2936/module-errors-obsfucated-by-codegen-usage

[wintry prism]

Yep, we're just dropping a LICENSE file and not a NOTICE today.
My impression is that in most cases, only the LICENSE is required and the NOTICE is optional/extra (unless it was there already).
https://www.apache.org/foundation/license-faq.html#Apply-My-Software

[kind carbon]

GraphQL casing

[sharp zealot]

@warped canyon where can I find your proxy module? I'm going to try using it for my tailscale module

[wintry prism]

I see one here: https://daggerverse.dev/mod/github.com/kpenfound/dagger-modules/proxy@f9cc629f6d5d98e620c4fc35312d27d3d6de48f7

[warped canyon]

that's it 👆

[wintry prism]

But I know he has a different version that would hit once python objects are in, right?

[warped canyon]

yeah usage looks like this right now:
https://github.com/kpenfound/greetings-api/blob/zenith/ci/main.go#L42-L48

but once 6016 is merged I can make this change: #1167259035662307398 message

[sharp zealot]

Nice

How about HTTP routing? 😇

[warped canyon]

should be totally doable

[sharp zealot]

I'm trying to fit GraphiQL into a single port (otherwise you have to leak host networking information - specifically hostname - into the module)

[warped canyon]

the module just templates in nginx configs as you add services to proxy: https://github.com/kpenfound/dagger-modules/blob/main/proxy/src/main.py#L36-L44 so it would be trivial to add a route as well/instead

[sharp zealot]

https://discord.com/channels/707636530424053791/1166876145522843668

Would love to get consensus on this so we can ship something in the next patch release 👆 It doesn't have to remove use right away, can co-exist for now

If I wanted to add support for passing a new type by the command-line (like secret, directory, etc) where in the source code should I go to do that (I'd like to try adding service)

Also, how do I pass a secret as argument from the command-line when calling a function? Zenith docs don't mention calling functions from the CLI at all, as far as I can tell

[warped canyon]

dagger call .... --token $NETLIFY_TOKEN
where token is my functions arg name of type secret

[sharp zealot]

FYI, small quality thing: https://linear.app/dagger/issue/DEV-2943/tui-hide-engine-version-by-default

[sharp zealot]

Wouldn't it be better to pass the name of the env variable, rather than its value?

[warped canyon]

could be, really it just accepts a string input

[sharp zealot]

It would have marginal security benefits, and importantly, it would leave the door open to pluggable secrets providers

If we set it up like this, for example:

dagger call --token env:NETLIFY_TOKEN

The env: could be optional:

dagger call --token NETLIFY_TOKEN would be sugar for env:NETLIFY_TOKEN

[wintry prism]

Yep, here's my module based on your HelloWorld that takes a secret for a greeting
https://github.com/jpadams/daggerverse/blob/main/helloSecret/main.go#L21-L25

Just a string today that gets converted internally.

[sharp zealot]

But now we're holding the spot for, say, 1password:NETLIFY_TOKEN in the future (could be a not-so-distant future)

[warped canyon]

Would be nice. In that world, secrets providers would be some sort of special extension to dagger and not modules I guess. Or at least not modules as they exist today

more of a middleware

[sharp zealot]

They could still be modules. We have a TBD design discussion on that part. But we've been designing secrets interfaces deliberately to leave space for that future design

would be a shame to waste it now

I'll create an issue to track this

[kind carbon]

https://github.com/dagger/dagger/blob/main/cmd/dagger/flags.go

[sharp zealot]

https://linear.app/dagger/issue/DEV-2944/cli-pass-secret-arguments-by-name-not-by-value

I apologize for filing / communicating issues that are not accessible to non-employees. That is definitely not ideal, and it's temporary. We're dealing with the complications of Linear and Github co-existing in our stack.

@kind carbon while you're around, I'm looking for the special --help logic that introspects available functions etc. I'm going to try making functions more like that 🙂

(at the moment --help is more helpful than functions)

[kind carbon]

That was ported from dagger do, the difference is it lists all functions in a module. With --help you'll only see the main object's functions, and then you need to continue chaining.

The --help logic comes from cobra. Nothing custom.

[sharp zealot]

Ah I see, the full cobra command hierarchy is generated in one pass, then --help is just the regular cobra introspection of that?

[kind carbon]

Yes

[sharp zealot]

Then I guess what I'm really looking for is the generation of the command hierarchy

[kind carbon]

That's built by the arguments you pass in.

I.e., before parsing an argument, we add all subcommands for current command. Then make cobra.Find a command based on arguments. If it doesn't find one, it errors. If it does, it's a valid argument and does the same thing for next argument until the end.

In functions, that is circumvented, but you already have all the module's functions in a data structure.

It's in fc *FuncCommand: https://github.com/dagger/dagger/blob/214bb027268aa8da9cd6662ef80c9c073fb5e251/cmd/dagger/functions.go#L44

Held in fc.mod: https://github.com/dagger/dagger/blob/214bb027268aa8da9cd6662ef80c9c073fb5e251/cmd/dagger/functions.go#L205

[sharp zealot]

Thank you 🙏

I can see the value of listing all functions from all types, but it's not what I'm looking for most of the time. I was thinking of moving that to a functions --all flag or equivalent

With functions being the standard entrypoint for discovering what can be done at the selected path

So, similar to --help but without the other baggage of --help (I dont care about various flags, just want a list of functions I can call at this path)

[kind carbon]

Being able to know which verbs can be used at a selected path is a todo. That was one of the main points about functions. Without it, any verb will show the same sub-commands when adding --help.

The verbs only do something different on the leaf command, to do something with the result. They can add extra flags to help with that but up to that point they all work the same.

[sharp zealot]

That's fine with me, I understand what you mean and also look forward to that improvement, but orthogonal to what I'm looking for here (I'm fine with a list of functions that is not "verb-aware", just all available functions)

[sharp zealot]

I'm getting a segfault in codegen on my tailscale module 😭

[sharp zealot]

Just got my tailscale working with dagger up 🙂

dagger -m github.com/shykes/daggerverse/tailscale up gateway --key "$TAILSCALE_AUTHKEY" --hostname hello-nginx

Note that there is a --backend SERVICE argument, which cannot yet be specified from the CLI (I am working on fixing that). In the meantime, this will spin up a nginx service container as a default backend.

Another bug report (I'll open an issue later): dagger up refuses to run a service with no exposed ports. But sometimes, services only connect out. Case in point: my tailsale service 🙂 I had to expose a bogus port for it to work

[restive shore]

I am trying to use the validator module but dagger mod sync throws this.
I did go mod tidy. Do I have to do something else besides that?

[rocky harbor]

I am trying to use the validator module

[sleek nest]

Does anyone have a module you'd like to show off during the Community call this week? I have room for one more demo 🙂 If you are interested, please DM me!

[sharp zealot]

https://tenor.com/view/oh-me-saturday-night-live-snl-uso-performance-thats-me-call-my-name-gif-19743416

[spiral whale]

I'm unable to read anything from the following code if the container fails. All output details return with error. Am I missing something?

stdout, _ := c.container.Stdout(ctx.Context)
stderr, _ := c.container.Stderr(ctx.Context)
``

[sharp zealot]

https://linear.app/dagger/issue/DEV-2947/dagger-download-o

[wintry prism]

Super baffled by https://github.com/jedevc/daggerverse/tree/main/hugo
anything I do with it seems to result in

✘ load functions ERROR [0.62s]
├ [0.62s] loading module
┃ Error: failed to get loaded module ID: input:1: host.directory.asModule failed to instal
┃ module dependencies: failed to install dependency "github": failed to install module sch
┃ a: schema validation failed: input:2173: Cannot redeclare type GithubAsset.
┃
┃       | ...
┃  2170 |
┃  2171 |
┃  2172 | """
┃  2173 | type GithubAsset {
┃  2174 |         """
┃  2175 |
┃  2176 |
┃       | ...
✘ asModule(sourceSubpath: ".") ERROR [0.57s]

hugo ➤ cat dagger.json                                                           git:main
{
  "name": "hugo",
  "sdk": "go",
  "dependencies": [
    "./github"
  ]
}

uses a github module in a subdir

hugo ➤ grep -r GithubAsset *                                                     git:main
main.go:func getMatchingAsset(ctx context.Context, assets []GithubAsset, keywords []string) (GithubAsset, error) {
main.go:            return GithubAsset{}, nil
main.go:    return GithubAsset{}, fmt.Errorf("could not find asset matching keywords %s", keywords)

github ➤ grep -ri type                                                           git:main
./api.go:type release struct {
./api.go:type asset struct {
./api.go:    ContentType string `json:"content_type"`
./main.go:type Release struct {
./main.go:type Asset struct {
./main.go:    ContentType string `json:"contentType"`
./main.go:type Github struct{}
./main.go:            ContentType: tmp.ContentType,

[sharp zealot]

@rocky harbor I'm trying to add support for service arguments in the CLI, I think I accurately duplicated the logic for other types (seemed pretty straightforard), but getting this error:

panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

Does that ring a bell?

[warped canyon]

Getting the hang of the Optional pattern in Go and I think it's pretty nice. That's all. 🙂

[warped canyon]

🚀

[sharp zealot]

@warped canyon do you find that you sometimes forget to publish your modules each time you push a new version to github?

(I do... Since nothing actually changes when I forget)

[warped canyon]

Yeah pretty much all the time, since it's not required to go use the new version wherever I need it. But no longer!

[sharp zealot]

FYI https://linear.app/dagger/issue/DEV-2948/go-sdk-doesnt-support-optional-string-array

FYI https://linear.app/dagger/issue/DEV-2949/cli-cant-pass-directory-arguments-outside-of-working-directory

@warped canyon I tried publishing but nothing happened..

[obtuse lion]

Is there info somewhere on how to create a custom sdk?

[kind carbon]

There is, but not for Zenith: https://github.com/dagger/dagger/blob/main/sdk/CONTRIBUTING.md

[warped canyon]

@wompfox I tried publishing but nothing

[rocky harbor]

It's very new, mostly untested and undocumented, so no not really 😄 If you're feeling brave:

1. Write a module that has a Codegen function (which returns a directory containing the generated code for a module based on the user's source) and a ModuleRuntime function (which returns the container that a module executes in in dagger)
   • You can see the signatures expected in the Python SDK module: https://github.com/sipsma/dagger/blob/068e9db6610429a840ee179787321788999a6321/sdk/python/runtime/main.go#L55-L55
2. To use the SDK, manually edit dagger.json to set "sdk" to a module ref that points to the sdk module you wrote above

We are going to flesh this out a lot more, we just have been focusing on getting our first 2 "official" SDKs in place, we'll have more time to polish+document the 3rd party sdk support post-KubeCon

[rocky harbor]

Getting the hang of the Optional

[sharp zealot]

I kind of wish I could connect my daggerverse repo to my daggerverse account, then instead of this:

dagger shell -m github.com/shykes/daggerverse/wolfi base container

I could do this:

dagger shell -m shykes/wolfi base container

(that first command works by the way 🙂

[sharp zealot]

It looks like the calls made by the CLI to pass arguments of type secret/directory/container etc don't show up in Dagger Cloud.

Is that right?

[sharp zealot]

I am developing a module that involves figuring out by trial and error which packages to install on a container. I am LOVING the devloop for that, module source on the left, dagger shell on the right, insanely productive and fun

[spiral whale]

Finally completed the initial version of running individual Github Actions as Dagger module. The next step is to figure out how to improve DX and select actions to generate as Dagger modules. Github Actions collection: https://github.com/aweris/gale/tree/main/daggerverse/gha

[sharp zealot]

amazing 😁

request @spiral whale : can the generator itself be distributed as a module?

[spiral whale]

actions/generator: https://github.com/aweris/gale/tree/main/daggerverse/actions/generator => generated dagger modues
actions/runtime: https://github.com/aweris/gale/tree/main/daggerverse/actions/runtime => runs given github actions directly

Yes, generator and runtime(lightweight wrapper around gale) also a dagger module.

[sharp zealot]

cc @warped canyon 😉

[quick wind]

Having a first look at Zenith, but hitting an error with calling modules from github - Error: no module specified and no default module found in current directory - The github url is of the same format found here: https://github.com/dagger/dagger/blob/main/docs/current/labs/project-zenith.md (e.g. github.com/<user>/<repo>@<branch>) and the repo has a dagger.json at the root from dagger mod init --name=<name> --sdk=python. Am I missing something or is this not expected to work yet?

I'm using the 0.9.2 release if that matters, not specifically the current main branch of Dagger

(Also https://daggerverse.dev appears to be down?)

[warped canyon]

Indeed, that's being worked on 🙂

[quick wind]

Or not, it was but is now loading fine, ignore that!

[warped canyon]

what command is giving you that message?

[quick wind]

dagger call <function-name> -m <github.com/<user>/<repo>@main

[kind carbon]

@quick wind Put -m before the function. On call or before.

[warped canyon]

Yup this is it, I'll update the docs

[quick wind]

Oh I already had that, just missed it when typing it out

Still not working however

[kind carbon]

Surely you get another error though.

[quick wind]

Nope:

✘ load call ERROR [0.00s]
├ [0.00s] loading module
┃ Error: no module specified and no default module found in current directory                                                                
• Engine: 35cbe9a91354 (version v0.9.2)
⧗ 1.07s ✔ 3 ✘ 1

[kind carbon]

That suggests it's not parsing -m. Where in the command do you have it?

[quick wind]

Is this possibly because I overwrote the generated pyproject.toml with a Poetry copy

[quick wind]

dagger call <function> -m <github-path>

[kind carbon]

As I said before, -m needs to be before <function>.

[quick wind]

Oh I misunderstood, I've been using this format

So that does change the error:

✘ load call ERROR [0.00s]
├ [0.00s] loading module
┃ Error: failed to load module config: failed to read local config file: open pytest/dagger.json: no such file or directory                  
• Engine: 35cbe9a91354 (version v0.9.2)
⧗ 0.68s ✔ 3 ✘ 1

[warped canyon]

Just opened a PR to correct the docs: https://github.com/dagger/dagger/pull/6036/files

[kind carbon]

Atm it won't parse the whole args, only until the first positional argument after call.

[kind carbon]

There's a few issues with the errors when loading a module. It suggests it can't find the file but I'm certain it's hiding another kind of error that prevents it from loading it.

Can you check if your dagger.json is well formatted?

[quick wind]

This format seems to progress further:

✘ load call ERROR [10.89s]
├ [10.89s] loading module
┃ Error: failed to get loaded module ID: input:1: git.commit.tree.directory.asModule.serve failed to load module types: failed to call module
┃ pytest" to get functions: failed to get function output directory: process "python -m pip install ." did not complete successfully: exit co
┃ : 1                                                                                                                                        
✘ serve ERROR [3.39s]
✘ exec python -m pip install . ERROR [1.53s]
┃ Processing /src                                                                                                                            
┃   Installing build dependencies: started                                                                                                   
┃   Installing build dependencies: finished with status 'done'                                                                               
┃   Getting requirements to build wheel: started                                                                                             
┃   Getting requirements to build wheel: finished with status 'done'                                                                         
┃   Preparing metadata (pyproject.toml): started                                                                                             
┃   Preparing metadata (pyproject.toml): finished with status 'error'                                                                        
┃   error: subprocess-exited-with-error                                                                                                      
┃                                                                                                                                            
┃   × Preparing metadata (pyproject.toml) did not run successfully. 

[quick wind]

Unedited from generation:

{
  "name": "pytest",
  "sdk": "python"
}

[kind carbon]

That's it, it's the pyproject.toml. Can you show what you have?

[quick wind]

I've overwritten the generated pyproject.toml with a Poetry version

Guessing that's unsupported at this stage

Going to generate another and try that, one minute

[kind carbon]

Not tested but should be supported as long as it's simple enough. That's why I asked to see.

Basically needs to be installable by pip install .. Poetry supports that.

[quick wind]

[project]
name = "main"
version = "0.0.1"

[tool.poetry]
name = "pytest-ci-module"
version = "0.1.0"
description = ""
authors = ["Mb141"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
dagger-io = "^0.9.2"
anyio = "^4.0.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Dependencies are so my editor autocomplete works, they're not actually in use

Just had a load of squigly red lines

[kind carbon]

Oh, for editor autocomplete, you just need to activate some virtualenv and install the sdk with pip install -e ./sdk (after dagger mod sync). Notice the editable install.

Don't add dagger-io as dependency.

The editable install is so you don't have to do it again after another mod sync.

See README here (at the end): https://github.com/helderco/daggerverse/tree/main/python

[quick wind]

That appears to have worked. The module errors but for a module reason (cannot return null...) but it runs

Thanks for helping me figure that out!

[kind carbon]

Yeah, def needs documentation. It's very recent.

[quick wind]

On the off chance it's of any interest, being able to use Poetry over pip would be cool

[kind carbon]

But you should be able to do that.

Because of the [build-system].

[quick wind]

I made a brief attempt at it, I got some poetry errors and left it alone. I'll re-try, just wanted to get a basic module running and callable from elsewhere first

[kind carbon]

If you also want to editable install it via poetry you can. Just add a dev dependency for the local dir "./sdk".

And add editable argument to the dependency.

I think it's develop = true

[quick wind]

Ok, got the callable module working:

✔ dagger call pytest [1.48s]
┃ README.md                                                                                                                               
┃ main.py                                                                                                                                
┃ myapp                                                                                                                                   
┃ pytest.ini 
┃ tests      

I'll re-try Poetry now

[restive shore]

Hey @rocky harbor , The above change may have introduced a regression. ExperimentalPrivilegedNesting doesn't seem to be pass through docker auth to the underlying dagger-in-dagger. I am not sure that's the exact cause though, but with 0.9.0 I can use it to run my tests and the nested dagger session is able to pull docker images from the private registry. Trying to upgrade to 0.9.2 that is failing saying docker image not found. So I am assuming that is because the docker auth is not passed through. I don't fully understand the change that went in with the above.

[quick wind]

So Poetry testing:

Adding dagger.io = { path = "./sdk", develop = true } to [tool.poetry.group.dev.dependencies] worked for autocomplete as you said

Calling the module doesn't work if I just change pyproject.toml:

┃   Preparing metadata (pyproject.toml): finished with status 'error'                                                                    
┃   error: subprocess-exited-with-error   
...
FileNotFoundError: [Errno 2] No such file or directory: '/src/README.md'                                                             
┃       [end of output]          

There is definitely a README.md in src/:

╰─ ls src/
main.py  README.md

[rocky harbor]

It looks like the calls made by the CLI

[warped canyon]

with the python object support in 0.9.2, can anyone give me an example of what it looks like to add/use fields on the main class? The two examples in the PR are awesome but they're both dealing with secondary classes

[rocky harbor]

with the python object support in 0.9.2

[warped canyon]

This surprised me

@object_type
class Proxy:
...
    ctr: Annotated[
...
    ] = field(default=init())

...

def init() -> dagger.Container:
...

doesn't work

NameError: name 'init' is not defined

[rocky harbor]

I think that might just be python being python 🙂 IIRC since the interpreter just reads in the file line by line at runtime you need init to be defined above where it's used; at least in a case like this where it's in the signature of the field on Proxy

[warped canyon]

that might just be python being python
That does not surprise me 😂 I'm rusty

[rocky harbor]

I'm very rusty too but pretty sure I have deep scars from that back when I wrote more python and remembered it 😄

[warped canyon]

and with the implicit class definition I had before it was probably loaded a different way (maybe?)

[rocky harbor]

Yeah I think it depends on exactly where it appears; you can reference a function that's not defined yet provided it's in a def block (or something like that).

[warped canyon]

Interesting. Moving it got me to a new error, so I guess that's good My class Proxy has a function that returns a Proxy object (so I can chain it). It's complaining about the return type of that function

[warped canyon]

Is there any way to do some introspection to see how my python code is getting called by the runtime? I can see the graphql query that looks correct, but the error I'm getting sounds like it's not getting called properly

I suspect the root of it may be that it's an async function, but in the generated Go code it doesn't take in a context

[rocky harbor]

If you provide both --debug and --progress=plain on dagger cli commands you'll get a ton of info, including e.g.

157: [0.62s] [DEBUG] dagger.mod._module: resolver => Test.foo
157: [0.62s] [DEBUG] dagger.mod._resolver: input args => {}
157: [0.62s] [DEBUG] dagger.mod._resolver: structured args => {}
157: [0.62s] [DEBUG] dagger.mod._module: result => 'yo'
157: [0.62s] [DEBUG] dagger.mod._module: unstructured result => 'yo'
157: [0.62s] [DEBUG] dagger.mod._module: output => '"yo"'

Which it sounds like is what you're looking for?

[warped canyon]

Yeah perfect, I had --progress=plain but not --debug

still broken 😞 let me see if I can get a simple repro to share

[latent trellis]

Daggerverse seems to be down or at least struggling to load for me.

[sharp zealot]

I am getting some very surprising cache misses in my zenith runs today. Some very straightforward runs that have no opportunity to miss cache - but still do, on the very next run. Very strange.

[kind carbon]

default needs to be either an immutable value or a callable: field(default=init). For more on this see default_factory in dataclasses.field.

[warped canyon]

oh hey 😃

[rocky harbor]

Daggerverse seems to be down or at least

[kind carbon]

Here's a module to stress test what dagger shell can do with lots of examples for rich terminal shenanigans: https://github.com/helderco/daggerverse/tree/main/textualize
Hint: most don't work! 😅

[sharp zealot]

i was just playing with shell and it seems it doesn’t work with custom entrypoints? is that a known issue?

also: shell is amazing

[kind carbon]

Custom entrypoints?

Like WithEntrypoint?

[kind carbon]

It does support it, but only if you don't set --entrypoint yourself. By default dagger shell will exec the default command (entrypoint + default args) that's defined in the container and default to sh if not.

[rocky harbor]

^ Can confirm because I'm working on examples for docs right now and just called dagger shell expecting sh but ended up dropping into an node shell because I used that as my base image and never reset the entrypoint 🙂

Dropping right into interpreters like that is going to be a cool use case for debugging+playing-around though

[sharp zealot]

mmm i must be doing it wrong

will send a repro later tonight

[rocky harbor]

@upbeat herald Want to use your node module in the zenith docs examples, PR to update it to work w/ the latest version: https://github.com/quartz-technology/daggerverse/pull/1

[rocky harbor]

@sharp zealot tiny tweak to the ttlsh module https://github.com/shykes/daggerverse/pull/5

[upbeat herald]

Merged! Should I publish the module again or daggerverse automatically fetch from the main branch?

I also need to update my other modules, I’ll take care of that today 🚀

[sharp zealot]

dagger mod publish still not publishing anything for me, as far as I can tell

[sharp zealot]

Then I think there's a bug somewhere. Here's a repro:

1. dagger shell -m github.com/shykes/daggerverse/tmate tmate

• Expected behavior: open a shell running tmate
• Actual behavior: it works ✅
• Implementation: WithDefaultArgs. This is a hack, not what I actually want (can't pass arguments to tmate) ❌

2. dagger shell -m github.com/shykes/daggerverse/tmate tmateWithEntrypoint

• Expected behavior: open a shell running tmate
• Actual behavior: silently exits without an error ❌
• Implementation: WithEntrpoint. This is what I want, and what supposedly works for everyone except me? ✅

3. dagger shell --entrypoint tmate -m github.com/shykes/daggerverse/tmate tmateWithEntrpoint

• Expected behavior: open a shell running tmate
• Actual behavior: this actually works
• Implementation: WithEntrpoint + explicit entrypoint in the CLI. I expect this to work, but I don't expect it to be required to work ⚠️

I would be grateful to anyone who can reproduce and share their thoughts 🙏

[kind carbon]

What's happening is that in the image, default args is /bin/bash, so when you set WithEntrypoint, it's doing exec tmate /bin/bash and that's what's failing. When you use --entrypoint, it's doing an exec with skipEntrypoint so it overrides both values. To fix your tmateWithEntrypoint you need to clear the default args: WithDefaultArgs(nil).

tmate works because by using default args, you're replacing /bin/bash with tmate which is what you want.

Age old dance of ENTRYPOINT & CMD 🙂

If you run shell --focus=false you can see what's the actual command that ran btw.

[sharp zealot]

oooh. But how did you figure out that was the problem? I didn't see an error anywhere?

[sharp zealot]

and there's the answer 🙂 Thanks.

I would consider that a bug though. Errors should always be visible, having to type --focus=false is a UX bug in my opinion

Also: IMO WithEntrypoint should clear default args by default, perhaps with an option to not do that.

[spiral whale]

I was going to write I had same issue with kind module. WithDefaultArgs(nil) it's nice to know this command will do the trick 🙏

[kind carbon]

Yes, not sure how shell is consuming that container.

[sharp zealot]

https://github.com/dagger/dagger/issues/6044

[sharp zealot]

@kind carbon I'm not getting any error from dagger shell -m github.com/shykes/daggerverse/tmate tmateWithEntrypoint, with or without --focus=false

Just silent failure in both cases

Latest run: https://dagger.cloud/runs/50497244-8d92-40fc-b5d0-0b7135166c53

[kind carbon]

Yeah I know. I was agreeing with you that it's a bug. Not sure if it's on the services side or shell. Not familiar with that code.

Can't access that run btw.

With --focus=false I just meant that I saw the command and it looked obviously wrong, it's not that it correctly errored out.

[sharp zealot]

FYI https://linear.app/dagger/issue/DEV-2956/dagger-shell-errors-dont-show

[kind carbon]

This doesn't fail as well: --entrypoint wat. Returns exit 0.

[sharp zealot]

While I'm at it: https://github.com/dagger/dagger/issues/6051

[wintry prism]

I did mine like this
https://github.com/jpadams/daggerverse/blob/main/trivy/main.go#L17-L26

whoops. You mean the default container args

[sharp zealot]

Still cool to see your latest version 🙂

It actually makes me wonder: how hard would it be to support a *Container argument instead of a ref?

Then you can inject scanning inline to any pipeline, without being forced to push to a registry

[wintry prism]

Like this? https://github.com/jpadams/daggerverse/blob/main/trivy/main.go#L32-L34

[sharp zealot]

Yes 🙂 Exactly. Nice

I'm guessing that version is a little slower

If that becomes a problem, we could rig a private registry, I could see that being cheaper

[wintry prism]

Yep, it seems to take a while to download the trivy vulns db. Haven't followed the idea of caching that content somehow.

[sharp zealot]

I got sidetracked building tmate from source 🙂 But ready to play with tmate for our battle 🙂 @warped canyon

[sharp zealot]

 % dagger mod publish
✘ dagger mod publish ERROR [0.03s]
• Cloud URL: https://dagger.cloud/runs/399e9d2e-a92a-4058-bb8d-78170155f779
• Engine: f663ba1ab6a9 (version devel ())
⧗ 0.43s ✔ 3 ✘ 1
Error: failed to get module path: failed to parse git remote origin URL: ssh://[e-mail]/shykes/daggerverse does not match "^(git@|https://)(github.com)[:/]([^/]+)/([^/]+)"

[sharp zealot]

@kind carbon @rocky harbor @deft rain I'm trying to implement dagger print, I really find myself needing an "object inspector" tool often in my dogfooding.

Implementation question: I can't use OnSelectObjectLeaf because it's too limited (I need to access more than one field, for example upon receiving a Container I want to expect many fields and print their contents.

So I have to use AfterResponse, which only has a result any.

Is there a way for me to introspect the type of that result argument?

[sharp zealot]

Looks like I can do a regular Go type switch against different ID types.

[sharp zealot]

Scratch that, no I can't

[wintry prism]

Hmmm...
Not sure how to proceed with python module. Seeing error mentioned by others, but they were messing with Poetry and such.

The "main" module could not be found. Did you create a "src/main.py" file in │
┃ │ the root of your project?

Got something working in thread

I tried with object type (with function under/within it) and then without.

[sharp zealot]

I'm trying to implement dagger print, and really struggling to get the behavior I need from the TUI engine. Is anyone reasonably familiar with how to use it?

[rocky harbor]

@Helder Correia @Erik Sipsma @jedevc I'm

[sharp zealot]

Basically, I want to:

1. call the specified pipeline of functions (same as the other zenith functions)
2. inspect the result
3. print a bunch of useful text about the result, on standard output

It's fine (and useful) for the TUI to show progress information while the functions are running, BUT once dagger print returns, I want to see the actual output that dagger print has written to standard output. I don't want the output of the TUI to overwrite it.

Technically, the correct content is written to standard output: if I pipe dagger print into another unix command, the behavior is correct. But, visually that is not reflected in the terminal

The second problem is that the cobra output is overwritten to go through the TUI writer. Not a problem in itself.. Except that does break what is written to stdout (adds decorative characters, color etc). The only way to disable this is to use --silent, but that disables all TUI display, which is not what I want.

At the moment I have to bypass the cobra output feature altogether, and fmt.Printf directly. But that might have other side effects I'm not aware of.

[kind carbon]

dagger print

[rocky harbor]

https://daggerverse.dev/mod/github.com/sipsma/daggerverse/yamlinvaders@bbdd8736c41ab7d867a62e3db32895c143e78248

dagger -m github.com/sipsma/daggerverse/yamlinvaders shell play

cc @void widget 🙂

[sharp zealot]

@rocky harbor @void widget @warped canyon FYI tomorrow for my community call demo slot, I want to setup a collaborative shell (via tmate), them invite whoever has a module to show off, to grab the keyboard and run a command to show off their module.

Like the above

[sharp zealot]

I’m starting to think that 1) interactive dagger shell needs a non-interactive counterpart, and 2) interactive and non-interactive could be collapsed into the same command, kind of like docker run [-i]. But possibly even better so that interactive/non-interactive can be auto-detected based on local tty being present or not . Basically like ssh [-tT].

I think dagger run [-i] would be a good candidate 😁

[latent trellis]

How would someone implement live reload (or rather watch and reload) with Dagger today?
Let’s say I have a bunch of services that I want to build and run in Dagger (instead of building and running them on the host machine with a docker compose setup).
I can certainly solve the watch part, but how would that work with Dagger. I don’t necessarily want to restart all services.

[strange arch]

Hello, I’m playing again with Dagger after a long pause and I’m excited to see the progresses done on project Zenith so far, congrats! I’m trying to adapt the CI code I had to make it work with newer versions of Dagger. For example, I have a Hugo package (see attached file) and I have two questions:

• If I turn it into a Dagger module, how can I handle the WithCustomizations() which takes functions mutating a container as arguments? I guess that functions can’t be serialized to be passed through GraphQL, right?
• If I can’t turn my package into a Dagger module, how can I access the Dagger client, should I pass it to every functions? (As a bonus question, what’s the rationale behind the removal of dagger.Context?)

[wintry prism]

Hello, I’m playing again with Dagger

[restive shore]

How would someone implement live reload

[warped canyon]

I think it doesn't like my terminal, super cool though!

[sharp zealot]

@warped canyon @dense canyon @rocky harbor @void widget @upbeat herald @wintry prism (and anyone else who has a cool module to show off). My tmate setup isn't ready (yay childcare). So instead I'm just going to run one-liners to show off as many modules as possible in my terminal.

If you want me to show off your module, prepare some one liners 🙂

[rocky harbor]

dagger -m github.com/sipsma/daggerverse/yamlinvaders shell play (press space once on the menu screen to start)

[dense canyon]

wget -q https://download.samplelib.com/mp4/sample-5s.mp4 && wget -q https://download.samplelib.com/mp4/sample-10s.mp4 && dagger download -m github.com/marcosnils/daggerverse/videostitch@main stitch --src-dir .

Make sure to have at least two mp4 files in the src-dir(https://samplelib.com/sample-mp4.html)

[sharp zealot]

@dense canyon can you have a function to download the samples please

[warped canyon]

dagger -m github.com/kpenfound/dagger-modules/golang download build-remote --remote github.com/dagger/dagger --ref main --module "./cmd/dagger" --platform darwin

[dense canyon]

won't be able to ship it before you present

[rocky harbor]

dagger -m github.com/sipsma/daggerverse/example up -n service - serves a dagger hello world react app on localhost:8080

[hexed flume]

✘ dagger up service ERROR [0.00s]
┃ Error: unknown command "-" for "dagger up service"
• Engine: d662ffe0eaec (version v0.9.2)

typo ?

oups sorry

my bad

[rocky harbor]

Phew, you scared me 😄 It did work for me locally on vanilla v0.9.2

[hexed flume]

Sorry, I did a bad copy paste

works like a charm

[spiral whale]

dagger call -m github.com/aweris/gale/daggerverse/gha/trufflesecurity/trufflehog@1dee85484cbfba12c609ba114839977547e94c7a run --repo dagger/dagger --branch main --with-path "."

[sharp zealot]

FYI https://linear.app/dagger/issue/DEV-2959/cache-cli-commands-locally

[wintry prism]

dagger call -m github.com/jpadams/daggerverse/drupalTest run

Zenith version of this Service Container example:
https://docs.dagger.io/757394/use-services/#example-mariadb-database-service-for-application-tests
Using drupal and mariadb modules.

Trivy two ways

dagger call -m github.com/jpadams/daggerverse/trivy scan-image --image-ref alpine/git:latest

dagger call --progress=plain -m github.com/jpadams/daggerverse/trivy scan-image --image-ref alpine/git:latest --format json | jq '.Results[].Vulnerabilities[] | {Title, Severity}'

@sharp zealot 👆

[sharp zealot]

This is just a rehearsal, while I'm live I'll check the amphitheater chat

🙂

[sharp zealot]

@sleek nest @bleak nest do you want to jump on #911305510882513037 to go over zenith docs together? Now or later?

[bleak nest]

Yes. 5 min? I'm trying to get the deploy preview working so we can look at it together, seems to be a netlify caching issue

[sharp zealot]

Just ping me when you're ready and I'll join

[sleek nest]

@hexed flume is this related to the issue I hit this morning?

[bleak nest]

In dev-audio @sharp zealot

[hexed flume]

Which PR are we talking about?

[sleek nest]

Looks like the preview works now, so we are unblocked. Sorry! https://deploy-preview-6015--devel-docs-dagger-io.netlify.app/zenith/

[sharp zealot]

Small CLI polish thing: https://linear.app/dagger/issue/DEV-2962/cli-add-support-for-git-sources-in-directory-arguments

[sharp zealot]

Fun fact there is an official implementation of the docker-compose spec in Go, and it's terrible

(or at least terribly frustrating to use)

[wintry prism]

Almost too tempting...

[sharp zealot]

Context: I'm making a basic docker-compose module (not surprising I'm sure)

[upbeat herald]

Btw your dagger module isn’t published, could you do it please so I can use it tomorrow?

[sharp zealot]

Which one?

[sharp zealot]

I'm trying to add support for git sources in directory arguments. But not sure what encoding to use. git:// makes sense, but then I don't know where to put the https/ssh distinction. Are git+ssh://... and git+https://... considered valid and good? wdyt?

The other possibility is just making it ssh:// and https://, but then it would close the door to eg. tarball-over-https which is another interesting source to support

[warped canyon]

github actions does not like zenith logs. Maybe related to this? https://github.com/dagger/dagger/issues/5791

GHA running dagger 0.9.2, the dagger call step shows as 24 minutes, but in dagger cloud it's 2.5 minutes

running with dagger -s seems normal. Just can't see anything

[warped canyon]

actually it's probably this https://github.com/actions/runner/issues/1031

I think it's the super long IDs. Would love to be able to suppress that somehow

raw logs for reference, IDs should be easy to spot: https://pipelinesghubeus11.actions.githubusercontent.com/XoA0Ih4phAy4fpnYZcLq7i6UrxFJNlAEyh3N6dZlS47D233BCS/_apis/pipelines/1/runs/102/signedlogcontent/2?urlExpires=2023-11-03T00%3A09%3A51.3736658Z&urlSigningMethod=HMACV1&urlSignature=y1PlSb78bMSOk7t2sn4LH47R7PlVhIKD1RgGoIrp45k%3D

[wintry prism]

{"count":11,"value":"Uri expired"}

[warped canyon]

https://github.com/kpenfound/greetings-api/actions/runs/6739131628 > view raw logs

[rocky harbor]

Fun fact: sorting the results returned for introspection queries to our graphql server makes fully cached load time 3x faster in one of Kyle's demos 🚀 https://github.com/dagger/dagger/pull/6056

In general anyone running modules that have lots of deps will probably greatly benefit from that (will go out in release tomorrow)

[warped canyon]

Give me this! 🙏

[warped canyon]

trying out a little thing for my demo on github actions

    steps:
      - name: Dagger
        uses: kpenfound/dagger-action@main
        with:
          args: ci-remote --commit $GITHUB_SHA
          module: github.com/kpenfound/greetings-api/ci
          cloud-token: ${{ secrets.DAGGER_CLOUD_TOKEN }}

no more actions/checkout 🙂 If there's no .git/ dir, the action does a git clone -b ${GITHUB_REF_NAME} --no-checkout https://github.com/${GITHUB_REPOSITORY} . to get the git metadata for dagger cloud

[sharp zealot]

how does it access the runner’s env?

[warped canyon]

it's executed on the runner https://github.com/kpenfound/greetings-api/blob/main/.github/workflows/test.yml + https://github.com/kpenfound/dagger-action/blob/main/action.yml

[sharp zealot]

also:

if there’s no .git dir

didn’t understand that part

very interested in this 🙂

[warped canyon]

overcomplicating it. At first I just dropped the actions/checkout since I'm executing the remote module. But the my cloud runs didn't have any metadata. So now the kpenfound/dagger-action will do a --no-checkout clone if there's no .git dir just to get the metadata for cloud. I check for .git in case there was a actions/checkout, maybe for some other steps

[sharp zealot]

oh I got it

“it” is your gh action

[warped canyon]

yeah it's just a little composite action to trim down yaml

[sharp zealot]

can’t wait to F&F the telemetry & viz stack

lots of low hanging fruits

really you should be able to get all git metadata just by inspecting directories flowing in between functions

[warped canyon]

maybe there could be some api to set the metadata

[sharp zealot]

better for cloud to get it from raw telemetry IMO

[warped canyon]

for sure. just trying to think of how we'd know where to get the metadata in this kind of scenario where I have no repo locally

[dense canyon]

👋 what's the recommended approach for devs trying to lint their module's code in CI? They should run dagger mod sync beforehand generate all the types that are generally not commited?

[kind carbon]

My module (dev module for the python sdk) can lint itself actually, no need for dagger mod sync: 🙂

dagger call -m dev lint --src dev check

And to fix (not CI):

dagger call -m dev lint --src dev format -o dev

But in general terms you can use a module that can lint your SDK's language:

dagger call -m <module with linter> lint --src .

Or use a git ref instead of local dir.

For mod sync you may need it for the linter to be able to go through the code paths. I don't need that in the dev module because the generated sdk in the module matches the one outside which is already installed before the linter runs. Curious to explore that in a more isolated case.

[dense canyon]

dagger call -m <module with linter> lint --src .

this will still fail if my querybuilder and dagger.gen (talking about the SDK which is the one I know at least) stuff is not present, right?

since src will mostly copy my *. go files into the linting module and it will run lint on them

and since I'm missing some types (unless I call dagger mod sync first), both compiling and linting the module will fail. cc @kind carbon since I haven't replied

[kind carbon]

Yes, that’s what I said I my last comment there.

[dense canyon]

lol.. I'm lost. Are you saying it should work or you're agreeing with me that it fails?

[kind carbon]

I’m agreeing that you need mod sync.

[dense canyon]

dang, just read this Helder. Completely missed this comment 😛. Now I understand why I was confused

[wintry prism]

So fun hacking from 35,000 feet on Zenith!

[sharp zealot]

https://tenor.com/view/sky-skydiving-cat-parachuting-gif-15024274

[wintry prism]

With @warped canyon's help, just shipped https://daggerverse.dev/mod/github.com/jpadams/daggerverse/infisical@7f64507480159500910b48f161ba8e256fa0f8ff
using dagger mod publish ✈️ (pair programming for the win!)

Dagger Python SDK plus Infisical Python SDK makes things so nice. Simple for now. Can enhance later.
https://github.com/Infisical/infisical-python
https://github.com/jpadams/daggerverse/blob/7f64507480159500910b48f161ba8e256fa0f8ff/infisical/src/main.py#L1-L10

https://github.com/jpadams/daggerverse/blob/7f64507480159500910b48f161ba8e256fa0f8ff/infisical/pyproject.toml#L5

I fought the law of the sky...and the sky won. Joining Zenith Community call denied 😭

[sharp zealot]

Another issue on my list, that I forgot to mention on the call:

• I wish I could invoke core functions with dagger call

dagger shell -m core container from --address alpine

A lot of very simple but very useful commands would be unlocked without having to develop wrapper modules

Which brings me to another issue: namespace conflict with core types is becoming a problem for me.

For example, it makes it basically impossible to develop a wrapper module for exposing core, because by definition, 100% of the types I want to expose, already exist in the core! Container, Service, Secret etc. I can't define any of them.

Of course that's an extreme example, but even in regular modules, it's common for me to hit a "reserved" type name and be mildly annoyed.

What's worse is that, by introducing new core types in the future, we may break a bunch of modules.

[kind carbon]

Dagger Python SDK plus Infisical Python

[whole maple]

I currently struggling with one of my modules here
https://github.com/schlapzz/dagger-modules/blob/main/git/main.go
When I call dagger call push --url [e-mail]:cschlatter/clone-test.git --username foo --email [e-mail] --branch main --key ./id I get following error

✘ dagger call push ERROR [0.08s]
┃ Error: response from query: input:1: Cannot query field "push" on type "GitRepository".                                                                       
┃ input:1: Field "git" argument "url" of type "String!" is required but not provided.                                                                           
• Engine: ec863c0c2634 (version v0.9.1)

I have no clue what I'm doing wrong. I code two other modules, and they worked as expected.... but on this one I'm a bit lost....

[sharp zealot]

FYI, support for directory args: https://github.com/dagger/dagger/pull/6058

Could I interest anyone with a tailscale account in trying my module, to reproduce an issue?

dagger up -m github.com/shykes/daggerverse/tailscale gateway . You will need a tailscale API key

[kind carbon]

GitRepository

[sharp zealot]

Another example: #1170079532410208376 message

[sharp zealot]

In Go SDK, is there a way to call my own functions without actually specifying a value for optional arguments? Do I just call Opt() with no argument? EDIT: that doesn't work

Still struggling to get dagger shell to work properly...

Could anyone reproduce this?

$ dagger shell --entrypoint /bin/sh -m github.com/shykes/daggerverse/tmate tmate --base ubuntu

[sharp zealot]

I'm peeling the layers of the onion that is dagger shell behavior...

1. CLI-side cmd/dagger/shell.go has some logic to interpret --entrypoint (which doesn't actually set the entrypoint), and introspect the container's actual entrypoint and default args. Generally this seems to add an extra layer of WithExec

2. Core-side, there's a semi-private API call ShellEndpoint

3. Below that, there's Container.runShell`

4. Below that, there's Container.Service. This is where the extra layer of WithExec is actually used.

So the source of my confusion is becoming more clear: there is entanglement between logic all the way at the top (CLI handling of a command) and all the way at the bottom (what happens when you spawn a service from a container, and how exactly it's decided what to exec)

[sharp zealot]

I still have no idea why this doesn't work

If I understand correctly, Container.Service doesn't honor the container's entrypoint & default args? Only the last WithExec? Or a combination of the two?

I feel like the answer to that question should be added to the reference doc of Container { asService }

Giving up for now

[spiral whale]

but you're trying to pass image address to container id

[sharp zealot]

yes, that works

the CLI interprets the string as an image address and makes the conversion automatically

[spiral whale]

┃ Error: response from query: input:1: Argument "base" has invalid value {address: "ubuntu"}.
┃ Expected type "ContainerID", found {address: "ubuntu"}.
┃ Error: unexpected resourceid.ID[github.com/dagger/dagger/core.Container] literal type: *ast.ObjectValue

this is the error I'm getting

[sharp zealot]

You may need to run it in main

[spiral whale]

ahh, I'm on 0.9.2. Hmm, that should be it.

[sharp zealot]

I've tracked down my mysterious tmate shell issue to this line:

 ctr = baseCtr.WithFile("/bin/tmate", r.StaticBinary())

If I replace it with:

ctr = baseCtr

Then dagger shell works

I can't see the actual error encountered by executing the shell, nor can I inspect the contents of the container, so I can only guess. But it seems that Container.WithFile() causes the container to break, somehow

side quest: just discovered this in the top-level graphql namespace:

[sharp zealot]

In my efforts to debug the mysteries of dagger shell and entrypoints, I developed this module:

https://daggerverse.dev/mod/github.com/shykes/daggerverse/containers

In case anyone else wants to interact with the low-level container API directly from the CLI. It's pretty neat 🙂

dagger shell -m github.com/shykes/daggerverse/containers \
  from --address alpine \
  state

dagger shell -m github.com/shykes/daggerverse/containers \
  from --address alpine \
  with-entrypoint --args=/bin/sh \
  without-default-args \
  state

[obtuse lion]

When using dagger call, all names (functions, arguments, struct fields, etc) are converted into a shell-friendly "kebab-case" style.

When using dagger query and GraphQL, all names are converted into a language-agnostic "camelCase" style

FWIW we were doing the same kebab/camelCase translation in a feature in acorn and just recently reverted. camelCase in a CLI is very yucky, but it got to confusing and was a source of frustrating when working with users to tell them to switch styles in different context. Too many, "oh yeah, sorry, it's not working because here you use kebab case, not camelCase." Other dumb things like "grep ARG" doesn't work because you are using the wrong case style. Just our own experience. Our use case could have been different, but just something to think about.

[sharp zealot]

Yeah that’s a tricky one. I was wondering about it too. In your case the camelCase is in your DSL?

[obtuse lion]

yes, camelCase is what we prefer

[sharp zealot]

It’s even worse in our case because there’s a third layer: the Go or Python functions behind the GraphQL scheme 🫠

• Go: HelloWorld
• Graphql: helloWorld
• CLI: hello-world

[obtuse lion]

we have warts too because we prefer camelCase but because of k8s and DNS names we are forced to kebab-case for some keys. Not pretty.

this is also why we moved away from translating kebab to camel is that technically our system doesn't enforce or assume a case style. It's just convention. But by doing the translation we ran into weird corner cases where people put kebab in the DSL and then it was super confusing because from the CLI you'd expect the DSL to have camelCase, but that's not what the user did. So in the end we just dont assume anything but tell people we like camelCase.

I've been fooling with zenith in weird ways, but just realized now I don't actually know how to use it in the most basic way. If I write a module, how do I consume the module from a regular dagger pipeline, like the basic go example in dagger.io home page but call my custom module. I didn't know how to generate the client code. Like a module calling another module, I can do that. Just a not module calling a module. I feel like I'm overlooking something really obvious.

[sharp zealot]

That’s on us, we haven’t spent a lot of time designing that use case in detail. We know we want it to work well, and we know it’s possible. That’s as far as we’ve gone. But you asking about it is the perfect excuse to dive in.

[obtuse lion]

I really want to be able to attach a debugger to a running module. Any tips on how you can do this?

[warped canyon]

bumping this for @latent trellis

[obtuse lion]

This probably makes no sense, but I got the following to work. It's using AML as dagger a module. So the below

// Fancy hello
define Hello: {
    Greeting: "Hello"
    Name: "World"

    // Say hi
    Message: function string {
        return: "\(self.Greeting) \(self.Name)"
    }

    // Say hi, but loudly
    Shout: function string {
        return: std.toUpper(self.Message()) + "!!!!!!"
    }
}

You can run dagger call with-name --name dagger shout and it will output HELLO DAGGER!!!!!!

You can do real stuff by just writting expressions like dagger.container().from("alpine").withExec(["apk", "add", "curl"]).withExec(["curl", "https://dagger.io"]).stdout()

I'm still working on rounding out the edges, specifically I need to write a custom sdk for this.

The simplest hello world module in AML looks like

HelloWorld: function string {
  return: "Hello World"
}

[warped canyon]

https://tenor.com/view/the-universe-tim-and-eric-mind-blown-mind-blown-meme-mind-explosion-mind-explosion-meme-gif-18002878

[obtuse lion]

seems like there could be some UX improvement for with-foo --foo bar That seems like a pattern to set a field, but it's verbose

For the internal state fields there seems to be an assumption that they are lower case for example. WithField("Foo") is assuming the json returned from the module is {"foo": ...}, not {"Foo": ...} So it's weird in AML because I need to define the field as WithField("Foo") so the that CLI arg becomes --with-foo, but then I have to internally treat the field as foo. But AML is basically just json so there not really a notion of marshaling tags to change the case. What I'd prefer that I could just return {"Foo": ...} from the module and it works.

[obtuse lion]

but more specifically with this I was trying to make it so that capitalized fields would be "public" and get a WithFoo method and lower case fields would stay private and just be marshaled state internal to the module.

[warped canyon]

Yeah I've generally gone away from that pattern for functions that are intended to be consumed by the cli vs used in another module

[obtuse lion]

Is there a better approach? What would be the style of the function to set a field?

[warped canyon]

I'll link an approach im trying when I'm back on a computer! I'm not saying it's the answer, but I'm trying it out and it feels good so far

[latent trellis]

What's going on with those timers?

[warped canyon]

Is there a better approach? What would

[obtuse lion]

Writing an sdk was ridiculously simple.

My first module: dagger call -m github.com/ibuildthecloud/daggerverse/hello with-greeting --greeting "OMG," with-name --name "this works" shout

Of course I couldn't just do a hello world, so this is using my custom AML SDK which integrates dagger and AML. Lots of broken stuff, but now that I go the end to end scaffolding I'm going to start using it to run the actual CI of AML itself. Because of course the CI for AML should be in AML 🙂

[warped canyon]

🤯

[obtuse lion]

This is so magical and cool. I love it.

I'm taking a break now, but I'll get to AML CI inception tonight.

[obtuse lion]

Made it official https://daggerverse.dev/mod/github.com/ibuildthecloud/daggerverse/hello@214762c0b6c5b6510cfa70e35be22118840d254c

looks like daggerverse is say "dagger mod use " when i looks like the CLI is "dagger mod install" now

[rocky harbor]

Good catch, will fix (dagger mod use still works, but is just an alias to install now, which we felt was a clearer command name)

[rocky harbor]

Great to hear! Congrats on writing the first third-party module SDK 😎

[rocky harbor]

I've tracked down my mysterious tmate

[rocky harbor]

Which brings me to another issue:

[obtuse lion]

it would be nice if ./vendor was supported in modules. You end up with an error like ```exec /usr/local/bin/codegen --module . --propagate-logs=true --introspection-json-path /schema.json ERROR [0.57s]
Error: load package ".": err: exit status 1: stderr: go: inconsistent vendoring in /src/amlsdk:
github.com/99designs/gqlgen@v0.17.34: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

    To ignore the vendor directory, use -mod=readonly or -mod=mod.                                                                          
    To sync the vendor directory, run:                                                                                                      
            go mod vendor                                                                                                                   

[obtuse lion]

And the reason I’m doing vendor is because 1) goprivate 2) go mod replace. It seems like the sdk module wants to do “go mod tidy” which is mostly impossible with my go.mod

[sharp zealot]

on a plane the dagger CLI takes a LONG time to exit, after it has completed its call and printed everything to the screen. I suspect cache flushing or other dagger cloud communication

cc @dense canyon