A module can't access the root dir anymore

[quaint crest]

QQ, A common pattern is having a "ci" directory with the module code. Does this mean I have to pass in something like --source=../ for all function commands? Or is the expectation that I will do the dagger mod init in my actual source directory and make "ci" a submodule? How does that workflow look like?

[deft gulch]

I don't think that pattern changes, unless I'm misunderstanding. The functions should take a directory as an argument, rather than assuming the project's code should automatically get loaded in with the module. For example https://github.com/kpenfound/greetings-api/blob/main/ci/main.go#L17 = dagger call -m ./ci test --dir .

[sullen pewter]

@quaint crest you can have the module be the root of your repo, and set source to ./ci

That way this works:

$ git clone YOUR_REPO repo
$ cd repo
$ dagger call test --dir .

The --dir . part is still more verbose than regular scripts, but we will get to solving that problem next 🙂

Your Dagger config would look like this (at the root of the repo):

$ cat dagger.json
{
 "name": "foo",
 "sdk": "go",
 "source": "./ci"
}

[quaint crest]

Oh.. that's requires a change in mindset :). Now source is the module source.. hmm

[sullen pewter]

Did that field mean something else before?

I'm interested in understanding how you're groking this change.

[quaint crest]

it (source) wasn't a thing before, but when I thought about "root" I mentally mapped that to "my application source code"

[sullen pewter]

Right. We're making it more explicit that Dagger modules, although they can live in the same repo as your app, don't have special ambient access to the rest of the repo. Where you host your module doesn't change which files it has access to. Source directories always have to be passed explicitly as arguments, etc

So they are really like code - just like Go modules, just multi-language, and targeting a different runtime

[quaint crest]

Makes sense. The one downside I see is having unwieldy, long CLI calls. Imagine having to pass in the source and then 3-5 environment variables. etc. Almost makes it necessary to create a Makefile.

I understand that may be just an intermediate state until more changes are done to the UX in the future.

[quaint crest]

Excludes and Includes will also be flags right? Today, with root I can do exclude and include in dagger.json

[wintry vessel]

Yep I have an idea for that. If Directory is --src we could also create --src-include and --src-exclude.

Or --src.include to avoid a conflict.

[quaint crest]

lots of flags

[ebon pawn]

If we’re saying that all pipelines are a collection of modules with functions, then we could say targets are the functions we want to call and their arguments.

While people can always use dagger up and dagger call, we could add a targets configuration to the json to codify targets in the location they are run.

[high pike]

I’m trying to understand what changes should be made when upgrading to the next version.
Having to pass includes/excludes to each command is definitely not a welcomed change for me though :/

I like using the dagger command. If I have to start using scripts/make/<insert your task runner> to avoid always typing a bunch of arguments…..well, that’s definitely a huge step back from a UX perspective.

[ebon pawn]

Those targets already exist, it’s just whether you want to codify them or not. This was easy pre-zenith, but now that modules are containers - I’m not sure what the alternative is? I don’t think we can programmatically call a module yet?

[placid fractal]

I think that the current method of passing environment variables to our modules via CLI arguments, along with this change, will make the process of handling complex executions a bit messier. In other words, we will need to pass a verbose set of arguments to the CLI to get certain behaviors to work. To simplify this process, a proper wrapper such as a Taskfile or makefile should be put in place as a command executor (for me, isn't a big issue; I'm already a big fan and user of Taskfile(s)). I understand that there are tradeoffs to be made here, but I believe that the composition model is a core part of Dagger and should remain as such — I can accept that ❤️ .
@ebon pawn — programamtically if you're installing a certain module, you can call it if the functions are exposed as members of the module https://docs.dagger.io/zenith/developer/go/191108/advanced-programming#use-modules-in-other-modules

[sullen pewter]

We plan on closing that UX gap with a thin convenience layer on top of functions.

In the early Zenith design we tried to solve it all at once ("entrypoints") but got pushback from testers because it was a lot to digest.

[quaint crest]

Curious what your thoughts are on how that convenience layer looks like. I think this is a great start in terms of isolation and organization.

Being able to use modules from non-module code may alleviate some of the concerns as we can write wrappers around dagger but that's not much better than using a task runner to accomplish the same. As you said, we need something to close the gap 🙂 . I only want to use dagger and no other tools. If I need other tools, I will encapsulate them in dagger.

[sullen pewter]

We're taking a layered approach instead, which is more digestible (functions; modules) at the expense of getting all the convenience all at once

[sullen pewter]

I think (design tbd) that it could be as simple as saving "dagger call" invocations in your dagger.json, calling that a "script", and invoking them with dagger run

or just pointing to a shell or node script (dealer's choice) and dagger running that

Then you're down to:

$ git clone MYREPO repo
$ cd repo
$ dagger scripts
build
test
deploy
$ dagger run build
$ dagger run test
$ dagger run deploy

And dagger call is the underlying plumbing for power users

[ebon pawn]

I feel at this point, you’d be better sdding a BUILD file and giving us a small DSL to add modules, source dirs, dependencies, arguments, targets, etc.

This gives us autocomplete and a code like experience without multi lang complexities and in container compilation.

[sullen pewter]

if @young cypress sees this message I'm in trouble. Prepare to hear about the wonders of Just 🙂

[sullen pewter]

I think we can avoid this. I would rather not have to evangelize one DSL over others - been there done that 😛

I'm thinking of something like npm scripts: just a small config that lists entrypoints and how to execute them. The default interpreter would be dagger call, but you can also execute the scripting tool of your choice - infinite choice of DSLs that way

Alternatively we could embed a lightweight unix shell, and make that the default.

I like that option but it would be more work - how do you embed something that is reasonably equivalent to bash into a Go tool?

[ebon pawn]

I'll leave it with you and your team, you always surprise me with solutions that exceed my expectations

Just don't forget the monorepo use-case 😄

[sullen pewter]

Ha ha thanks 🙂 To be clear your feedback is super useful and always appreciated.

And yes monorepo support is crucial, impossible to forget even if we wanted to

@ebon pawn do you happen to know of any pure Go shell implementation with a good reputation? Kind of a niche project but I'm sure others have felt the need

Well look at that 🙂 By our friend mvdan https://github.com/mvdan/sh

Aha! This tool uses mvdan's package exactly the way I had in mind:

https://github.com/cortesi/modd

By default modd interprets commands using a built-in POSIX-like shell. Some external shells are also supported, and can be used by setting @shell variable in your "modd.conf" file.

[ebon pawn]

mvdan/sh is the only one that I've seen that's still maintained

Do you need a shell though, or should the charm projects provide enough?

https://github.com/charmbracelet

Oh, modd seems cool

You'd need to swap out the dependency on growl though for macOS

fsnotify should do

[high pike]

fsnotify is a pain, but works reasonably well given the challenges it faces (namely, cross-platform support). 😄

[ebon pawn]

It is a pain, but it's just one of those tricky macOS things

We fought with it a lot at InfluxDB too

[high pike]

My PTSD with it comes from Viper

I'm a little bit concerned that fragmenting the Dagger execution path may lead to suboptimal developer experience. That's exactly why CircleCI's config format is such a pain.

But I haven't been disappointed in Dagger so far, so I'm confident you'll come up with the right solution. 🙂

[sullen pewter]

Can you clarify what you mean by "fragmenting the Dagger execution path"? Too many ways to do the same thing?

[high pike]

Too many different places to do one thing.

Right now dagger call ci -> a function called Ci in my CI module.

If I understood the above proposal right:

• dagger run something -> some "task runner" config file calling a module and a function
• you figure out the function from there and go to another place
• which steps/configuration go where?

The mental model is more complex this way IMO. Also, if everything is in one place, it's easier for newcomers to figure out what to do, even if they have no idea what they are doing.

[sullen pewter]

Yes that's a risk. We'll have to be careful. I think the benefits are worth trying though.

The key is to make sure it is layers, not French fries. If everyone always wonders whether to use run or call, then the design isn't working.

I think of it like git's plumbing vs. porcelain. There's a use for both, but nobody is confused on which to use when. That's an example of a good layer design. https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain

[high pike]

Some people may argue with that statement: most people have no idea about the difference and when they ask what to do, the answer is usually "don't use porcelain" 😄

But I get the point

[sullen pewter]

I personally never use git plumbing 😛 Porcelain all the way over here

[high pike]

Well, avoid stackoverflow then 😛

That may be good advice anyway 😄

[sullen pewter]

(unless I'm scripting git, which I only intend to do once: to write the ultimate Git Dagger module, then use that forever for all my Git automation needs)

[high pike]

Also, I might not be the best person to test a new structure on: I'm used to Dagger's current model and shifting to a new one is always hard, especially when I'm already running Zenith in production, so there is that. 😄

[ebon pawn]

dagger call has no access to the host, so in theory when dagger run is done, call should really never be used again?

[high pike]

But I do wish that there were better changelogs. With every new release, there were a few things I had to figure out the hard way. :/ (I know, I know, it's not stable yet). But a change like this should be well-documented.

[high pike]

That sounds like an easy to understand explanation.

[sullen pewter]

cc @distant fern @tired gull @wintry vessel

[sullen pewter]

call is more flexible. You can chain functions into a dynamic pipeline, quickly iterate in the terminal. It also is stateless, you don't need to edit a config file to then run its contents

[distant fern]

Agreed. Early on it was decided to not put anything about Zenith in the release changelogs, since it was not a feature that we wanted to advertise. Here is another example of that: https://github.com/dagger/dagger/pull/6546 . This approach allowed us to go fast, and keep shipping big changes, including breaking ones, with no time spent on the release changelog. I honestly don't think that we could have done anything better given all the constraints and requirements. I expect us to be a lot more strict about capturing breaking changes once Zenith is public.