#Bikeshed: default module source dir?

1 messages · Page 1 of 1 (latest)

fallen violet
#

🧵

#

Bikeshed: default module source dir?

#

Suggestions so far:

  • ./functions (where the code goes for your module's functions. If you don't set this directory, your module has no functions)

  • ./dagger_src (source code for your dagger stuff)

  • ./dagger (your dagger stuff)

severe fog
#
  • ./dag (your subgraph of the :onedag: 🤓)
#

Is there further reading for this change? Curious what use case it's optimized for (or is that part of the question?)

#

e.g. is this for Dagger module authors creating repos with the express purpose of module publishing, or is this for Dagger module consumers who are using Dagger within an existing repo (i.e. the product they build)?

fallen violet
#

It came out of a bunch of threads related to the CLI polish

#

no good paper trail I'm afraid, sorry

severe fog
fallen violet
#

To summarize:

  • dagger init is for everyone now: consumers and producers of modules.
#
  • dagger init by default creates a codeless module, ie. a receptacle for dependencies. It allows to dagger install LONG and dagger call -m SHORT
#
  • Later the consumer graduates to writing their own "main" module, when gluing dagger call in a shell scripts becomes too much. At that point they dagger develop their first functions. At this point the module now has source code for functions of its own.
#
  • Of course sometimes you intend to publish the module from the start, in which case you would either dagger init && dagger develop right away, or as @golden jewel proposed it: dagger init --sdk as a convenience so you don't have to run 2 commands
#
  • In the case of the consumer: the question is where should I call dagger init? The answer is: the root of your repo. That way you get, finally, a standardized entrypoint 🙂 git clone && dagger call
#
  • Finally: when a consumer upgrades to developing their first functions, with dagger develop: where should the source code go? Today the answer is: the root of the module! In other words: "source": ".". But that tends to pollute your app repo. Most people will want something like "source": "./ci". So maybe we should default to that? Except "ci" maybe isn't the right path to default to. Hence the current bikeshedding.
#
  • Due to how "source": works, no matter where the module's source lives, it can still be at the root of the repo. So you still get to git clone && dagger call
severe fog
#

Ha, it's kind of funny that we ended up with dagger.json in the root of your repo after all, but it makes sense (and is still not required, just a sane default)

fallen violet
#

Right, you can still have sub-modules anywhere you want, and local imports in any layout you wish

severe fog
#

What if we had a .dagger directory instead, and put the dagger.json + source code there?

#

I think @fresh imp suggested that in one of our office hours

fallen violet
#

For cosmetic reasons you mean?

#

I say we reserve the right to support that as an alternative form, in the future. We could just decide to check for both. It seems a bit late to break the current layout

#

Supporting it as an alternative form, would also help writing a dagger.json to the user's home directory by default (like npm -f). I prefer having ~/.dagger/dagger.json than ~/dagger.json. Instead of making a special case just for home, it kind of makes sense to support both everywhere

severe fog
#

Oh interesting. Yeah I don't remember the original motivation, but I mentioned it this time because then we just have to decide on one toplevel name in someone's repo (.dagger), rather than naming two (dagger.json + ???)

topaz escarp
#

you mean like DAGGER_MODULES=$HOME/ci ?

fallen violet
#

Oh one important detail: we discussed defaulting to <dir>/<sdk> in other words, namespace source code by sdk by default. This makes it easier to handle things like switching SDKs without figuring out which files to wipe.

golden jewel
golden jewel
fallen violet
#

@severe fog one downside of .dagger is that it's weird to store source code in a hidden directory. I know because we did this in (some old cue-based version, can't remember how old) and that feedback came up

severe fog
#

Yeah it does feel weird especially if the modules are the focal point of the repo

#

On the plus side, it would appear above .github in repos on github 😇

fresh imp
#

I agree it’s weird but feels like that’s how GHA and circleci do it, so seems like there is at least some normalization of this (recognizing it’s yaml and not code)

severe fog
#

Is this a convention forming, to put tool-specific configs in dotfiles in a repo? I see we also have .changes in our own repo for Changie. It kinda makes sense I guess, wonder how people feel about it generally. People probably spend a lot of time editing .github at least.

(Yielding the floor to further bikeshedding)

golden jewel
#

FTR my vote is dagger/, but no problem with dagger_src/ or dag/ either. My gut feeling is that putting actual code in a . prefixed dir is odd and surprising, but can't really justify it fully. The fact that we use code is sufficiently different from yaml configs to differ from that loose convention. But I also wouldn't strongly object to it either, just a gut feeling.

fallen violet
golden jewel
severe fog
#

I'm good with dagger/ too, only slightly dreading getting it mixed up with all my other dagger dirs, but that's a me problem

river heart
#

I generally dislike directories with underscores, but I can’t really reason why

#

dag may overlap with an actual dag package in the software, ao it should be a setting that can be changed

golden jewel
river heart
fallen violet
river heart
#

Env vars are for changing defaults in a lot of cases (eg. I wanna use mydaggerstuff, because dag and dagger are taken by my software packages)

#

As far as I understood, this is similar to setting DAGGER_MODULE=ci , so I don’t have to pass it to dagger every single time. That’s my new default. I can still override it with -mod if I want to.

#

Anyway, this is a little off topic now I guess.

fallen violet
#

it's just that env variables are for settings exposed to the caller. But source is not exposed to the caller as a setting. It's part of the module's layout, like its dependencies. It would be like having a DAGGER_DEPENDENCIES env var.

#

that particular default is for module authors not consumers

alpine abyss
#

I don't think I am clear why source needs to be defaulted. I feel like there will be a myriad of ways people will name their source directory. Would enforcing it for dagger develop be an option? If you type dagger develop without a --source kindly ask to provide one.

#

Or make it a TUI question if source doesn't exist

#

"Where do you want your module source code to be?" for eg

fallen violet
#

but most devs won't know what to answer. "you tell me", they will think in frustration 😁

#

we default to . currently. It's just not the best default now, with the new best practice of rooting main module in your git root

alpine abyss
river heart
fallen violet
#

Once the source value is written to dagger.json, you never have to type it again

golden jewel
dry cape
#

Surely the case of using multiple SDKs like this isn't the general use case? Surely, if the modules are different, we'd want to have actually different names.

#

I don't generally understand why https://github.com/dagger/dagger/issues/6607 is a feature we want - surely, if we've already set an SDK for a module, then it should be in that language. If you've set it accidentally and want to change it, we should require that users force that option (with something like --force). like hugo new site ./dir/ or git clone ... ./dir works, if ./dir exists and has contents.

golden jewel
# dry cape I don't generally understand why https://github.com/dagger/dagger/issues/6607 is...

Part of that issue is just support for renaming, which I think comes up in common cases where you change your mind about what you want the name to be. Currently that's a fairly painful manual process but could be automated.

I agree that changing SDK is gonna be more obscure and probably only something that would come up fairly early in the development process, e.g. I started writing a module in go but then remembered I hate go and switched to python instead 😄 But it does feel reasonable to support (eventually, not a top priority in my mind) in one way or another.

Having <sdk> in the default source path just slightly eases things in that it would result in the new source not overwriting the previous source. It only applies in that case where the default was used though, so not a general solution. In my mind, it was otherwise harmless, so may as well go with it even if the benefits are marginal. What's the concern with having it there?

dry cape
#

it's just another layer of directories - even though github and lots of tools will collapse this into one, not everything does this.
i just don't really like the extra directory level, if it feels very unlikely that it's actually gonna get committed and used

#

tbf, i am having trouble coming up with reasons that aren't "i just don't like how it looks and would want to clean it up all the time"

fallen violet
#

I don't feel strongly attached to it. It just seemed like it might help keep things simpler by not mixing files from different SDKs. But I don't feel strongly about it.

fringe rivet
#

Yeah, agree that it's unnecessary. If I wanted to change the SDK I'd just rename dagger dir to dagger_go or dagger/go (manually) before new dagger init.

fallen violet
#

@fringe rivet you like dagger/ best?

fringe rivet
#

Yeah, just dagger or .dagger.

golden jewel
dry cape
#

i can still see myself often overriding it, e.g. for my daggerverse

#

but i think, for the case of using dagger modules in a monorepo, the dagger default makes sense

fallen violet
fringe rivet
#

Yeah, me too... using --source. But I get why.

dry cape
#

exactly yeah

alpine abyss
fringe rivet
#

This was a bit strange: dagger init --sdk=go --source=. some/subdir 🙂 Makes sense if you know source is relative to PATH, but you think intuitively it's relative to your workdir, so I almost tried this first: dagger init --sdk=go --source=some/subdir some/subdir.

fallen violet
#

my remaining worry (maybe needs a separate thread) is in making the decoupling between init & develop really work.

#

The goal of decoupling is to make it easy and natural to initialize a module, install and call dependencies before you're ready to choose an SDK answer write code.

How to design this decoupling to make it work great, I don't have strong opinions on

golden jewel
fringe rivet
#

I really like not having to make two commands 🙂 Replacing mkdir ... && cd ... && dagger init .... with dagger init ... && dagger develop.

#

I've been thinking about grouping flags like we group commands. Lots of CLIs do this.

#

So the flags to develop could be separate in --help. Same with global flags for TUI for example.

fallen violet
#

Not following sorry. What would I run for:

  1. Initializing a module without code

  2. Adding code to an existing module

  3. As a convenience: combining the two when that's what you need

#

init in all three cases?

fringe rivet
#
  1. dagger init
  2. dagger develop ...
  3. dagger init --sdk ...
fallen violet
#

I think that's the current design 😁 So you're saying you like it the way it is?

fringe rivet
#

Yes, I'm saying I like it how it is 🙂 Not to require init + develop when authoring.

alpine abyss
#

Initializing a module without code

Dumb question probably. Why is this a common usecase?

#

is it to support dagger install externalModule?

fallen violet
#

yes, and calling them from CLI and shell scripts

alpine abyss
#

Can't dagger install externalModle initialize if not already initialized?

fallen violet
#

as an additional convenience, maybe. As the only way to initialize a module without code, I'd say no.

alpine abyss
#

I can see the parallel with git init so it's something devs are used to doing

fallen violet