#multiple repos?

1 messages · Page 1 of 1 (latest)

bronze solar
#

preemptive 🧵just to separate concerns from original question

#

here are some pros: Go tooling "just works", GitHub releases aren't confusing (linear history, 1 product per repo), establishes conventions for third-party SDK authors to follow, probably easier to find, issues/discussions are categoriezed appropriately without needing labels/1st-level triage, org-wide Projects already mean it doesn't matter how many repos we have for the roadmap

fresh sentinel
#

@bronze solar well, I was actually picturing your recent day to day as an example

  • Make a few changes to the API

  • Open a PR

  • Merge

  • Go to the Go SDK (and every other one after that)

  • Bump the engine dependency (how? go get @ commit? not do it until we release ?)

  • If something broke how to deal with that? (Go SDK CI didn't run on the API PR)

  • How do you write tests using the Go SDK for the new API pieces? There's a chicken & egg problem -- you need to push the API bits, update engine dep on SDK, re-gen SDK, go back to main repo, update engine dep on SDK, etc

#

maybe can be solved with some replace rules in go.mod, but the PR coordination problem / inter-versioning etc will still be there

#

And that's just one facet.

For instance, I recently changed the comms between extension and core so that core injects a DAGGER_HOST inside the container running the extension, and the SDK honors that

In multi repo, it would have been a mess to do that -- while I was able to do it in a single PR

#

But I agree it's not clear cut -- there's a lot of cons

bronze solar
#

you're right it does introduce extra toil today. I wonder if this is an acute pain felt today though, which might trend downward as the API stabilizes and in total be a lesser pain to the downsides of a monorepo, which are more permanent(-ish)

fresh sentinel
#

To me it just seems "safer" to go the monorepo route, and perhaps revert that once we are more stable/staffed/equipped to pay the toll

bronze solar
#

i'd be curious to look into go.work patterns here, but it's totally true that managing concurrent PRs for one change would be painful

#

or maybe all changes to the SDK are fully automated, downstream from the API

fresh sentinel
#

API stabilizes and in total be a lesser pain to the downsides of a monorepo, which are more permanent(-ish)

Yeah, that's the part that sucks,

Aside from the vanity url for the Go SDK, everything else is not permanent

#

Like, we can split out node SDK with zero trouble

bronze solar
#

yeah true

fresh sentinel
#

and we can deal with Go SDK versioning as well with zero trouble

#

the only part that's annoying, is the import path

#

we can actually split the Go SDK into a separate repo after, that works by updating the vanity URL meta tag

However, we would either need to mimick the same import path or break imports

#

e.g.

We update this the meta tag from:

    <meta
      name="go-import"
      content="dagger.io/dagger git https://github.com/dagger/dagger"
    />

To:

    <meta
      name="go-import"
      content="dagger.io/dagger/go/sdk/dagger git https://github.com/dagger/dagger-go-sdk"
    />
#

and that should allow to split it out

but the import path is nasty

bronze solar
#

i wonder if you could make sdk/go a symlink to ../../

#

for backwards compatibility

fresh sentinel
#

yeah, in the previous snippet backward compat is not broken

#

we're saying the import prefix is dagger.io/dagger/go/sdk/dagger (rather than dagger.io/dagger) and it points to the new repo

bronze solar
#

oic

fresh sentinel
#

but it's a bit nasty 🙂

#

but yeah, we could then try to nicely migrate over a shorter import path (e.g. symlink like you were suggesting)

#

or, since it's version locked, we release a new major and people have to update import path, not a big deal I guess?

half salmon
fresh sentinel
#

anyway:

  • I agree with you there are a ton of pros in splitting up

  • however, I think we'll pay the price day to day. At least until we're semi-stable ish, which we're definitely not today

  • it's still possible to move away from a monorepo -- and I think we should probably start from that

half salmon
fresh sentinel
#

Some day that kind of work will be much more occasional. But today it's where we need it the most IMHO

half salmon
#

my guess is that the extension work will be primarily in the engine code & not affect sdks much except maybe once in a big bang way when we launch extensions

ruby furnace
#

This is definitely really stupid, but for the sake of throwing out every option... 😅 What if we had both? As in all development is in dagger/dagger, but some post-commit process syncs dagger/dagger:sdk/go/dagger -> dagger/dagger-go-sdk:/. Then dagger/dagger has a local replace for the sdk, but dagger-go-sdk gets independently versioned and vanity url

bronze solar
half salmon
#

you guys want to chat about this live? I have to say the multi-repo option is tempting to me. But @fresh sentinel ‘s concerns are valid and usually ignoring his concerns gets me in trouble

#

(in trouble with the universe)

ruby furnace
#

I'm free

bronze solar
#

I can join in a sec

fresh sentinel
#

mind if we do this a little bit later? i didn't have lunch yet

half salmon
#

ok 🙂

fresh sentinel
#

👍

#

going to get something to eat

rain rune
#

I'm gonna go even further than @ruby furnace... What if we host our own git repos at our vanity urls... and proxy them back to github?

bronze solar
#

maintains a single source of truth so there's still just one atomic PR across API/SDK/etc

rain rune
rain rune
half salmon
#

One important piece of context in case it helps: I believe eventually all SDKs will be 100% generated. We will provide a cloud service that, given a certain dagger project (source code + extensions) will generate all the SDKs for you and distribute them to your devs, either 1) by opening a PR to your repo, or 2) hosting a special repo, or 3) distributing the native language packages.

I think of our own core SDK as the first "customer" of this service.

#

Not sure if this helps or makes things more complicated 🙂 But certainly it makes me less allergic to the idea of developing custom tooling or automation, as long as it can become "dogfooding" of the above

small rock
half salmon
#

I think the problem with the "one monorepo + proxys" is that it's more moving parts for development. You still need everything to work when developing in the monorepo

ruby furnace
half salmon
#

THEN you also need everything to work in the proxies

#

And now you need to worry about interference between the two

#

Either that, or development does not work in the monorepo alone, you need a special proxy tool to develop

bronze solar
#

since you can just use go.work/go.mod to keep everything within-repo, and the external repos are like readonly views

#

pretty sure we'd just pretend they don't exist (except when we're releasing I guess)

half salmon
#

Does the original monorepo plan (with sdk & its vanity url in subfolder) work properly in-repo today?

#

ie. it's only when you try to import the sdk from that pretty url that things break?

bronze solar
#

i think so yeah. but the original monorepo plan also involves two modules, which comes with some minor inconveniences like having to cd sdk/go && go generate ./...

#

iusntead of running that from the root

half salmon
#

sounds pretty minor 😉

fresh sentinel
#

Done with lunch — walking back

bronze solar
#

well, that's just one thing 😛 it's any go command. it's minor, but it's the kind of thing that bubbles up to all kinds of tools and can add up to extra cycles (go generate didn't actually hit your package => push package => come back to failed build => begin the cycle anew)

fresh sentinel
half salmon
#

@bronze solar I see

bronze solar
#

again not a huge deal, but something avoided by the neat repo extraction hack

half salmon
#

@fresh sentinel we're discussing a variation though

fresh sentinel
#

We got some of the downsides of multi repo, while keeping the atomic nature of monorepo

half salmon
#

(ideas of a proxy etc, see above)

#

that maybe could mitigate some of the downsides, but I was trying to understand which

fresh sentinel
#

Ah, yeah. That’s a possibility to get a cleaner dist url

#

Requires some automation or infra

#

Not sure what are the downsides though. Do the commits match?

half salmon
#

I don't think they can

#

different tree -> different commit

bronze solar
#

yeah that's one downside

#

i was thinking copy sub worktree (keep history) + go.mod => go mod tidy => commit

fresh sentinel
#

What about filter-tree? That’s still différent commits?

ruby furnace
#

And this is temporary until the sdk is fully generated right?

bronze solar
#

i don't think there's any avoiding that

#

could be wrong

fresh sentinel
half salmon
#

@fresh sentinel for now at least 😇

#

@bronze solar correct 2 different trees will always result in 2 different commits, no way around that

fresh sentinel
#

Back

bronze solar
#

👋

half salmon
#

So... live chat? Or not the right time?

bronze solar
#

works for me

half salmon
#

Nothing like an invigorating exchange of ideas

fresh sentinel
#

works for me

bronze solar
#

you have inspired me for a new emoji alias: invigoratingexchangeofideas

half salmon
#

lol

bronze solar
#

@ruby furnace you free?

ruby furnace
#

Yup!