Same dagger module multiple git locations/refs

[errant bronze]

We have situation where I have multiple Gitlabs (A, B, C) each with their own url git.enclaveA.local, git.enclaveB.local, etc, Data is sync'ed among them but all CI software talks to their local Gitlab and can't talk to the others. Suggestions to managing dagger modules in this situation. The main ideas I have come up with:

1. Monorepo and all module are local
2. Use git submodules with relative paths and load modules locally
3. Find a way to do git url rewrite rules. gitconfig supports this via insteadOf. Not sure if the engine supports custom gitconfig. Is there a way to inject gitconfig into the dagger engine?
4. Something else I am not thinking of.

[hollow quarry]

Does the CI configuration have an env variable with the address of the current gitlab server? If so you could use that to parametrize the dagger call command

eg. somewhere in the CI config: dagger call -m git.$ENCLAVE.local/daggerverse/mymodule ...

[errant bronze]

Yes. Gitlab supports that. But what about transitive dependencies?

I am mostly thinking about the module dependency case

[hollow quarry]

oh right! Didn't think of that.

Do you have a solution for Go packages? They probably have the same problem

Is this a commercial or free version of Gitlab?

[errant bronze]

Athens and git url rewrites for go. Here is an example of what we do with go: https://docs.gomods.io/configuration/authentication/

Commercial everywhere. And one nuance: A is source of truth for B and C. And B is source of truth for C. So what that means is that if I write modules in A I don't want to muck with the git history in B or C.

[hollow quarry]

It's probably the safest bet for Dagger modules as well, they are heavily insourced by Go modules.

[errant bronze]

There are 2 cases for athens. Go cacheing proxy. So all modules are downloaded as packages and they can be moved to other Athens instantiations. Or go packages we write ourselves, which is the Athens + rewrite rule.

Which is where option 3 came from tbh.

https://docs.gomods.io/configuration/authentication/#atlassian-bitbucket-and-ssh-secured-git-vcss

That is a better link to the rewrite rules.

[url "ssh://[e-mail]:7999"]
    insteadOf = https://git.example.com/scm

https://git-scm.com/docs/git-config/#Documentation/git-config.txt-urlltbasegtinsteadOf

I guess I am uncertain who does the module dependency resolution. Is that the cli or the engine? So not sure where or how I would inject the git config and if it would be respected.

[hollow quarry]

It's the engine.

[errant bronze]

Any ideas on injecting git config? I am not blocked necessarily since I have options 1 or 2…

[hollow quarry]

paging @old obsidian ...

you're going to love this one 😛

[old obsidian]

We don't have support for that type of git url rewriting atm (dependency resolution happens engine-side, to answer the question there), so 1 or 2 is the best bet in the immediate term.

I'm digesting the problem still, but if nothing else I think we could support loading url-rewrites from the caller's .gitconfig. That might be simplest for end users.

The only other thing knocking around in my head is more generic support for re-mapping any "sources" like this, which is something buildkit upstream has some support for and might cover both this situation and others (i.e. the same conceptual problem could happen with image registries). But that approach also increases the complexity potentially for users and us as dagger devs, so not sure if it's the right first step.

(thinking more )

[old obsidian]

Yeah after thinking more I suspect adding support for those url rewrite rules is our best bet here. I remembered that we actually do already support registry url rewrites if you run a custom engine and give it a custom config file. So it makes more sense to just add support for the equivalent on git urls too rather than trying to get super generic with arbitrary source rewriting (too much of a separate can of worms).

We can support the git url rewrites being set in the engine config file and optionally loading them from the caller's .gitconfig (which is particular is going to be way more convenient 99% of the time I think). This would also be consistent with supporting optionally using the caller's SSH_AUTH_SOCK to get private repos.

I'll open an issue

https://github.com/dagger/dagger/issues/7787

[errant bronze]

Awesome. 🙏

Also do you have a link to the registry rewrite config or a quick explanation?

[fresh cipher]

@errant bronze thinking out loud here if a stopgap here couldn't be adding some sort of common DNS amongst the different gitlab instances.

i.e: if you are in git.enclaveA.local and you try to resolve git.local or something similar within that enclave, the DNS with always resolve to the local enclave. This way, all your modules could use git.local and they should work regardless of which encalve is running the pipeline.

Thoughts?

[errant bronze]

Big problem is ssl and trust

We could maybe do something but then I have to disable strict ssl in the engine for git

[fresh cipher]

Is that so? If you're managing your own certificates, each gitlab enclave could also have a certificate for git.local as well as their own enclave fqdn

[errant bronze]

Each enclave has an its own DNS and certificate authorities managed by others. Think large multi tenant enclaves.

[fresh cipher]

Makes sense.. it should work though, shouldn't it? I guess the most prohibiting aspect is that this requires core infra changes 😬

[errant bronze]

Yep. Hoping to avoid it though. Def an option

[errant bronze]

It would be similar to AWS code commit. https://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-git Each region has its own endpoint and if I was only allowed to talk to the region( or that was my preference) I am in but my modules are replicates across regions. And I can’t ask AWS to support a local dns and provide a SAN.

[fresh cipher]

that makes sense. That's a good argument

[hollow quarry]

@errant bronze in the context of go modules, do you every rely on vendoring or pre-baking to simplify the problem? Pre-fetch dependencies in a less restrictive environment to be less dependent on networking/url rewriting hacks in prod?

[errant bronze]

Yes we do with Athens and go cache. Basically let the proxy cache to storage capture and move

And the host Athens each place with the captured modules

Summary: go proxy everywhere and move the cache from less restrictive

[hollow quarry]

But go already has a cache, no? If you disabled go proxy & pre-baked the regular go cache, wouldn't you get the same result?

[errant bronze]

Pre bake where? Per container and developer machine?

[hollow quarry]

My guess is that perhaps standard go tooling doesn't guarantee offline builds, even though in theory it could