#Engine wide GOPRIVATE and GOSUMDB

1 messages · Page 1 of 1 (latest)

worn horizon
#

This is a good question. There used to be a way to set custom env vars that got picked up by the engine using a _DAGGER_ prefix but I can't find any of that documentation anymore. When I last tried setting GOPRIVATE using that env var within my custom engine it worked. Now I can't find it. I was hoping to see it here - https://docs.dagger.io/configuration/custom-runner

mossy sigil
mossy sigil
worn horizon
mossy sigil
worn horizon
tiny gull
# worn horizon Thank you! Even with that code search, I am not really seeing what needs to be s...

👋 we've refactored this recently and we've moved those variables to the dagger.json SDK config so you don't have to specify them at the engine level: https://docs.dagger.io/api/packages/?sdk=go

I think GOPRIVATE is supported for now . For GOSUMDB, I'd assume we'd need to add another system level engine variable like GOPROXY here: https://github.com/dagger/dagger/blob/4116659290b496b2e749973e0484c8097834151a/core/sdk/go_sdk.go#L576-L584

worn horizon
# tiny gull 👋 we've refactored this recently and we've moved those variables to the `dagge...

Hey @tiny gull , for something like GOPRIVATE, GOSUMDB etc, we don't want to have consumers of our modules define it on every module. Those values don't change so we would want to be able to set them as a engine default variables when we build our custom dagger engines. Similar to the proxy config. Last I checked GOPRIVATE itself didn't work. Had to set it as a _DAGGER_CUSTOM_ENGINE_VAR_GOPRIVATE or something like that.

tiny gull
#

_DAGGER_CUSTOM_ENGINE_VAR_GOPRIVATE Unless I'm misreading the code, this setting at the engine level should not be working since a while back

worn horizon
#

My situation is a bit awkward because we currently don't require either GOPRIVATE or GOPROXY. But it's imminent as we are UAT testing those for blocking external directly module traffic from public proxies. So, when that happens I will need these to be properly set up.

GOPROXY afaiu, is not enough by itself. Internal git repos are not proxies via GOPROXY. That requires GOPRIVATE. I am unsure about GOSUMDB though, that maybe satisfied by GOPROXY.

As for where to set them up, if they are in the user's dagger.json, every user AND module author will have to remember to set them in their dagger.json? That sounds counterintuitive for these global vars. We'd want to set it up once as we already distribute a custom engine.

Related Q: if a module author sets those values in their dagger.json do consumers inherit it? How does that interaction work?

tiny gull
worn horizon
#

I remember testing it

tiny gull
#

GOPROXY afaiu, is not enough by itself. Internal git repos are not proxies via GOPROXY. That requires GOPRIVATE. I am unsure about GOSUMDB though, that maybe satisfied by GOPROXY.

My understanding is that it really depends how the proxy is set. If your proxy has access to internal repos, then it should be able to serve the cached artifact for the package

#

otherwise, you need ot fallback to GOPRIVATE which currently relies in the user's .gitconfig or .netrc to authenticate towards the target package

tiny gull
#

Related Q: if a module author sets those values in their dagger.json do consumers inherit it? How does that interaction work?

not sure what do you mean by "consumers inherit it"

#

Dagger will honor whatever GOPRIVATE's are set in the module basically

#

so, in summary @fierce lintel, if you set _DAGGER_ENGINE_SYSTEMENV_GOPROXY in your dagger engine, that should work

worn horizon
tiny gull
# worn horizon There's a module hierarchy right? Let's say I have module A (internal daggervers...

there's two things happening here:

  1. If you're in the context of an organization and you have your own internal GOPROXY, then setting GOPRIVATE shouldn't be necessary as the proxy should be able to fetch the modules for you.

  2. If you're not using a global GOPROXY then yes, each module will have to define their own GOPRIVATE depending on the private modules that they use. If you think about it from the least privilege concern PoV, it actually makes sense that each module only specifies the particular packages that it requires private access to.

Again, the reason for setting that at the dagger.json level is so you don't have to reconfigure your engine for GOPRIVATE to work. The engine now dynamically reads the SDK config variable and adjusts the corresponding Go runtime for the module appropriately.

iron vessel
worn horizon
#

the reason for setting that at the dagger.json level is so you don't have to reconfigure your engine for GOPRIVATE to work

I can't validate GOPROXY vs GOPRIVATE yet so that'll have to wait from my end. However, since we have to set GOPROXY at the engine level, why would we ask users to set GOPRIVATE I don't see a benefit as we still have to create a custom engine. Users already have a lot to configure (things like _EXPERIMENTAL_DAGGER_RUNNER_HOST), the more we ask them to configure the harder it is to get adoption.

tiny gull
#

However, since we have to set GOPROXY at the engine level, why would we ask users to set GOPRIVATE I don't see a benefit as we still have to create a custom engine

if you have GOPROXY set at an org level and the proxy can access private modules, then GOPRIVATE shouldn't be necessary

#

GOPRIVATE at the SDK level is mostly for users which don't have a private GOPROXY and are still required to pull packages from private repos

worn horizon
#

Ok, i guess we'll cross that bridge when we get to it 🙂 I don't have a way to test that yet. I was under the assumption, private repos don't go through the GOPROXY.

#

Another thing to note is that it should be rare that a dagger module itself would require a private Go module to operate. I don't have that use case yet.

tiny gull
tiny gull
# iron vessel Hi <@336241811179962368> , afaik, if I'm using private go modules, setting `GOPR...

IIUC the GOSUMDB is entirely optional. From the Go spec (https://go.dev/ref/mod#private-module-privacy)

In addition to proxies, the go command may connect to the checksum database to verify cryptographic hashes of modules not listed in go.sum. The GOSUMDB environment variable sets the name, URL, and public key of the checksum database. The default value of GOSUMDB is sum.golang.org, the public checksum database operated by Google (privacy policy). See Checksum database for details on what is transmitted with each request. As with proxies, the go command does not transmit personally identifiable information, but it does transmit the full module path being requested, and the checksum database cannot compute checksums for non-public modules.

In addition to that, there's also this:

A proxy may mirror the checksum database. If a proxy in GOPROXY does this, the go command will not connect to the checksum database directly.

Also re authenticating modules: (https://go.dev/ref/mod#authenticating)

If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules.

So again, haven't verified this but seems like the checksum DB is generally used when downloading the files locally to your project. Once the go.sum is present (generally in CI), it shouldn't query the GOSUMDB since the package hash can be verified locally

worn horizon
# tiny gull > However, since we have to set GOPROXY at the engine level, why would we ask us...

Hey Marcos, so this came up internally today. We haven't implemented the proxy blocks yet but will soon, so this will become a blocker for us at some point.

You are right in that if GOPROXY is a fully private proxy, there's no need for GOPRIVATE. However, because of internal authentication, GOPROXY will require a username/token like GOPROXY=https://user:token@artifactory.mycompany.com. Even then we have highly confidential stuff we don't expose to the internal GOPROXY. In that case we will need the GOPRIVATE as well as GONOSUMDB set alongside GOPROXY

We use artifactory so more about how this is set up in Arty is documented here - https://jfrog.com/blog/why-goproxy-matters-and-which-to-pick/

Bottomline, we will need the ability to set GOPRIVATE engine wide if we want to use custom Go modules within (to supplement) dagger modules. I assume setting username/token to GOPRIVATE will be painful, so maybe even a way to pull that into the engine from outside like how .gitconfig is pulled in today.

If we are only using publicly available Go modules, there is no need to set user/token in the URL for GOPROXY

tiny gull
worn horizon
#

Hey @tiny gull , thoughts on this?

tiny gull
#

However, because of internal authentication, GOPROXY will require a username/token like GOPROXY=https://user:token@artifactory.mycompany.com

^ this shoudln't be an issue shouldn't it? Given that GOPROXY can currently be set as en engine env var.

worn horizon
#

It is, because I don't want to hard code a credential inside the engine env var

tiny gull
#

@worn horizon question about access to private repos in your case. Is your requiremento based on:

  • Accessing private repos to be able to call modules i.e: dagger -m $private_repo call fn
  • Accesing private repos Go libraries with Go module's i.e: private dependency in the module's go.mod file
tiny gull
#

in your local machine, you could start the engine with docker run --privileged... -e GOPROXY="op://wherever/my/cred/is"

tiny gull
worn horizon
# tiny gull not sure how you'd make this different given that these types secrets are usuall...

hmm, so this is not a Go specific problem. I've reported something similar for Maven and I'm assuming the same exists for Python. These package managers, require credentials. If the dagger module itself is using private dependencies, it's not ergonomic to inject the creds into the engine.

Let's focus on Go for now. You mentioned the local engine can be started with docker run --privileged... -e GOPROXY="op://wherever/my/cred/is". But I don't want the user to manually start the engine. We don't require them to manually pass in the gitconfig, or docker credentials today. The engine picks them up from the Host. This is inconsistent in this case.

Right now, in all our use cases we use the CLI to provision our custom engine. We don't do docker run at all

#

The thing that's unique right now in Go, is that the only way (AFAIK) to set the proxy creds is in ENV var. But with maven you can set it in settings.xml, and python in pip.conf

#

Bottom line is, we can avoid having to set credentials at all by setting GOPRIVATE in the engine. What's the drawback of enabling that for custom engines?

tiny gull
worn horizon
#

Correct, the second part (using user git credentials) is already handled by the engine.

tiny gull
worn horizon
#

It goes even further, it supports PAT auth with HTTPS too 🙂

tiny gull
#

PAT support was initially designed so it's possible to call modules in private repos

worn horizon
#

You mean for the engine? The git creds that get pulled in definitely works for private modules

tiny gull
#

oh nvm, yes it'll work

#

found the code that does that

worn horizon
#

@kind sinew covered most of the use cases, if not all 🙂

#

I haven't had that use case myself to test, so I couldn't immediately confirm. Glad you found the code

tiny gull
#

so hardcoding a credential in your .gitconfig is going to work

#

but credentials managers wont

#

let me double check with Guillaume today

kind sinew
kind sinew
#

I think it might be worth creating a corresponding issue Marcos -- happy to do it

tiny gull
kind sinew
worn horizon
#

Great analysis @kind sinew ! The one thing I want to clarify is, would the proposed solution work as a default for custom engines? Or would all module creators have to add the config to dagger.json?

tiny gull
tiny gull
#

@worn horizon just a heads up that @kind sinew will be leading the effort to come up with a proposal so we can standardize all the authentication mechanism so the engine, module and the SDKs they all play along together 🙏