ExperimentalPrivilegedNesting regression in v0.9.2

[vague moss]

makes me add a message to start a thread 🙂

[lapis ermine]

Ah, that actually may make sense. The auth is associated w/ the session and ExperimentalPrivilegedNesting now results in each exec getting its own new session...

How are you setting up the registry auth? Do you just do a docker login on the host and then it gets picked up automatically or are you using explicit SetSecret and WithRegistryAuth calls?

I think the new behavior is probably what we'll want to stick with (the fact that the session was shared before was more incidental than intentional), but I realize that's annoying to deal with. So based on the above I'll figure out the changes you can make to get the previous behavior (and/or if there's anything we can add to make it easier)

[vague moss]

I do a docker login initially and in some cases I am also passing in WithRegistyAuth. I can't remember exactly how I'm passing in for these tests. I can look a bit closer when I get back online. I use WithRegistyAuth for publishing iirc and not docker pulls.

[lapis ermine]

Okay so the basic idea is that you need to explicitly provide those credentials to the nested dagger container now, so I just tried this on my Linux box and it worked:

    testRef := "eriksipsma/testdagger:testpush"

        // read the host's docker/config.json as a secret
    dockerCfg := c.Host().SetSecretFile("dockerconfig", "/home/sipsma/.docker/config.json")

    out, err := c.Container().From(golangImage).
        WithMountedFile(testCLIBinPath, daggerCliFile(t, c)).
                // mount the host's docker/config.json as a secret into the container
        WithMountedSecret("/root/.docker/config.json", dockerCfg).
                // this is a nested exec (sets ExperimentalPrivilegedNesting: true)
        With(daggerQuery(`{container{from(address:"debian"){publish(address:"` + testRef + `")}}}`)).
        Sync(ctx)

Was able to successfully publish once I added that WithMountedSecret, failed w/ permissions error before, so it sounds like the same situation you are in.

Let me know if that makes sense + works for you

[vague moss]

Thank you Eric! I will need to wrap my head around this tomorrow. My scenario is I am running my SDK (golang) integ tests which in turn spawn a dagger session and do dagger things. Going by your solution, I could technically do WithRegistryAuth instead right? I would rather do that than have a dependency on the host system docker/config.json as that could technically be in a non-default location.

[vague moss]

hmm, I tried setting WithRegistryAuth to the client container that is spun up as part of my tests. But I'm seeing something strange.

=== Failed
=== FAIL: pkg/images TestMavenBuildImage/TestSidecarAttached (1.09s)
    images_test.go:80:
                Error Trace:    /src/pkg/images/images_test.go:80
                Error:          Received unexpected error:
                                input:1: container.from failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://auth.docker.io/token: 401 Unauthorized
                Test:           TestMavenBuildImage/TestSidecarAttached

=== FAIL: pkg/images TestMavenBuildImage (2.36s)

The docker:dind image I am pulling without going through my company proxy and I am able to pull it on my local machine (outside dagger). However, within the pipeline it's failing. Why it's trying to auth for simple pull of a public image is what's baffling me. I am assuming that WithRegistryAuth messes up something. It's not the same as copying my .docker/config.json like you did above. I am yet to try that method as that takes more work to get going and needs more planning.

[vague moss]

I can technically get around that by using my company proxy like mycompany.com/docker:dind but still curious why the non proxy way fails.

One more issue I am facing with having to manually pass in creds (even WithRegistryAuth) is now I have to make my dagger-in-dagger tests aware of my actual credentials 😦 which isn't fun. Furthermore, if I do pass-through the .docker/config.json file like you suggested above, it may not work for everyone because in a desktop you tend to use a credsStore (which I assume is not passed to nested container?). My .docker/config.json doesn't actually have any auth in it. It looks like this

{
        "auths": {
                "mycompany.redacted.com": {}
        },
        "credsStore": "desktop",
        "currentContext": "desktop-linux"
}

I'd be surprised if that actually worked with nesting.

[vague moss]

Hey @lapis ermine any further thoughts on this? I am unable to upgrade to any new versions of dagger because this breaks things :(. I can try some other options if you have suggestions as well.

[lapis ermine]

Hey Nipuna, didn't forget about you! 🙂 There's been a long backlog of stuff accumulating, trying to get to everything as quickly as possible but it's still taking a bit 🙏

Why it's trying to auth for simple pull of a public image is what's baffling me
I believe that if you set credentials, those will always be used, even on public images. The reason being that even public images are impacted by creds due to stuff like throttling (if you're auth'd you get less throttling, etc.).

So if those creds aren't recognized by dockerhub and you are hitting dockerhub direct instead of corp proxy, then that may add up.

any further thoughts on this?
Yeah I have a few more ideas worth throwing out:

1. Modules actually don't have this problem; they are nested and also connect back to the same session (they were intentionally designed that way, as opposed to the other older nesting where it was more undefined behavior). So, technically if you took your tests as they are today but called them directly inside a module rather than directly on your host, that may actually solve the problem.

I haven't seen this pattern done yet; it's a little tricky because you'd actually need to call go test direct in your module code (i.e. w/ os.Exec), but that should be totally possible and would actually be extremely useful as a general use case outside just your own.

This would solve the problem because you'd invoke your tests from your host with dagger call, your dagger CLI will have access to the creds on the host the same way it does today, which would hook them up to the session that's then available for use in the module.

If this sort of route sounds in the cards, let me know and I can try it out quick to verify the idea. The main roadblock I could imagine is if your go test calls have some sort of strong reliance on executing on your host (besides the creds issue) which gets broken in a module container.

2. If the above is a non-starter, there may better approaches to the whole problem w/ trying to use .docker/config.json directly that you mentioned above (which I totally agree with).

There's a library github.com/google/go-containerregistry/pkg/authn that we use in the dagger CLI that lets you read the authentication token needed for interacting w/ container registries using the same logic as is used by docker.

Our use of it is slightly different (https://github.com/sipsma/dagger/blob/56d14a57d241f0cc6abf4e7d0a54ebad09243485/engine/client/docker.go#L54-L54) but I think what you'd want is something along the lines of authn.DefaultKeychain.Resolve(<registry resource>).Authorization() (I did not try this yet, just going off their code/docs).

That should give you the same creds you get normally. The main possible problem I could see happening is if your company doesn't use username/password and instead uses bearer auth, which we haven't incorporated into the WithRegistryAuth API yet.

Let me know if this sounds plausible though, we can keep looking down this path if so.

3. Technically I thought of a hack you could do where you run dagger listen on your host and then use host proxying to provide that to your nested execs. I almost didn't mention this because it feels so hacky, but I guess I will for completeness sake 😄

4. The last option is that I can just add another bool flag to the API that lets you decide whether your nested exec gets connected back to the same session or not (which was the old behavior). I'm not 100% opposed to this, just would prefer to avoid expanding the API if at all possible.