v0.9.0 regression in buildkit proxy config | Dagger | Page 1

toxic solstice Oct 23, 2023, 3:46 PM

#

I have a dagger pipeline where I am explicitly setting the ~/.docker/config.json (before starting the engine) so that the runner picks it up and sets it so that I can pull public docker images within my corp network. This has been working great until v0.9.0.

After updating to v0.9.0, the docker pull does timeout if it's trying to pull from public docker which leads me to think the proxy is not properly set within the container.

To help pinpoint the issue this seems to happen when I try to run the docker:dind service container as a sidecar. The only change I did was change the service container to a *dagger.Service. So this leads me to think that the regression is with the services V2 code. The error also happens in one of my unit tests, which is using dagger-in-dagger.

The pipeline still works fine with v0.8.8.

steady charm Oct 23, 2023, 3:53 PM

#

What's the part that's actually timing out?
Is it the starting dagger engine step, or is it actually something in the pipeline?

Do you have any minimal logs or something that you could share?

toxic solstice Oct 23, 2023, 3:54 PM

#

It's timing out pulling docker:dind. Within dagger engine. That is being pulled as part of a service

steady charm Oct 23, 2023, 3:55 PM

#

do you have a snippet of the buildkit proxy config you could share? censored is fine obviously

#

for context - we took a couple of patch updates in buildkit for v0.9.0, so it's also possible that some of those might have had some bugs in them

toxic solstice Oct 23, 2023, 3:57 PM

#

Sure! Do you read jenkins? 🙂

stage ("Prepare") {
          steps{
          script{
            def pconfig = [proxies:[default:[
                httpProxy:"$env.HTTP_PROXY",
                httpsProxy:"$env.HTTPS_PROXY",
                noProxy:"$env.NO_PROXY"
                ]]]
            def jpconfig = jsonBuilder(pconfig)
            println(jpconfig)
            
            sh("mkdir -p ~/.docker")
            sh("echo '$jpconfig' > ~/.docker/config.json")
          }
            }
        }

steady charm Oct 23, 2023, 3:57 PM

#

ok i think that makes sense 🎉

#

hm, so as far as i'm aware - this would just make docker use the HTTP_PROXY
dagger itself would still not be using it, which would cause the pull to not succeed
cc @quaint shoal

quaint shoal Oct 23, 2023, 4:06 PM

#

~~@toxic solstice what version did you upgrade from?~~ nevermind I see it at the end - 0.8.8

toxic solstice Oct 23, 2023, 4:06 PM

#

This above config has been working for me. You are right it sets the docker proxy. But in Jenkins we are running docker-in-docker. So when buildkit starts up it picks up the docker http proxy

#

v0.8.8

quaint shoal Oct 23, 2023, 4:14 PM

#

trying to find anywhere in our stack that uses that file for proxies specifically, haven't found anything yet. only auth.

#

Buildkit is coded to respect "standard" proxy env vars from the environment, so you should be able to just set those env vars like normal on the engine side

#

https://github.com/moby/buildkit/blob/86e25b3ad8c2/util/resolver/resolver.go#L200

toxic solstice Oct 23, 2023, 4:31 PM

#

quaint shoal <https://github.com/moby/buildkit/blob/86e25b3ad8c2/util/resolver/resolver.go#L2...

My apologies, this was a false alarm. The actual "Prepare" stage (where I set the proxy config) was not run because I re-ran the pipeline from the blue ocean plugin. 🤦‍♂️ . I only realized once I opened the regular Jenkins view. I am not a fan of Jenkins at all! Hence my desperate attempt to adopt dagger to get away from it! At least what I can control. My org is still going to use Jenkins for ci/cd orchestration.

#v0.9.0 regression in buildkit proxy config