Dagger module with coorporate proxy | Dagger | Page 1

long belfry May 27, 2024, 9:23 AM

#

Hi, I try to use dagger module behind coorporate proxy. I use the dagger 0.11.4. This version must include PR thar permit to use dagger module with proxy. Is not the prupose of https://github.com/dagger/dagger/pull/7255 to use dagger module with proxy ?
I have checked that my dagger engine is on the right version (0.11.4) and have the HTTP_PROXY and HTTPS_PROXY setted (via ~/.docker/config.json) ..

When I run dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11 it seems to access direclty on internet instead to use proxy ...

GitHub

engine: support for system proxy settings by sipsma · Pull Request ...

(related issue: #6599)
This adds support for:

Automatically passing standard proxy settings (HTTP_PROXY, HTTPS_PROXY, NO_PROXY, ALL_PROXY, FTP_PROXY) from the engine to all containers.
Automatical...

civic copper May 28, 2024, 1:25 PM

#

@long belfry seems to work here:

130|marcos:tmp/lala (⎈ |kind-kind)$ HTTPS_PROXY=http://foo.bar dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11
10:24:59 WRN canceling... (press again to exit immediately)

● moduleSource(refString: "github.com/kpenfound/dagger-modules/golang@v0.1.11"): ModuleSource! 1.0s
  ● cache request: git://github.com/kpenfound/dagger-modules#golang/v0.1.11 0.4s

10:24:58 WRN failed to resolve image; falling back to leftover engine error="Get \"https://registry.dagger.io/v2/\": proxyconnect tcp: dial tcp: lookup foo.bar on 127.0.0.53:53: no such host"
Error: failed to get module ref kind: returned error 502 Bad Gateway: http do: Post "http://dagger/query": context canceled

as you can see, setting HTTPS_PROXY to some random value effectively makes the dagger install fail

#

any chance you can share an output of your dagger install?

long belfry May 28, 2024, 1:50 PM

#

If i put bad value, It failed like you

[user@workspace2f5337dac8e4400e-d8c9ff485-zr4q6 dagger]$ HTTPS_PROXY=http://127.0.0.1:8080 dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11
✘ moduleSource(refString: "github.com/kpenfound/dagger-modules/golang@v0.1.11"): ModuleSource! 2m9.1s
! failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128416 ms: Couldn't connect to server

  ✘ cache request: git://github.com/kpenfound/dagger-modules#golang/v0.1.11 2m8.4s
  ! failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128416 ms: Couldn't connect to server


13:48:01 WRN failed to resolve image; falling back to leftover engine error="Get \"https://registry.dagger.io/v2/\": proxyconnect tcp: dial tcp 127.0.0.1:8080: connect: connection refused"
Error: failed to get module ref kind: input: moduleSource resolve: failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128416 ms: Couldn't connect to server

#

If I use the real proxy, It just stuck to timeout

[user@workspace2f5337dac8e4400e-d8c9ff485-zr4q6 dagger]$ HTTPS_PROXY=http://squid.squid.svc.cluster.local:8080 dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11
✘ moduleSource(refString: "github.com/kpenfound/dagger-modules/golang@v0.1.11"): ModuleSource! 2m8.9s
! failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128458 ms: Couldn't connect to server

  ✘ cache request: git://github.com/kpenfound/dagger-modules#golang/v0.1.11 2m8.5s
  ! failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128458 ms: Couldn't connect to server


Error: failed to get module ref kind: input: moduleSource resolve: failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 128458 ms: Couldn't connect to server

#

I doesn't understand how to local proxy enter in the game. When I run the command, I look that is create container on docker engine (dagger-engine-xxx). It consule the ~/.docker/config.json to set the HTTP_PROXY, HTTPS_PROXY and NO_PROXY on this container

civic copper May 28, 2024, 2:03 PM

#

@long belfryjust checking if HTTPS_PROXY seems the right env value here given that I see that your proxy is currently exposing an http (no TLS) endpoint

#

does it fail with the same error with HTTP_PROXY?

long belfry May 28, 2024, 2:08 PM

#

 HTTP_PROXY=http://squid.squid.svc.cluster.local:8080 dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11
✘ moduleSource(refString: "github.com/kpenfound/dagger-modules/golang@v0.1.11"): ModuleSource! 2m10.2s
! failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129782 ms: Couldn't connect to server

  ✘ cache request: git://github.com/kpenfound/dagger-modules#golang/v0.1.11 2m9.8s
  ! failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129782 ms: Couldn't connect to server


Error: failed to get module ref kind: input: moduleSource resolve: failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129782 ms: Couldn't connect to server

civic copper May 28, 2024, 2:11 PM

#

ok, it just hangs

#

have 5m to jump into #911305510882513037 and check this out really quickly?

long belfry May 28, 2024, 2:34 PM

#

hum I only read / write english. I doesn't speak it

#

the HTTP_PROXY that we set juste before dagger install. It only used on local dagger cli (http client under the wood). Or it somewhere transfered to dagger engine ?

long belfry May 28, 2024, 2:53 PM

#

Hum, I Think i understand. It will create a kind of container inside buildkit that run inside the docker-engine container ...

#

If I run regular dagger pipeline. I can create container and set HTTP_PROXY, HTTPS_PROXY with same value here, and all right fine. I can download files on internet throght proxy (with curl)

civic copper May 28, 2024, 3:23 PM

#

@long belfry ok, I think I know that happens. You need to set those variables to the engine yourself since the CLI won't pass then automatically when starting it

civic copper May 28, 2024, 4:01 PM

#

how are you currently deploying the engine?

long belfry May 29, 2024, 9:09 AM

#

the dagger cli create automatically to dagger engine container on top of docker. I can enter on engine (docker exec -ti dagger-engine-xxx sh). The env variable HTTP_PROXY and HTTPS_PROXY are right setted.

#

They are a way to look if HTTP_PROXY / HTTPS_PROXY are right setted on container created on the flow on top of buildkit when invoke the module installation ?

civic copper May 29, 2024, 1:19 PM

#

long belfry They are a way to look if HTTP_PROXY / HTTPS_PROXY are right setted on container...

setting the proxy at the engine level should be enough AFAIK. @long belfry can you try by not setting the proxy in the CLI and only have it configured in the engine? could you also validate that if you do a docker exec -ti $engine you actually have internet connection in the engine?

long belfry May 29, 2024, 3:29 PM

#

[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ unset HTTP_PROXY
[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ unset HTTPS_PROXY
[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ unset NO_PROXY
[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ echo $HTTP_PROXY

[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11
✘ moduleSource(refString: "github.com/kpenfound/dagger-modules/golang@v0.1.11"): ModuleSource! 2m10.1s
! failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129565 ms: Couldn't connect to server

  ✘ cache request: git://github.com/kpenfound/dagger-modules#golang/v0.1.11 2m9.6s
  ! failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129565 ms: Couldn't connect to server


Error: failed to get module ref kind: input: moduleSource resolve: failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128
stderr:
fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129565 ms: Couldn't connect to server

#

and then

#

`[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c522737e7d6c registry.dagger.io/engine:v0.11.5 "dagger-entrypoint.s…" 2 minutes ago Up 2 minutes dagger-engine-f3da0aa799eb2fd3
[user@workspace2f5337dac8e4400e-d8c9ff485-4kc5j dagger]$ docker exec -ti c522737e7d6c sh
/ # apk add --update curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/1) Installing curl (8.5.0-r0)
Executing busybox-1.36.1-r6.trigger
OK: 28 MiB in 44 packages
/ # curl -L https://github.com

<!DOCTYPE html>
<html
lang="en"

data-color-mode="light" data-light-theme="light" data-dark-theme="dark"
data-a11y-animated-images="system" data-a11y-link-underlines="true"

`

#

Like you can see, the proxy work fine insine the dagger engine container

civic copper May 29, 2024, 4:11 PM

#

can you please check if in your engine container logs you see any errors when calling dagger install?

long belfry May 29, 2024, 5:05 PM

#

the only error that I look here is

#

fatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129989 ms: Couldn't connect to server time="2024-05-29T16:58:16Z" level=warning msg="error resolving moduleSource(refString: \"github.com/kpenfound/dagger-modules/golang@v0.1.11\"): ModuleSource!" error="map[error:failed to resolve git src to commit: failed to load cache key: failed to fetch remote https://github.com/kpenfound/dagger-modules: git error: exit status 128\nstderr:\nfatal: unable to access 'https://github.com/kpenfound/dagger-modules/': Failed to connect to github.com port 443 after 129989 ms: Couldn't connect to server\n kind:*fmt.wrapError stack:<nil>]"

civic copper May 29, 2024, 7:47 PM

#

hmm running out of ideas how we could troubleshoot this further. cc @potent cape any thoughts?

TL;DR: the engine seems to have the right HTTPS_PROXY and HTTP_PROXY variables. Execing into the engine and running a curl seems to work. However, dagger install targeting the engine with the proxy variables seems to hang indefinitely. Debug logs don't show anything relevant

potent cape May 29, 2024, 7:53 PM

#

Ah, I wonder if this is that upstream buildkit issue around git subcommands not getting envs (including proxy settings) propagated...

#

I have a couple things in my queue but will check that later

civic copper May 29, 2024, 7:55 PM

#

potent cape Ah, I wonder if this is that upstream buildkit issue around `git` subcommands no...

lol, now I recall I was the one messaging you about that issue 🤣

potent cape May 29, 2024, 7:56 PM

#

civic copper lol, now I recall I was the one messaging you about that issue 🤣

Yeah exactly that one! It's gonna require upstream fixes most likely, but should be trivial (assuming that's what this is)

civic copper May 29, 2024, 7:56 PM

#

potent cape Yeah exactly that one! It's gonna require upstream fixes most likely, but should...

AFAIK it doesn't require upstream changes given that we have our own fork of the git CLI, right? Or has that changed recently?

#

isn't this what we need to change? https://github.com/dagger/dagger/blob/main/engine/sources/gitdns/cli.go#L103

potent cape May 29, 2024, 7:58 PM

#

civic copper AFAIK it doesn't require upstream changes given that we have our own fork of the...

For some reason I thought we only went through that code path for certain configurations of git, but I could be misremembering

civic copper May 29, 2024, 7:58 PM

#

potent cape For some reason I thought we only went through that code path for certain config...

oh ok, that might be the case. In any case.. this is very likely to be the issue

#

so basically HTTP_PROXY doesn't work for accessing git yet 🙏

potent cape May 29, 2024, 8:00 PM

#

Oh no actually we use that code for every git op, so yeah it should be fixable in just our code 🎉

civic copper May 29, 2024, 8:01 PM

#

I'll open a quick issue so we don't forget about it

potent cape May 29, 2024, 8:03 PM

#

civic copper I'll open a quick issue so we don't forget about it

SGTM, my next queued up task was to open a release thread since we have some fixes to get out ASAP, given this is simple enough I'll sneak it in too

civic copper May 29, 2024, 8:03 PM

#

https://github.com/dagger/dagger/issues/7503

GitHub

🐞 git operations don't work if they need to use an http proxy · Iss...

What is the issue? If the git operations in the engine require the HTTP_PROXY to be set, they'll not work given that the proxy values are not passed to the underlying git commands executed. ref...

potent cape May 29, 2024, 8:22 PM

#

Hopefully will get the fix in for this release #1245472263294291978 message

long belfry May 31, 2024, 8:43 AM

#

I just tested on dagger 0.11.6. It's seems better. No more error, but I doesn't found the way to test it.
I run
dagger init --sdk=go . dagger install github.com/kpenfound/dagger-modules/golang@v0.1.11 dagger functions Name Description container-echo Returns a container that echoes whatever string argument is provided grep-dir Returns lines that match a pattern in the files of the provided Directory

#

I apologize to not look the golang module function... I miss understanding something ?

long belfry May 31, 2024, 12:39 PM

#

I finnaly found. I need to use -m github.com/kpenfound/dagger-modules/golang. It's work. Thx for your stuff

#Dagger module with coorporate proxy